Monday, March 4, 2024

Screen Scraping GIS Tax Ditch Lines Substandard

Better to get the GIS data someone else has already digitized than to try to digitize from a screenshot. That is what I tried a few months back before I found better source data recently delineating where stormwater drainage goes.

The idea that ditches are taxable, or tax exempt, is still mysterious to me, but we have them in Maryland.

https://geodata.md.gov/imap/rest/services/Agriculture/MD_NutrientManagementSetbacksFromWaterways/MapServer/5

"Description: This is a 35 foot buffer area along PDA ditches where fertilizer applications are restricted on croplands."

A PDA is a "Public Ditch Association." Essentially we've already done the environmental damage cutting down the forests and then terraforming the soil for industrial crop harvests. The ditches stretch for miles in some Eastern Shore counties. Taxes pay for the maintenance, as roadside ditches will fill in with dirt and trash otherwise. 

Initially I found maps showing the ditches, and the surrounding service area, as online maps.



And a map of fire coverage.


One good starting point for "ditches" is https://data.imap.maryland.gov/search?collection=dataset&q=ditch. I followed QGis tutorial lessons [https://docs.qgis.org/3.34/en/docs/training_manual/forestry/stands_digitizing.html], digitizing the above into 2 layers, one for the boundaries and one for the lines. Not very good resolution when you zoom in.






I found, after searching with different criteria, similar tax ditch geo-data in the Maryland State site, and the Eastern Shore collective site. So instead of trying to redo the digitizing effort I could skip that part. 

When I downloaded the shape files, they looked good. But when I tried to pull them into a PostgreSQL database, the points flew somewhere else. I am unsure what was missed. I made a second attempt by downloading a KML file instead of a set of shape files. These lines and areas worked, after a fashion. My PostGIS skills are still pretty fresh, like only this year did I run a local database. 


I experimented with arrows for directions, as this helps visualize the land slope, even if miniscule. There is a great example here: https://docs.qgis.org/3.34/en/docs/training_manual/vector_analysis/network_analysis.html. The "loop" above is an oddity from the ditch crossing the highway intersection. Changing the scale and map view sometimes triggers taffy-pull looking connection lines.

This page has more about rules: https://plugins.qgis.org/planet/tag/thematics/.

Next step for me is altering the labels so that roads are distinct from ditches, because there is both a Chicken Bridge Road and a Chicken Bridge drainage area/network. 


Tuesday, February 20, 2024

Well Sample Data and GIS Development

 I looked at water quality tests for a few private wells in Maryland to be able to better use GIS tools. After reviewing public information I took note of key fields such as well depth. In order to show connections among the various aspects I started an entity relationship diagram, going from basic SQL to end up with 8 tables.


In order to keep consistent a few constraints were included, such as:


ALTER TABLE

  well

ADD CONSTRAINT

  well_fkey_property

FOREIGN KEY

  (property_fk)

REFERENCES

  property(name)

;


  I decided to connect wells to a property, and buildings likewise. Normally one well is allowed per property, though while a replacement is being drilled one property might have 2 wells, and if a well runs dry you might say the property has no well.

 The old forms (pre-2000?) have Maryland grid coordinates, with accuracy to 1,000 feet, while the newer forms have a place for latitude/longitude. Water quality tests done by commercial lab contain values of constituents ranging from coliform bacteria to lead and arsenic.

 Sometimes the applications have typos.




Oops, mixed up east and north. I did the same when I first tried to load point data into GIS; once I noticed the drift and switched the values the wellhead came online where it should be.

 After the table definitions and test data load I used the QGis example tutorials to include the bits required for my hand-curated data to be mapped correctly.

See: 
https://docs.qgis.org/3.28/en/docs/training_manual/spatial_databases/simple_feature_model.html
https://docs.qgis.org/3.28/en/docs/training_manual/spatial_databases/geometry.html

> insert into geometry_columns values ('','public','well','the_geometry',2,4326,'POINT');

> update well set the_geometry = 'SRID=4326;POINT(-75.882996 38.938629)' where name = 'CO-15-0020' ;

 






An 8-way join, or is 7-way?

select
  r.value ,                             -- value
  r.uom,                                -- units
  p.name as name_p,                     -- param
  substr(r.name,1,12) as name_r,        -- result
  s.well_fk as well,                    -- well
  s.name as name_s,                     -- sample
  substr(s.sample_place,1,4) as place,  --
  substr(p.descr,1,10) as param_desc,   --
  substr(c.name,1,8) as name_c,         -- corp
  w.depth,                              -- well
  w.pump_installed as pump,             -- well
  substr(o.lot,1,10) as lot,            -- prop
  e.name as person,                     -- people
  s.sample_date
from
  people e,
  building b,
  lab_corp c,
  lab_sample s,
  lab_sample_results r,
  parameter p,
  property o,
  well w
where
  e.building_fk = b.name     -- people in building
and
  b.property_fk = o.name     -- building on property
and
  s.well_fk = w.name         -- sample from well
and
  w.property_fk = o.name     -- well on property
and
  r.parameter_fk = p.name
and
  r.lab_sample_fk = s.name
and
  s.lab_corp_fk = c.name
;


What have I accomplished? Now I have environmental data in a reachable database, which I can edit via command line, or even using LibreOffice forms, and now view and update via QGis.
Next I will figure out the step up from points data (the well heads) to polygons (property lines and building walls). Then, with sample data, set up views with contaminant levels on the map.



Looks a lot like dBASE IV, right? 

Sunday, February 4, 2024

Turning "Meats and Bounds" into a Map

I came across a section of a scanned municipal blueprint, finding an inset map labelled with "Meats & Bounds", an obvious misspelling of the surveying phrase "Meets & Bounds", the term for property boundary descriptions in a specific style.



As I wanted to learn more about the town boundaries for equity reasons I also wanted to continue my journey learning how to use QGIS and related tools. In the QGIS tutorials, I found ways to bring shape files into a PostGIS database, and export shape files from the database. This might be elementary to some, and obscure to others. My notes that follow are the steps I took to go from a relatively simple property boundary (a rectangle) into digital formats, with comments on accuracy or lack thereof.

It turns out the description of the original town boundaries are encapsulated in state law or regulations, https://mgaleg.maryland.gov/Pubs/LegisLegal/Muni-Charters/2019-municipal-charter-ridgely.pdf 

Beginning at a point where the center of Central Avenue intersects the center of Pennsylvania Railroad tracks and running with the center of said tracks in a northeasterly direction one–half mile to point in center of said tracks, which said point shall be the beginning for the outlines of said corporation;

The Pennsylvania Railroad has not existed as a corporation for decades, and the tracks through the town are pulled up in places and turned into hiking and biking trails, making the "center of said tracks" less obvious now than it was in 1937.

I began by running Viking on NetBSD 10 (RC3) to create points and lines. I tried both kilometer and mile markers, wishing for feet and yards given passages such as "parallel with Central Avenue forty–eight hundred and seventy–five feet." 


Eyeballing the distances and bearings, I came out close to the starting rectangle edge, though clearly not a professional surveying effort.



I exported the waypoint (Origin) and the tracks as lines 1 through 6. With these files loaded into Google Earth Pro via a KML format, I found the shapes where I expected them, off a bit from OpenStreetMap boundaries, but not too bad for a first try. To simplify the shape, I drew a rectangle/polygon, then loaded that into QGis as a vector layer.


You can see the "miss" on the lower left corner of the municipal boundary. Half a city block, or less...

To go further, I exported the shape from QGis into the 5-sided Esri shape folder, then brought the result into a PostGIS database following the tutorial steps: https://docs.qgis.org/3.28/en/docs/training_manual/spatial_databases/import_export.html




The shape looks like the one above (as it should); the database columns include the renamed "description" as "descriptio".

Warning 1: Field Name of width 255 truncated to 254.
Warning 6: Normalized/laundered field name: 'Description' to 'Descriptio'
Warning 1: Field Descriptio of width 255 truncated to 254.

Now I can load the same shapes into different QGs versions (Windows/BSD/Linux) and not worry about "version too old/new" messages.

Monday, January 22, 2024

Revamp Unboxing SAP Community 2024

The SAP Community site revamp of 2024 is ongoing and promises to be unveiled Tuesday so here are my predictions on Monday.

At the moment blog reads increment on reload questions do not. Implying more work on the latter.  We'll lose all previous read page counts and start from 0. Links to other community members will be about as correct as a week ago.

Questions may be stricken from the record. I bookmarked 2 to see if they make the cut. Criteria were established in the plan where irrelevant topics get purged, and as the TOC reads, SAP at their sole discretion may delete any content. No takebacks.

Broken links and missing images are likely. Like other platform changes, fix or tape over. I know I missed the chance to tag my hundreds of posts, always expecting some algorithm to do it for me.

Blogs that were planned for demolition has a stay of execution so no one should vanish. The "just JSON plan" faltered in the light of day. I had already planned my own recovery process.

When we thought some posts would vanish, I ran a mini project to archive.org my own posts.





Some had been snapped before, some not. Questions might be archived but I didn't look hard except for a couple test samples and a review of the answer thon.




Bio pages were to be revamped, but before the read-only phase started were skeletal or minimal.

Messages and inboxes are a mess, as are contacts given prior art. Remains to be seen how much previous net connections need to be renewed.

What to expect?

Community members will begin to engage in the new space, or decide to skip town. Leader boards and gamiification only go so far.

Search engine optimization will be improved. After the "deadwood" gets flushed from the search caches, which takes time.

Blogs posts will be slower than before given potential permission changes, and the tool belt shuffle going from Word Press to Not Word Press. I am particularly interested in the finer points of code and text snippet displays once I saw a bit of the HTML differences.






Friday, December 29, 2023

Bengies Top 10 for 2023, Barely

O, 2023, what happened?

Besides getting older and slower, the Bengies drive-in movie theatre had bad weather (fog, in December?) and a slew of R-rated flicks we skipped and a few kiddie flicks also given the slip. Still, the Scout Camp-In continues despite the usual suspect challenges (high staff turnover, slow tech progress).

Here are my annual "10 Best Movies I Watched At The Bengies" where "watched" is at least part of the film, if not the whole thing. 



Yes, let's start at 

Number One,

a Ten in Ten of Ten, including the crowd on a weeknight. For fun, we printed our pre-paid ticket order on pink paper. Plenty of folks dressed up, and the happy crowd made for more fun. The snickering jokes about Ken 


Hocus Pocus, in the rain, a second time on the big screen, is Number Two.





Number Three

Guardians 3

https://chaos.social/@jspath55/110392830400706164

The sequel to the sequel, which seemed almost like the same plot, with ennui for the raccoon that doesn't know it is a raccoon. Self -denial runs deep.

Four, and Counting

Indiana Jones & the Dial of Destiny

Phoned in...



Whatever "The Boogeyman" was. we skipped it. PG-13 or no.

And 5 Through 10

The Flash and the Spider Verse

https://chaos.social/@jspath55/110589657062948842

First time I recall a double feature with DC and Marvel! The Flash didn't bring back sixth-grade memories of science lab as I expected. (5,6)

Wonderful Life / Elf

Of these two, let's say I split the difference arriving late and leaving early. Elf was not as much of a wince as I recalled, though minimal dosing helped. And because I had other plans, did not stay through It's A Wonderful Life once the alternate reality began. Seemed a bit like now, sadly. (7,8)




Rebel Without A Cause / Forbidden Planet
https://chaos.social/@jspath55/110498676821941834

To round out the 10, two movies from the 1950s, around the time the Bengies opened, shown on their Drive-In Anniversary special--"FORBIDDEN PLANET" stars Walter Pidgeon, Anne Francis, Leslie Nielsen AND Robby the Robot.

The Under 11

TMN Turtles, unshelled. All right, the Scouts liked it I guess.

Last, and Twelfth, like Iscariot:

Gran Turismo. I definitely asked for Barbie to play, but instead of a somewhat socially conscious show, we got another salute to burning petroleum and continuing our carbon dioxide emissions.


2022 (last year's write up, and links to past Top Tens)

jspath55.blogspot.com/2022/12/bengies-top-10-2022.html


Saturday, December 23, 2023

NetBSD 10 RC1 Review

 








Upgrades from earlier 10.0 beta images went easily except for the Raspberry Pi systems with the kernel in a different place. 


 WIFI


  • amd – external
  • i386 - none
  • pi4 – no wifi, hard-wire ok
  • pi3 – no wifi; hard-wire ok
  • pi02w – wifi yes; hard-wire added
  • pi0w – wifi yes



 Automated Test Framework

I kicked off periodic user test suite cases, noting the minimum and maximum errors per platform. The Pi4 worked best, and the 0W the worst as I never completed a full run, trying various scratch write media to avoid stall-outs on basic shell command runs.

 When the platform would allow full tests runs more than one a day I set up cron jobs; the fastest tested had 60 or so when I stopped the cron.

    3 failed test cases.

    4 failed test cases.

    3 failed test cases.

    3 failed test cases.

    6 failed test cases.

    4 failed test cases.

    4 failed test cases.

    3 failed test cases.



  • amd – (2/7)
  • i386 - (0/3)
  • pi4 – (1/4)
  • pi3 – none yet
  • pi02w – November 26 last run (16 / 20)
  • pi0w – 30 (one good run)



The variation in failures from phase to phase I've called Heisenbergars, elsewhere.

X11

I had 2 minor regressive behaviors after RC1 tests compared to earlier betas, where the AMD machine had flaky screen text. I need to recheck the cables since it went onto a floor mount instead of desk level. The other slowness was on a new Pi4 install, where I struggled through getting the latest UEFI version on an SD card, and ending up with a working install, and 8GB (don't forget to set that last in the UEFI menus). Comparing the CPU temperature to other Pi 4s, it may work better with a running fan. More tests later; the last straw was launching QGIS (except with a Postgre 14 client for package sanity).

WIFI

I guess the fix for using wifi, at least on the Pi 4, is external dongles. The Pi3 worked for me at one point, but the dmesg tempts with "almost, almost":

[     3.368964] bwfm0: Found Firmware file: brcmfmac43455-sdio.bin

[     3.388965] bwfm0: NVRAM file default:    brcmfmac43455-sdio.txt

[     3.388965] bwfm0: NVRAM file model-spec: brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt

[     3.388965] bwfm0: Found NVRAM file: brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt

[     3.388965] bwfm0: CLM file default:    brcmfmac43455-sdio.clm_blob

[     3.388965] bwfm0: CLM file model-spec: brcmfmac43455-sdio.raspberrypi,3-model-b-plus.clm_blob

[     4.999119] bwfm0: CHIPACTIVE

[     5.099132] bwfm0: address ..:..:..:..:..

[     5.099132] bwfm0: wl0: Mar  1 2015 07:29:38 version 7.45.18 (r538002) FWID 01-6a2c8ad4


The ifconfig shows recognition of the wifi circuit but wpa supplicant stumbles.

bwfm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

pi4:

Also, almost

[     2.447895] bwfm0: NVRAM file default:    brcmfmac43455-sdio.txt

[     2.447895] bwfm0: NVRAM file model-spec: brcmfmac43455-sdio.Raspberry Pi 4 Model B.txt

[     2.447895] bwfm0: autoconfiguration error: NVRAM file not available

[     2.447895] bwfm0: CLM file default:    brcmfmac43455-sdio.clm_blob

[     2.447895] bwfm0: CLM file model-spec: brcmfmac43455-sdio.Raspberry Pi 4 Model B.clm_blob



Monday, December 4, 2023

Avenza on the Trail

I got to use the Avenza-hosted Reservation map for the first time onsite in early December 2023 while winter camping with a Scout troop. They needed basic map and compass tutoring, and adding the digital version in sequence reinforces the concepts.

This story is generally chronological for my one-day observations, by site.

Conowingo / Pine Grove

I camped in the field near the "X" shown on this screenshot.  I did a polygon line drawing around the cabin, trying to figure out about the projected skewness. This is my own geoPDF, with a scale of 1:250 and has 150 dpi.

The screenshot was from Friday evening, where I had 95% battery to start.




Flagpole and former memorial sign.

A control point of sorts; someone knows which sign was here, presumably now at RHQ.



Flint Ridge

Latrine surveying. I walked around the pad, taking measurements and images with camera and the Avenza app. In the morning, the power level was at 86% and here shows 81%



Chimney was photographed and geo-tagged. I was definitely *not* inside. Not 9AM yet and 77% power level.





The Dam and the Yellow Trail



Turning on tracking and measuring a few hundred feet left battery at 72%.



I did a track with Avenza, then viewed it later with Google Earth. Interesting.


Frontier

This site has only an official entrance across a bridged stream.





The blue line is a "get there from here" option. Straight line, probably helpful but not guaranteed a safe route! I was not on the bridge here.


Red trail 

Camp entrance








I only posted two screen shots above, from between 10 am and 1PM; the battery level dropped from 70 to under 60%. When I returned to my tent I plugged in an external battery, and by 5PM the phone was back to 100%.

Et cetera

I experimented with a 1:8000 scale, 600 dpi geoPDF with both the official camp map and the unofficial OSM layer.

.



I needed to allow "all the time" location access to Avenza to have the "show a route" function work. Other operations usually work with allowing access "just the once," though often Avenza will claim "not on map" and I needed to stop/start the app. Maybe a bug.

The blue dot can dance around, showing where the app thinks you are, and shows a shrinking diameter as more satellite signals are locked in. My testing was partly the squiggles of the original transformed map image, the delay and aberrations of the Android device (iPhone testers needed too), and the time needed to get a "good enough" spot for a place on the earth.