Tiling for Geo Map Referencing in QGIS

 

Q GIS

JPEG to TIFF to PDF

 QGIS on X11

I needed to convert a JPEG image into a georeferenced PDF, which I was able to do, finding things to fix later along the path.

The issue with starting at the top is the image is too big to share cleanly, so breaking the problem into parts led to tiles showing a fraction of the big picture.

Source of data I used to feed into QGIS were a history file of geo-tagged Panoramio images (such as this one of Oest Chapel; just kidding it's the Marble Hill campsite latrine).

QGIS versions

Turns out the Raspberry Pi 4 (aka 400) factory warranty operating system has an older QGIS release than other systems I tried to use. Windows had 3.32.2-Lima, OpenSUSE had QGIS 3.32.0-Lima, and a couple *BSD were at  qgis-3.28.11 or so. Raspbian had QGIS 3.10.14-A Coruña.

Thus, I worked on OpenSUSE and Windows. Would have like to do more on FreeBSD as that box has the best processors in the house. On the downside, OpenSUSE didn't have Viking, and I couldn't build the venerable XPaint (was able to use screengrab for the docs here in the house).

The "Georeferencer" button is either there or it isn't. I've looked on the older versions and despite the copious documentation pages the function escapes me.

One note I read said you could get away with only 3 control points; the program insisted on at least 4 per run. I used 10 in an earlier projection reference which was tedious but not too complex.

The icons are a little fuzzy to me and I tend to use the pull-down menus. Maybe after doing a few dozen I'll get the shortcuts down.

Georeference and the GCP table

The above image shows a map (loaded from a JPEG file) with 4 control points in the table at the bottom. I've circled those points on the image as there are no text labels in this view.

For this example, I chose the northeast (more or less) corners of a couple buildings - the main warehouse, and Prospect cabin, and the corner of the pool, and about where the water tower is. Even though the spots seem pretty close, there is definite warpage of the trails on different map levels.

After running the referencer code a TIFF image file is produced that contains the geotagging needed to correct use.

Broad_Creek 2023_225dpi-01-OEST-CHAPEL_modified.tif:     

TIFF image data, little-endian, direntries=16, height=477, bps=206, compression=none, PhotometricInterpretation=RGB, width=615

Broad_Creek_2023_225dpi-01-SAFFRAN-QM-TILE_modified.tif: 

TIFF image data, little-endian, direntries=16, height=506, bps=206, compression=none, PhotometricInterpretation=RGB, width=508

The PDF production

I have not set the geo-PDF as a default so must check both of these boxes to get a PDF.

Then it just works.

Loading in Avenza is the best test, even better to run it within the map borders to get the location shown.

Mini vision quest upstream

 I intended to test getting elevations as well as latitude/longitude with a slightly better
than my Coolpix along a stream/drainage channel. Ended up seeing a small vision of what the streambed looked like before we disturbed it.

This figure was taken from the NPS publication on their web site - https://www.nps.gov/hamp/learn/upload/EOA-Hampton-TracingLivesFinalReport-508.pdf - (page 143) and shows "property" and water channels from around 1800 (?). From prior to the Loch Raven Reservoir being created. Arrows show an almost southerly direction on 3 unnamed tributaries of the then-named Petersons Run (I have not located that name on recent maps but haven't looked too far yet).

Researching landmarks/placemarks/benchmarks I found a 1938 "monument" by the USC & GS (then-named) in the vicinity. So I started there in an elevation quest (to level-set one might say). 

My bookmark (pin) is:

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">
<Placemark>
<name>BM Hampton</name>
<Point>
<coordinates>-76.59533879999999,39.4293622,0</coordinates>
</Point>
</Placemark>
</kml>

The records show the location is obscured.

 

I found a lot of topo data from the JHU Sheridan Libraries site

• JScholarship Home
• Library (Sheridan) General Collections
• Maps and Atlases
• Maryland State, County and Baltimore City Atlases 

 - https://jscholarship.library.jhu.edu/handle/1774.2/36112

Enough to get to the monument later, after the stream stomp. Good conditions for a field inspection given the lack of recent rain. More dry footing, and on the flip side, good mosquito breeding conditions. Not to mention poison ivy growth spurts.

The most downstream section goes into a culvert under a state/county highway, and I walked "up" from there. I'd been on the banks, obscured by overgrowth, before, just not at waters-edge.

Small pebbles, gravel, rocks, and at one stretch, broken-up concrete rubble fill (more or less). Not until I reached a section of shale (?) outcropping did I understand I was on land undisturbed by people. And by people, yes, white people.

I adjusted this image by 3 degrees once I spotted the building window that is unnaturally plumb.

And realized the streambed formerly went toward where the building and parking lot are now. Those streams that went southward can't anymore, though the sinkholes and disappearing streamflow says otherwise.

 Also see: https://www.baltimorecountymd.gov/departments/environment/watersheds/index.html

Beta Testing NetBSD 10; the Intermittent Result Set

After running sets of tests on the NetBSD 10.0 beta release, I had opened about 2 dozen problem reports of varying severity, and thought it time to take stock. Some have been closed, many have identified solutions, and a few are head scratchers. 

Full test cycle on an ARM soc

I set up a cron job to run the /usr/tests/ suite on a Pi 02W; it was taking a few hours to complete which allowed 8 hour restarts. The slice is roughly 8 hours, and includes CPU temperature, interrupts, and memory. Sorry I cut off the scale here.

This image is from 48 hours and shows CPU core temperature (Celsius) during the automated test cycle

The range goes from 45 to 50 with no tests running, then maxes to 60 at points.

Reviewing the test results (failures, in other words) reveals a few issues worth opening tickets, as did reading up on the test algorithms and reference pages. The aspect that intrigued me primarily were tests that did not pass or fail consistently. A typical case is running out of memory caused by creating sample data of variable dimensions. Side effects of this, for example:

[ 61059.604145] UVM: pid 12801 (h_libarchive), uid 0 killed: out of swap

Sometimes this is visible with vmstat commands:

 0 0     7960 330868    0   0   0    0    0    0  0 8519   18  36  0 0 100
 0 0     7960 330868    0   0   0    0    0    0  0 8521   18  38  0  0 100
Mon May  1 23:24:01 UTC 2023
 procs    memory      page                       disk faults      cpu
 r b      avm    fre  flt  re  pi   po   fr   sr l0   in   sy  cs us sy id
 1 0     1968 337656   47   0   0    0    4    4  1 8503  184  67  0  1 99
 1 0    10828 328948 7245   0   0    0    0    0 99 9117 5113 989  7 13 80
 2 0    19112 320688 2576   0   0    0    0    0 42 9392 4107 1979 12 16 72
 0 0     8568 331328 1534   0   0    0    0    0 89 8919 1788 949  1  5 95
 0 0     7896 332000  201   0   0    0    0    0  0 8534  254  44  0  0 100
 0 0     7896 332000    0   0   0    0    0    0  0 8527   18  36  0  0 100

Given those results are not indicative of system faults, looking at the other test case failures and sorting them by fixed/workarounds/etc, I have 3 that I don't know the root cause, and that's all of consequence. Two of them are wifi-relates and the third is a failure to compile profiling data into an an executable on a 32-bit Pi 02W. 

One of the intermittent has corresponding out-of-memory messages (57291). While it's a low priority issue because the tests would pass with more storage space, there is probably a way to avoid runaway space requests and still have useful tests.

Tickets with unknown root cause:

1. misc/57303 [serious/medium]: ATF unit test usr.sbin/tcpdump/t_tcpdump fails when wireless active on amd64
2. toolchain/57321 [non-critical/medium]: ATF test case usr.bin/cc/t_hello:hello_profile fails on RPI02W/evbarm only
3. bin/57366 [serious/medium]: Automated test usr.sbin/tcpdump/t_tcpdump:promiscuous fails on Rpi3 with wifi active

Tickets with known cause/workaround:

• kern/57185 [non-critical/medium]: Python build fails on 10_BETA due to no entropy on Atom CPU system
• misc/57286 [serious/medium]: Unit test fs/tmpfs/t_vnode_leak fails in ATF Tests suite
• misc/57291 [serious/medium]: Unit test for lib/libc/regex/t_exhaust fails in ATF Tests suite with signal 9
• lib/57314 [serious/medium]: 
• ATF unit tests fail on 3 of 7 cases in program lib/libc/c063/t_utimensat on evbarm/Rpi 02W
• kern/57320 [serious/medium]: ATF test case kernel/t_magic_symlinks:machine_arch fails on RPI02W/evbarm only [?]
• lib/57331 [serious/medium]: Automated unit test lib/libc/net/t_servent:servent fails on amd64 only
• misc/57361 [non-critical/medium]: Automated test t_archive fails 2 test cases on an Rpi3

Tickets fixed:

• misc/57284 [serious/medium]: Unit test for envstat fails in ATF Tests suite on one machine
• kern/57319 [serious/medium]: ATF test case kernel/t_magic_symlinks fails as non-root instead of showing expected fail message

Tickets with intermittent pass/fail results:

• misc/57291 [serious/medium]: Unit test for lib/libc/regex/t_exhaust fails in ATF Tests suite with signal 9
• kern/57345 [serious/medium]: Automated test kernel/kqueue/t_empty fails intermittently on an amd64 machine
• toolchain/57351 [serious/medium]: Automated test usr.bin/c++/t_tsan_vptr_race:vptr_race fails intermittently on an amd64 machine
• kern/57371 [serious/medium]: Automated test fs/vfs/t_vnops:nfs_rename_reg_nodir fails intermittently on Rpi3 and Rpi4
• kern/57385 [serious/medium]: Automated test case for puffs file system fails intermittently on different architectures

Documentation tickets:

• misc/57318 [non-critical/low]: Minor typos in an automated test case - atf/tools/atf-run_test
• misc/57332 [non-critical/low]: Replace 'http' with 'https' on netbsd.org links found in man pages
• misc/57343 [non-critical/low]: Typo in automated test rumpkern/t_vm.c  ('this' should say 'thus')
• misc/57344 [non-critical/low]: WIki page for evbarm port missing rpi4 mention
• misc/57347 [non-critical/low]: Several man pages have obsolete file location references under /usr/share/doc
• misc/57397 [non-critical/low]: Minor comment typos in t_vnops.c test program

Upsizing a Pi Zero (2W) to a Pi 3 Footprint

Recently I learned, and was intrigued, by two products from WaveShare, both enabling a small footprint Raspberry Pi Zero (2W in my case) to fill a Raspberry Pi 3 case by including a suite of adapters, plugs, and sockets. With the uninspiring brand name, "Raspberry Pi Zero [] to 3B Adapter." 

Raspberry Pi Zero To 3B Adapter, Alternative Solution for Raspberry Pi 3 Model B/B+

Raspberry Pi Zero 2W To 3B Adapter, Alternative Solution for Raspberry Pi 3 Model B/B+

Since I have 2 Pi's running NetBSD 10 BETA, I decided to try each version in a different system. Pictures follow; the basic difference between the A and B versions is the former may work with the vanilla Pi Zero while the latter probably doesn't.

Good/Bad/Ugly...

The "A" Board

Good: Cheaper of the 2 boards. HDMI full size socket (though now what do you do with the micro-HDMI already at hand?)

Bad: Does not relocate the microSD socket, making case adjustments necessary. See images.

Ugly: Ethernet is 100 BT.

Assembly is a little challenging for two hands, as the three connections for power, USB, and video must be made simultaneously. The connections seemed firm and sound once made, but I'd want to be careful not to torque the board any when joining to the Pi. I wouldn't think you would change this once plugged on (I don't plan to) either.

Access to the 40 pin bus is not obstructed as the board design wraps the opposite side.

NetBSD compatibility

Once I powered up the combined boards and booted a previously running system, I was happy to see the startup succeeded, with the provided RealTek ethernet stack working fine with NetBSD (some versions don't, or seem to only provide 100BT while claiming gigabit).

The USB V2 ports were fine; given the Zero doesn't have USB-3 I wouldn't expect an add-on board to provide accelerated throughput.

The "B" Board

Good: Includes an audio jack. Really good: Providing a micro-USB socket in the expected Pi3 location, somehow rewiring the onboard socket to an outboard one (I wonder what happens if you try to put chips in both sockets?) The funny aspect of this feature was that it went unremarked in the online feature set.

Bad: Case fit was a little tight; tolerances are fair. The connections were different than the A board, as only the HDMI port  is plugged in, while other connections get routed through "Pogo" pins. It sounded dubious (to me, who grew up reading Pogo in the comics) and I'm happily converted.

Ugly: Really can't see any downsides yet. Well, to be fair, being unable to access on-board wireless is a downside, requiring external wired or wireless dongles which might (or might not) work well under NetBSD.

Wifi on a pi3 with NetBSD?

http://mail-index.netbsd.org/port-arm/2019/08/31/msg006102.html

https://wiki.netbsd.org/ports/evbarm/raspberry_pi/

• RPI3 and RPI0W builtin WiFi - bwfm(4)
• "Note that the built-in WiFi in the RPI3 is not yet supported."

FreeBSD doesn't (yet) support Pi Wifi either - see https://wiki.freebsd.org/arm/Raspberry%20Pi

NetBSD compatibility

As this board appears to include an identical ethernet adapter logic as the "A" board, this system also rebooted with no issues (other than the usual DHCP name shuffling due to alternate MAC addresses than earlier adapters.

The "B" board has an audio jack, which the other does not. Though it seemed to be a valid device, my first tests playing music were failures of varying degrees. VLC/MPG123 worked once, then got really choppy. Error logs filled with messages.

Errors/messages:

A

[     5.502165] ure0: Realtek (0x0bda) USB 10/100 LAN (0x8152), rev 2.10/20.00, addr 6

B

[     3.828299] ure0: Realtek (0x0bda) USB 10/100 LAN (0x8152), rev 2.10/20.00, addr 4

[     4.428379] uaudio0 at uhub1 port 6 configuration 1 interface 0

[     4.438382] uaudio0: Solid State System Co.,Ltd. (0x0c76) USB PnP Audio Device (0x1203), rev 1.10/1.00, addr 5

Mar 25 23:00:00 b /netbsd: [ 5929.7443850] uaudio0: pintr error: IOERROR

Mar 25 23:00:00 b /netbsd: [ 5929.7443850] uaudio0: uaudio_chan_pintr: count(40) != size(1920), status(13)

Wrap-Up

The Pi Zero started out as a $5 computer.... [ raspberrypi.com/news/raspberry-pi-zero/ ]

Then it crept up a bit with the more powerful 2W. OK, it tripled at $15, but that doesn't include a necessary power supply, much less nice to have features like USB ports (mainboard has one, in the micro style). Spending as much or more on an expansion board needs justification, or at least a comparison between features versus price. I like the "all-in-one" feel of the expanded system in a tidy case, compared to a board with plugs hanging out left right and center.

Images

A

Top view. I left off a fan as the Zero is not a heat source of consequence.

Side view, showing the plastic frame cut away to allow the microSD card to stick out where it is on the Zero, but in a Three case.

The cutaway.

B

Zero, screwed in on top of the adapter board.

Can't close the case with the SD card on the Zero.

The closed case with the card in the new location.

It fits!

The end. For now.

32/64-Bit and NetBSD 10 BETA

 I've had "Pentium" type machine for decades, starting with using NetBSD 0.9 on a 386 or 486 in 1997?, and now, trying out various processors and systems for beta testing 10.0.

Initially, one "hand-me-down" box seemed like an older, 32-bit processor, based on how MS-Windows viewed the details, complaining the OS version could not be updated beyond Windows 9 32 bit, or so. Thus my first pass with NetBSD was to install the i386 release, testing it for several weeks before the light dawned that this weak looking machine had more to offer if set up right. Motherboard has a 2014 date stamp and my research shows this board was designed for laptops then slipped into a largish case because marketing/.

Processor/kernel

[ 1.000000] NetBSD 10.0_BETA (GENERIC) #0: Mon Jan 23 16:02:49 UTC 2023

[ 1.000000] mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC

[ 1.000000] NetBSD 10.0_BETA (GENERIC) #0: Sun Feb 12 12:39:37 UTC 2023

[ 1.000000] mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC

/netbsd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for NetBSD 10.0, not stripped

[     1.000004] cpu0: AMD A8-6410 APU with AMD Radeon R5 Graphics 

[     1.000004] cpu0: node 0, package 0, core 0, smt 0

[     1.000004] cpu0: SVM disabled by the BIOS

[     1.000004] cpu1 at mainbus0 apid 1

[     1.000004] cpu1: AMD A8-6410 APU with AMD Radeon R5 Graphics  

[     1.000004] cpu1: node 0, package 0, core 1, smt 0

[     1.000004] cpu2 at mainbus0 apid 2

[     1.000004] cpu2: AMD A8-6410 APU with AMD Radeon R5 Graphics  

[     1.000004] cpu2: node 0, package 0, core 2, smt 0

[     1.000004] cpu3 at mainbus0 apid 3

[     1.000004] cpu3: AMD A8-6410 APU with AMD Radeon R5 Graphics   

[     1.000004] cpu3: node 0, package 0, core 3, smt 0

4-cores, N threads(?)

Two for the price of one

First win: the AMD64 beta ISO fits on a standard CD image footprint. Yay.

Next, big win: the machine has both VGA and DVI outputs, which may sound archaic with HDMI and higher resolutions available (more on the higher end AMD system later), but it turns out this little board contains 2 independent video outputs. With a couple adapters, I've got dual-HDMI screens and X Windows stretching in them.

Minor fault: the onboard ethernet adapter only runs at 100BT. On closer examination, there is an onboard mini-PCIE connector that could conceivably allow a Gigabit board in its place. Given the risk of breaking the installed board (which NetBSD works fine with), or installing a replacement board that might not work, I decided to go with an outboard USB ethernet dongle (more on that hardware also later).

Memory

Without changing the system (I guess there be BIOS mysteries here), going x86 kicked up the visible memory seen by NetBSD.

[     1.000000] total memory = 2759 MB

[     1.000000] avail memory = 2684 MB

[     1.000000] total memory = 7863 MB

[     1.000000] avail memory = 7581 MB

Also good news, there is an empty memory slot onboard. Now, to find chips that fit...

I wonder how old that CR2032 button battery is? Hmm.

Video

[     4.877451] [drm] Radeon Display Connectors

[     4.877451] [drm] Connector 0:

[     4.877451] [drm]   DVI-D-1

[     4.888080] [drm] Connector 1:

[     4.888080] [drm]   VGA-1

[     4.947445] radeondrmkmsfb0 at radeon0

[     4.957450] [drm] Initialized radeon 2.50.0 20080528 for radeon0 on minor 0

[     4.957450] radeondrmkmsfb0: framebuffer at 0xc0615000, size 1920x1080, depth 32, stride 7680

X Windows starts up just fine; the switch from twm to cwtm is still pleasant (one app installed twm and I quickly rewrote that xtartup script).

XSCreensaver works nicely; the range of hacks and their speed are always a system performance indicator, as well as hardware and software library depth. Two separate hacks running at the same time is pretty cool, maybe the first UNIX system I've had (and there have been many) bifurcating for me.

Oops:

$ file xscreensaver-get.core

xscreensaver-get.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), NetBSD-style, from 'xscreensaver-get', pid=18023, uid=1000, gid=100, nlwps=1, lwp=18023 (signal 11/code 32767)

Network

The "net" in NetBSD implies internet operations should be simple and thorough. I've installed this OS enough times to know where interface are configured, how Network Time Protocol works best, and basic home mesh set up. Good news for this motherboard containing a supported ethernet board.

I was unsatisfied with the restricted 100BT connection and tried several alternatives for wired and wireless.

For wifi, I have one USB dongle that NetBSD is happy with, and others that either don't work at all, or have partial connectivity. I switched out one that as been running fine on an even older i386 server and it's been fine (the big test will be running wireless only, rather than dual connections).

Think Penguin sells open source hardware. Linux OS targeted, but sometimes NetBSD uses benefit from that communal spirit. Not always, though. Short version (more below): the wireless dongle didn't work for me on NetBSD, but the gigabit USB dongle did.

Built-in:

[     1.055110] re0 at pci1 dev 0 function 0: RealTek 8100E/8101E/8102E/8102EL PCIe 10/100BaseTX (rev. 0x07)

[     1.055110] re0: interrupting at msix1 vec 0

[     1.055110] re0: RTL8106E (0x4480)

There appear to be antenna wires connected to the PCIE board, though NetBSD doesn't report any interface other than the wired one.

Built-out:

[     2.727276] rgephy0 at axen0 phy 3: RTL8211E 1000BASE-T media interface

[     2.777269] rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto

Mar  8 21:32:16 amd64 /netbsd: [ 771903.1443413] athn0 at uhub1 port 2

Mar  8 21:32:17 amd64 /netbsd: [ 771904.1242585] : Atheros AR9271

Mar  8 21:32:17 amd64 /netbsd: [ 771904.1242585] athn0: rev 1 (1T1R), ROM rev 15, address 

Mar  8 21:47:22 amd64 dhcpcd[4518]: re0: carrier lost - roaming

Mar  8 21:47:22 amd64 dhcpcd[4518]: axen0: changing route to ...

interface rates:

ping times:

Arrows are before the ThinkPenguin AXEN0 interface was connected

Performance

For beta testing, I like to install familiar software, run benchmarks, and look for anomalies. The first oddity was the CPU temperature increase after switching from an i386 NetBSD beta build to the amd64.

On the left side, i386 running under 27 degrees Celsius average, with the amd64 on the right closer to 38 degrees. Inexplicable so far.

Byte Bench

Dhrystone 2 using register variables     6703557.1 lps   (10.0 secs, 10 samples)

Double-Precision Whetstone                 1321.6 MWIPS (10.0 secs, 10 samples)

Recursion Test--Tower of Hanoi           116108.7 lps   (20.0 secs, 3 samples)

Dhrystone 2 using register variables        116700.0  6703557.1      574.4

Double-Precision Whetstone                      55.0     1321.6      240.3

H Bench (or its close rival lmbench)

lmbench1.1 results for NetBSD ...

...

Process fork+exit: 1002.1667 microseconds

Process fork+execve: 2887.0000 microseconds

Process fork+/bin/sh -c: 6732.0000 microseconds

$ cat  /usr/pkg/share/hbench/Results/netbsd10.0-x86_64/amd64/lat_syscall_getpid

0.2494

0.2534

0.2522

0.2526

0.2546

0.2584

0.2549

0.2530

0.2502

0.2486

NetBSD Unit Tests

There are test suites under /usr/tests on recent NetBSD systems, which I was unaware of until recently, so running these tests are as much for me to learn about the system architecture as it is to find errors o omissions. In a prior post, I reported more failures on architectures for as ARM than on i386 or amd64, which makes a little sense given their relative code base ages. 

Fortunately, I managed a complete test run while this system was running the NetBSD i386 build, and have finished another round using the amd64 build. Interesting results in that a few failures are mutual while most of the very small set are from only one build.

Line counts for the test runs:

   12101 testsuite-amd64.csv

   11957 testsuite-i386.csv

Common (shared) failure(s), as extracted from the CSV output (timestamps omitted for clarity):

• tc, sbin/envstat/t_envstat, zerotemp, failed, Test case was expecting a failure but none were raised
• tp, sbin/envstat/t_envstat, failed

Failures seen only on i386:

1. tp  include/t_paths  failed
2. tp  kernel/kqueue/t_empty  failed
3. tp  lib/libarchive/t_libarchive  failed
4. tp  lib/libc/kevent_nullmnt/t_nullmnt  failed
5. tp  lib/librumphijack/t_tcpip  failed
6. tp  net/net/t_bind  failed
7. tp  net/net/t_unix  failed
8. tp  net/altq/t_cbq  failed
9. tp  crypto/opencrypto/t_opencrypto  failed

Failures seen only on amd64:

1. tp  lib/libc/net/t_servent  failed
2. tp  net/if_wg/t_basic  failed
3. tp  usr.bin/cc/t_tsan_data_race  failed
4. tp  usr.bin/make/t_make  failed
5. tp  usr.sbin/tcpdump/t_tcpdump  failed
6. tp  fs/tmpfs/t_vnode_leak  failed

Apologies if this semi-manual error search missed any useful test results. 

[     1.055087] amdtemp0 at amdnb_misc0: AMD CPU Temperature Sensors (Family16h)

[     1.055123] amdtemp0 at amdnb_misc0: AMD CPU Temperature Sensors (Family16h)

i386:

<tp id="sbin/envstat/t_envstat">

<tc id="zerotemp">

<so>amdtemp0 = 39.250 =~ 39</so>

<so>Skipping non-existent coretemp0</so>

<so>Skipping non-existent acpitz0</so>

<failed>Test case was expecting a failure but none were raised</failed>

amd64:

<tp id="sbin/envstat/t_envstat">

<tc id="zerotemp">

<so>amdtemp0 = 45.125 =~ 45</so>

<so>Skipping non-existent coretemp0</so>

<so>Skipping non-existent acpitz0</so>

<failed>Test case was expecting a failure but none were raised</failed>

I presume the root cause for this hardware might be similar to the other 686?

[     1.000004] cpu0: AMD A8-6410 APU with AMD Radeon R5 Graphics 

$ /sbin/dmesg | /usr/bin/grep -i temp

[     1.055087] amdtemp0 at amdnb_misc0: AMD CPU Temperature Sensors (Family16h)

[     1.055123] amdtemp0 at amdnb_misc0: AMD CPU Temperature Sensors (Family16h)

$ /usr/sbin/envstat

                      Current  CritMax  WarnMax  WarnMin  CritMin  Unit

[amdtemp0]

  cpu0 temperature:    39.375                                      degC

On an AtomPC i386, envstat was fine:

tc, sbin/envstat/t_envstat, zerotemp, passed

<tp id="sbin/envstat/t_envstat">

<tc id="zerotemp">

<so>Skipping non-existent amdtemp0</so>

<so>coretemp0 = 58.000 =~ 58</so>

<so>acpitz0 = 127.000 =~ 127</so>

<passed />

$ /sbin/dmesg | /usr/bin/grep -i temp

[     1.010406] coretemp0 at cpu0: thermal sensor, 1 C resolution, Tjmax=100

$ /usr/sbin/envstat

                      Current  CritMax  WarnMax  WarnMin  CritMin  Unit

[acpitz0]

       temperature:    52.000  127.000                             degC

[acpitz1]

       temperature:    51.000  127.000                             degC

[coretemp0]

  cpu0 temperature:    53.000                                      degC

Think Penguin

Their wifi dongle has only worked on a small number of OSes I've tried, though I had hoped it would 'just work.' Part number: TPE-N150USB.

The gigabit dongles have been fine so far, and no losses on a temperamental Pi 02W. Though it has gone offline once (no dump). TPE-1000NET3

Mar 19 00:00:00 aa syslogd[974]: restart

Mar 19 12:31:31 aa syslogd[806]: restart

Mar 19 12:31:31 aa /netbsd: [   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, [...]

1.0000000]     The Regents of the University of California.  All rights reserved.

Mar 19 12:31:31 aa /netbsd: [   1.0000000] NetBSD 10.0_BETA (GENERIC) #0: Fri Jan 13 19:15:32 UTC 2023

Mar 19 12:31:31 aa /netbsd: [   1.0000000]  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC

Mar 19 12:31:31 aa /netbsd: [   1.0000000] total memory = 448 MB

Mar 19 12:31:31 aa /netbsd: [   1.0000000] avail memory = 422 MB

The Other 686

Alas, my idea to install NetBSD on a homebrew AMD board hasn't worked yet due to video card matching and budget. FreeBSD skirted the "bad" board by least common denominator rules which kicked in vanilla VGA X/console window to at least boot. I had 9.x running and decided for the time being not to keep swapping out hardware to have yet another NetBSD 10 beta testbed. 

There are a few programs that I haven't got working to my satisfaction on the 6-core/12-thread machine, thus I'm using the NetBSD "laptop in a biggish box" as the main X display and putting databases or apps not needing video interaction on FreeBSD.

One program that really snaps with faster hardware is ocrmypdf; transactions that were taking a minute on an ARM processor kick over in 10 seconds now.

[     1.000003] cpu0: AMD Ryzen 5 5600X 6-Core Processor   

[     1.005359] amdzentemp0 at amdsmn0: AMD CPU Temperature Sensors (Family19h)

[     1.005359] amdzentemp0: autoconfiguration error: unable to register with sysmon (error=22)

Zabbix migrations, in place / out of place

I've had Zabbix running on PostgreSQL for over a year now it seems, and during that time Zabbix has released a couple major updates, and I've tried to build the newest servers and agents I could on a variety of BSD and Linux flavors. My go-to platform, NetBSD, doesn't have the latest and greatest of every app out there, so I moved on to FreeBSD as the primary Zabbix home base, er at home.

I've had 2 levels of the Zabbix server on NetBSD, once on i386 and once on Arm but for this post I'll ignore those branches and focus on 2 FreeBSD directions for data and applications with the history ramifications.

When I started Zabbix 5 server was the best I could manage; as I tried other variations I had 6.2 up and running eventually. I figure out that exporting and importing history/trends from one Zabbix box to another is simple but time consuming as valuable data feeds increase.

Platform 1: Upgrade database in place; conform to schema changes

In early 2022, I started monitoring with Zabbix en masse, as they say.

-rw-r--r--  1 root  wheel  20385599 Aug  8  2022 postgresql-11.17.tar.bz2

-rw-r--r--  1 root  wheel  22132996 Aug  8  2022 postgresql-14.5.tar.bz2

While upgrading, I needed both versions 11 and 14; FreeBSD would not install one if the other was already. Hence, skipping the "make clean" part meant slightly quicker re-installs.

Installing zabbix62-server-6.2.3...

pkg-static: zabbix62-server-6.2.3 conflicts with zabbix54-server-5.4.9 (installs files into the same place).

===>  zabbix62-frontend-php74-6.2.3 conflicts with installed package(s):

      zabbix54-frontend-php74-5.4.9_1

===>  Installing for zabbix62-server-6.2.3

===>  Checking if zabbix62-server is already installed

===>   zabbix62-server-6.2.3 is already installed

 Installed packages to be REMOVED:

        zabbix62-server: 6.2.3

Starting version (2022):

• Jan 30 21:58:37 freebsd1 pkg-static[69272]: postgresql11-server-11.14 installed

Upgrade phases (2023):

• Feb 20 15:46:38 freebsd1 pkg[11293]: postgresql11-server-11.17 deinstalled
• Feb 20 16:59:17 freebsd1 pkg-static[65464]: postgresql11-server-11.17 installed
• Feb 20 17:14:14 freebsd1 pkg[69391]: postgresql11-server-11.17 deinstalled
• Feb 20 17:15:10 freebsd1 pkg-static[69814]: postgresql14-server-14.5 installed

The "deinstalls" probably include the pre-packaged server version that only supports mysql/mariadb.

FAIL:

Configuration file error

DB type "POSTGRESQL" is not supported by current setup. Possible values MYSQL.

PHP

Feb  3 23:17:58 freebsd1 pkg-static[89086]: php80-pgsql-8.0.15 installed

Feb  3 23:57:55 freebsd1 pkg-static[15828]: php80-pgsql-8.0.15 deinstalled

Feb  4 02:51:20 freebsd1 pkg-static[52566]: php74-pgsql-7.4.27 installed

This sequence shows the installed package mix failed to include PHP 8.0; this limit was something I searched for in early installs. Zabbix didn't work with PHP 8 on the systems I could muster.

Data changes

Along the way, Zabbix changed history table keys so newer versions may not deal with older versions if there is redundancy (as I understand it). So history needed to be dumped and reloaded, being renamed along the way so that there was a fall back. If it worked.

  38636366 Jan 17 03:14 backup_zabbix_history_text.sql
 302137397 Jan 17 03:15 backup_zabbix_history_uint.sql

 174197724 Jan 17 03:16 backup_zabbix_trends.sql
 101438770 Jan 17 03:16 backup_zabbix_trends_uint.sql

=> select count(*) from history_old;

  count

---------

 9281308

Platform 2: New database, import hosts and templates, manually import history.

Database Server versions

(2022)

May 12 15:13:26 freebsd2 pkg[11749]: postgresql14-server-14.5 installed

May 12 15:49:32 freebsd2 pkg[14127]: postgresql14-server-14.5 deinstalled

May 12 16:13:38 freebsd2 pkg[15371]: postgresql14-server-14.5 installed

My notes aren't clear on why 2 tries, but at least that's still correct on the Zabbix application server side. I moved the database in 2023 to a different system meaning no need to upgrade the original host unless I need to do a refresh there sometime (mmm, backups).

Feb 18 23:17:41 freebsd3 pkg-static[33301]: postgresql15-server-15.1_1 installed

Feb 18 23:26:00 freebsd3 pkg[42270]: postgresql15-server reinstalled: 15.1_1 -> 15.1_1

Feb 18 23:40:46 freebsd3 pkg[45979]: postgresql15-server-15.1_1 deinstalled

Feb 18 23:47:06 freebsd3 pkg-static[64934]: postgresql15-server-15.1_1 installed

I struggled a bit here with getting the package/port to include PostgreSQL. If I slipped, the make had to be re-done with particular incantations to forget my previous mistake. So I won't do it again, it's [ make rmconfig ].

Hammer time

20230220:143137.337 Unable to start Zabbix server due to unsupported PostgreSQL database version (11.17).

20230220:143137.337 Must be at least (13.0).

20230220:143137.337 Use of supported database version is highly recommended.

20230220:143137.337 Override by setting AllowUnsupportedDBVersions=1 in Zabbix server configuration file at your own risk.

=# INSERT INTO history SELECT * FROM history_old ON CONFLICT (itemid,clock,ns) DO NOTHING;

INSERT 0 9281304

[...]

COPY 1357

COPY 3828

COPY 0

COPY 18

COPY 1

COPY 9311373

COPY 5024

COPY 0

COPY 0

=> select count(*) from history;

  count

---------

 9311373

(1 row)

=> select count(*) from history_old;

  count

---------

 9281308

PHP

Here, unlike the legacy system, the PHP 8 version is working with Zabbix.

(2022)

May 12 15:50:41 freebsd2 pkg[14127]: php82-pgsql-8.2.0.r2 installed

May 12 16:12:16 freebsd2 pkg[15371]: php82-pgsql-8.2.0.r2 deinstalled

(2023)

Jan 10 20:20:05 freebsd2 pkg-static[50148]: php81-pgsql-8.1.14 installed

Data migration

To have a faster response time, I used system 3 as a database server, and installed PostgreSQL 15 over the previously running version 14.

Basic export import steps follow, along with dump size.

1. Shutdown Zabbix application server
2. Run a database export
3. Set up target system at least the same database version
4. Create database and schema as needed
5. Run database import
6. Shutdown original database
7. Alter Zabbix server configuration
8. Viola (ha)

2023-02-19 06:25:02.997 UTC [9256] FATAL:  terminating connection due to administrator command

-rw-r--r--  1 postgres  postgres  562198532 Feb 19 06:01 /dump/zabbix_freebsd2.dump

This is the wholly grail; if I miss this screen or fail to check the right box, it's not show time, it's dump and reload time.

From then to now:

-rw-r--r--  1 freebsd  freebsd  24382685 Dec 23  2021 zabbix-5.4.9.tar.gz
-rw-r--r--  1 freebsd  freebsd  24510838 Jan 31  2022 zabbix-5.4.10.tar.gz
-rw-r--r--  1 freebsd  freebsd  41038757 Dec  5 08:44 zabbix-6.2.6.tar.gz

Meanwhile 6.2.7 and 6.2.8 are probably out, on some platforms at least.

History results (from platform 1):

The gaps are due to sensors or test systems going offline or elsewhere.

Deltas

What did I find most different after trying the upgraded platform and the fresh platform? One obvious nicety is the delivery of open street maps in the base server. This only shows up for me in the fresh install, though.

Next big difference is the change from a single agent status level to seeing two of them, squeezed into one lamp.

Perhaps multiple interfaces may be configured to show here, though my first tries to add second adapters didn't pan out.

Option: HeartbeatFrequency is available on later agent versions, so check if this needs to be set (missing means an earlier configuration file).

This dash display works simply enough:

Groups and other metadata containers/tags have changed somewhat from the earliest versions I've used (5).

Which way for future upgrades? It depends, as usual. If the keys change, that's doable. Larger data collections are unlikely at my pace; for others trends would need to be analyzed. Avoiding "database unsupported" is always a good idea.

Ephemera

Along the way, I found this bug (hit it myself):

ZBX_NOTSUPPORTED: Cannot obtain a descriptor to access kernel virtual memory.

[ https://www.zabbix.com/forum/zabbix-help/429220-zbx_notsupported-cannot-obtain-a-descriptor-to-access-kernel-virtual-memory ]

Although the question refers to NetBSD 8 and Zabbix 4 (both outdated in 2023), the error may still hit if you use an older agent software base. NetBSD patches to get an agent the correct system internal metrics are out there, maybe on package source work-in-progress, and maybe in a distribution near you. I went through each system I test to get a Zabbix 6.2.6 agent running, so they'd match the server version and not have quirks with features absent or deformed.

The FreeBSD kernel limited the import speed, I presume.

Feb 20 17:40:48 freebsd1 kernel: Limiting open port RST response from 223 to 200 packets/sec

Feb 20 17:40:50 freebsd1 kernel: Limiting open port RST response from 224 to 200 packets/sec

Feb 20 17:41:24 freebsd1 kernel: Limiting open port RST response from 220 to 200 packets/sec

References:

https://jspath55.blogspot.com/2023/01/another-zabbix-chapter-software-upgrade.html

https://jspath55.blogspot.com/2022/06/zabbix-server-build-trial-and-error.html

https://jspath55.blogspot.com/2022/02/building-out-intra-home-data-aggregator.html

NetBSD 10 Beta installs and testing from i386 to ARM

I've run NetBSD for quite some time, going back to the days of Dr. Dobbs magazine and the release of the BSD4-based 386BSD (which I installed from floppy disks...). The chance of beta testing the newest release came at the end of December 2022. I had installed NetBSD-current on Raspberry Pi platforms, and also had different levels of NetBSD 9 on systems.

This story is on my attempts to upgrade or install fresh operating systems "in house". Rather than strict chronological order, the sections below are by CPU architecture, starting with the older i386 then going to Arm processors. I have not upgraded an AMD-based system (yet), and will add notes at the end when I can.

i386

I had a running 32-bit small system where I've had different releases of NetBSD over the years, starting with spinning disk and now with solid state. After going through more hurdles than I expected, that "Atom" powered machine is on 10-Beta and being tested ("h1"). Then I remembered a donated PC that would support NetBSD, although I either fumbled the boot tracks or the BIOS is just not BSD-friendly. That machine ("h2") got a fresh install from the ISO image.

H1 BEFORE:

Jan  1 20:16:19 h1 /netbsd: [   1.0000000] NetBSD 9.3 (GENERIC) #0: Thu Aug  4 15:30:37 UTC 2022

H1 AFTER:

Jan  1 21:22:51 h1 /netbsd: [   1.0000000] NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 31 04:55:53 UTC 2022

NetBSD h1 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 31 04:55:53 UTC 2022  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386

H2 AFTER:

Jan 29 16:08:18 h2 /netbsd: [   1.0000000] NetBSD 10.0_BETA (GENERIC) #0: Mon Jan 23 16:02:49 UTC 2023

NetBSD h2 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Mon Jan 23 16:02:49 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386

The first install took longer than I wanted because I first tried to use a CD-ROM disk for the install image, but unfortunately the standard distribution size has now grown pas 640MB for i386. Then I installed to a clean external drive via a USB adapter before managing to lay down the correct OS image and all of the parts.

One self-imposed problem was, using sysupgrade as a novice, making the fstab file empty or wrong. Fortunately, somehow got around this using the other boot drive to repair the goof.

$ ls -trl fst*

-rw-r--r--  1 root  wheel   348 Jan  2 13:48 fstab-10-fail

-rw-r--r--  1 root  wheel  1100 Jan  2 14:05 fstab-10-almost

-rw-r--r--  1 root  wheel   953 Jan  2 14:18 fstab

About 3 weeks elapsed between the first i386 kernel build to the second, so some slight changes have occurred it seems.

-rwxr-xr-x  1 root  wheel  25611864 Dec 31 04:55 /netbsd

-rwxr-xr-x  1 root  wheel  25611940 Jan 23 16:02 /netbsd

Typically the first package I'd install not in the base distribution is the bash shell, requiring steps to initiate a package source/executable tree. To get up-to-date, I refreshed pkgsrc via CVS on both; the first box has almost 2000 top-level distfile items and the second just 100 so far. For more complex packages with extensive dependencies I've ended up with pre-built packages generally. More below.

 

Entropy

Jan 11 18:03:44 h1 /netbsd: [ 176217.1118082] entropy: pid 19289 (python) blocking due to lack of entropy

Stuck on this for a while until I noticed the screen message. Repaired. Didn't happen on the second, newer, machine, with a quite different CPU (Intel Atom versus AMD A8).

Packages

The most glaring omissions below the first tier (64-bit) systems on NetBSD I've run into are web browsers, audio software, and the LibreOffice package. Thunderbird is another with a massive compile time is you do it yourself.

Zabbix 6.2 server and agents weren't there in pkgsrc and I've installed them from source across the NetBSD architectures I have. Most common hurdle has been the Perl regular expression library pcre which seems like it should have trivial build or run requirements (i.e. ldd works).

The net.c file needed to be pulled in from the pkgsrc "work in progress"

$ ls -l  zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c*

-rw-r--r--  1 me  us  9245 Jan 27 00:57 zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c

-rw-r--r--  1 me  us  8129 Dec  1 07:47 zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c.orig

Alas, pkgsrc.se went away quite recently, so I cant backtrack to the source; maybe this will help:

https://mail-index.netbsd.org/pkgsrc-wip-changes/2023/02/04/msg026289.html

"zabbix62-*: Update Zabbix to 6.2.7"

Oh, right, this:

https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=tree;f=zabbix62-agent;hb=HEAD

lynx sometimes shows

SSL error:unable to get local issuer certificate-Continue? (n) 

then works:

    Linkname:LYNX - The Text Web-Browser                                                       URL:https://lynx.invisible-island.net/

     Charset:us-ascii                                                                      Server:Sucuri/Cloudproxy                                                                Date:Sat, 04 Feb 2023 14:37:40 GMT                                                Last Mod:Wed, 03 Apr 2019 08:25:24 GMT 

Firefox loads and runs.

Thunderbird doesn't like 32-bit on my attempts.

no such installed package thunderbird-78.12.0nb9

For fill-ins of lost regexp parts:

https://rpmfind.net/linux/rpm2html/search.php?query=pkgconfig(libpcre)

Benchmarks

I should have run times from a lot of historic tests, except those under NDA (nondisclosure agreements) or other reasons. The BYTE benchmark is one of the oldest, and I probably ran "sieve" tests in the 1980s for C compilers.

The package source runs the older 4.x version (5.x is in the FreeBSD ports tree).

$ ls -l /var/bytebench/report 

-rw-r--r--  1 root  wheel  3792 Jan 31 14:28 /var/bytebench/report

Dhrystones: 2,285,582 h1

            4,820,219 h2

AMD:

           64,565,735 (ryzen 6-core)

Old times, good times:

Dhrystones:   570,441 (pentium ~2005)

            3,437,974 (pentium ~2005)

            3,484,307 (pentium ~2007)

            7,126,944 (opteron ~2007)

Benchmark Run: Sat Jun 26 2010 20:06:15 - 20:06:15

192 CPUs in system; running 192 parallel copies of tests

Even older:

-rwxr--r--  1 377  wheel   63570 Mar 11  1997 DHRY1ND.EXE

-rwxr--r--  1 377  wheel   59570 Mar 11  1997 DHRY1OD.EXE

-rwxr--r--  1 377  wheel   67466 Mar 11  1997 DHRY2ND.EXE

-rwxr--r--  1 377  wheel   67598 Mar 11  1997 DHRY2OD.EXE

hbench

I began collecting results from the hbench-os package because I've used it in the past and had rough ideas what values might be expected on high-end processors. Lower end modules like the Pi Zero 2W show degradation in the results.

I snagged on remote OS calls in the run sequence, and commented them out in a rough way.

$ cat  /usr/pkg/share/hbench/Results/netbsdelf10.0-i386/h1.1/lat_syscall_sbrk

0.0259

0.0257

0.0275

I guess Byte bench should write to /usr/pkg instead of /var.

$ cat  /usr/pkg/share/hbench/Results/netbsdelf10.0-i386/h2.1/lat_syscall_sbrk

0.0160

0.0157

0.0156

0.0148

0.0160

(the 9.x amd system shows):

$ cat  /usr/pkg/share/hbench/Results/netbsd9.2-x86_64/amd.3/*sbrk

0.0024

0.0024

0.0024

test suite

For whatever reason I had not previously looked at the /usr/tests suite of system tests that have been included in NetBSD for far longer than I would have guessed. I can't find a lot of cross references so just dug into the docs and tried things out. Many tests failed when /usr/sbin or /sbin wasn't in the PATH, and a few others I have checked work if logged in as root. Though I don't have enough experience with this suite, getting values from all the systems I could access made sense in trying to help isolate either test or system faults.

i386

Failed test cases:

    include/t_paths:paths, kernel/kqueue/t_empty:sock_tcp, 

    kernel/t_magic_symlinks:realpath, 

spurious artifacts

[ 104957.413584] WARNING: pid 19263 (t_futex_robust) lwp 7568: exhausted robust futex limit

ARM

NetBSD a1 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC evbarm

NetBSD a2 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC evbarm

NetBSD a3 10.0_BETA NetBSD 10.0_BETA (GENERIC64) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC64 evbarm

Like one of the i386 upgrades, I made errors using sysupgrade, with the worst problem being removal of necessary security files. When normal groups came up as "not found" I began to worry. With some backups, the majority of issues were fixed eventually. The workaround was again, mounting the flawed install elsewhere and updating the gaps.

One fix didn't catch immediately; setting the last touched time to "now" did the right thing.

(not supposed to be zero:)

-rw-------  1 root  wheel     0 Jan 17 16:38 

Clean install on a Pi Zero 2W

One oddity I found was date stamps on the upgrade sets. On the website the dates were nearly identical but the cached versions differ.

$ ls -l /var/cache/sysupgrade/

-rw-r--r-- 1 root wheel 37548220 Jan 16 10:22 base.tar.xz
-rw-r--r-- 1 root wheel 54971904 Jan 16 10:26 comp.tar.xz
-rw-r--r-- 1 root wheel   447912 Dec 20 00:56 dtb.tar.xz
-rw-r--r-- 1 root wheel   495604 Dec 23 00:04 etc.tar.xz
-rw-r--r-- 1 root wheel  2614024 Jan 16 10:20 games.tar.xz
-rw-r--r-- 1 root wheel  1103864 Jan 14 22:43 gpufw.tar.xz
-rw-r--r-- 1 root wheel  7566312 Jan 16 10:21 man.tar.xz
-rw-r--r-- 1 root wheel  4135072 Jan 16 10:20 misc.tar.xz
-rw-r--r-- 1 root wheel  6185508 Jan 16 10:20 modules.tar.xz
-rw-r--r-- 1 root wheel  7852529 Jan 14 22:37 netbsd-GENERIC64.gz
-rw-r--r-- 1 root wheel  2694436 Jan 16 10:20 rescue.tar.xz
-rw-r--r-- 1 root wheel 10290016 Jan 16 10:20 tests.tar.xz
-rw-r--r-- 1 root wheel  1900648 Jan 16 10:20 text.tar.xz
-rw-r--r-- 1 root wheel  6021116 Dec 31 21:03 xbase.tar.xz
-rw-r--r-- 1 root wheel  6026656 Jan 14 22:44 xcomp.tar.xz
-rw-r--r-- 1 root wheel    27840 Jan  4 15:11 xetc.tar.xz
-rw-r--r-- 1 root wheel 28997008 Jan 16 10:21 xfont.tar.xz
-rw-r--r-- 1 root wheel 21325496 Jan 16 10:21 xserver.tar.xz

Packages

Firefox isn't entirely working for me on the arm systems. Still checking options.

LibreOffice, PostgreSQL and Thunderbird work in all tests so far. 

Thunderbird email header:

User-Agent: Mozilla/5.0 (X11; NetBSD evbarm; rv:78.0) Gecko/20100101 Thunderbird/78.12.0

Zabbix agents are at 6.2 with local compiles. I had a Zabbix server working on NetBSD current which is not set up at the moment.

catclock: yes

One of the cooler packages to get running is minidlna, which can share out media from MP3s to MP4s. Well, that's not many types. Probably M4A and WAV and more too. FreeBSD shines here as well.

[2023/01/13 04:28:01] minidlna.c:1121: warn: Starting MiniDLNA version 1.2.1.

VLC

pkgin search vlc
cleaning database from https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/9.0/All entries...
reading local summary...
processing local summary...
processing remote summary (https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/10.0/All)...

pkgin search vlc-3

vlc-3.0.17.4nb8 <    VideoLAN media player and streaming server

=: package is installed and up-to-date

<: package is installed but newer version is available

>: installed package has a greater version than available package

ctwm

I made local "dot" ctwm rc files after seeing the NetBSD X standard change. If I use the system defaults with no local dot file, I get application menus from pkgsrc. But with my local tweaks I can't find a way to run that menu. The CTWM app doesn't make a lot of noise, which is generally good.

Benchmarks

Pi0

Dhrystones: 2087553 a1

Dhrystones: 2207961 a2

Pi4

Dhrystones: 8777214 a3

hbench

sbrk pi 4

0.4160

0.4468

0.4161

0.4132

pi0:

pkgin search hbench

No results found for hbench

Still trying to run this particular flavor

https://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/benchmarks/hbench/index.html has no 32-bit arm/evbarm when I checked.

bash-5.1$ file  bin/netbsd10.0-evbarm/lat_pipe

bin/netbsd10.0-evbarm/lat_pipe: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 10.0, compiled for: earmv7hf, not stripped

These will make more sense with legends. The old way I created these was with ploticus, which meanwhile has evolved such that my old configurations do not work.

test suite

Failed test cases:

    include/t_paths:paths, kernel/kqueue/t_empty:sock_tcp,

    kernel/t_magic_symlinks:realpath,

[...]

Network adapter failure(s)

I opened a PR after getting repeated reboots when using an external ethernet adapter with a Pi 02W. After several days/weeks, I installed a second system. That had no problems with a second adapter. Then I switched the hardware, and surprise, no problems. Oof.

Jan 16 09:06:50 a1 /netbsd: [ 12626.1912013] Skipping crash dump on recursive

 panic

Jan 16 09:06:50 a1 /netbsd: [ 12626.1912013] panic: lock error: Mutex: mutex_

vector_enter,516: assertion failed: !cpu_intr_p(): lock 0x90d7087c cpu 0 lwp 0x9

0af8040

(last one seen as of 04-Feb-2023)

messages:

Jan 16 05:37:46 a1 /netbsd: [   3.9488398] ure0: Realtek (0x0bda) USB 10/100/1000 LAN (0x8153), rev 2.10/31.00, addr 4

Jan 27 20:16:13 a1 /netbsd: [   2.9264624] ure0: Realtek (0x0bda) USB 10/100 LAN (0x8152), rev 2.10/20.00, addr 3

End notes

In getting ancient "Byte Benches" to run, I found a few obstacles and retreaded some old territory. In reviewing the test conditions, I realized only the Linux variants on the Pi4 were modifying the CPU clock dynamically. NetBSD had the clock set to 1500 MHz, and FreeBSD 600 MHz.

Also wondering which tests used all 4 cores, given the evolved benchmarks that run 4 parallel tests when 4 cores are detected.

Found the kernel control for CPU clock throttling on FreeBSD with powerd, while NetBSD has a package called estd. I have both running, though the FB activity seems to be a see-saw from high to low, while the NB box just keeps running at top speed. In future tests I will add more load and also watch the CPU temperatures (only the Raspbian box has no fan right now).

I opened problem reports for kernel crashes and the entropy hangs. The other issue, if I can document it, is the audio faults after a period of time. It's great to have music work out the headphone jack, as well as HDMI, as earlier builds of non-Raspberry OS had issues.

AMD64 - future path to upgrade will either be a live USB or a CD-ROM. Probably I'll test with the latter first since the i386 image outgrew ye olde 640MB disk size.