Wednesday, February 8, 2023

NetBSD 10 Beta installs and testing from i386 to ARM

I've run NetBSD for quite some time, going back to the days of Dr. Dobbs magazine and the release of the BSD4-based 386BSD (which I installed from floppy disks...). The chance of beta testing the newest release came at the end of December 2022. I had installed NetBSD-current on Raspberry Pi platforms, and also had different levels of NetBSD 9 on systems.

This story is on my attempts to upgrade or install fresh operating systems "in house". Rather than strict chronological order, the sections below are by CPU architecture, starting with the older i386 then going to Arm processors. I have not upgraded an AMD-based system (yet), and will add notes at the end when I can.

i386

I had a running 32-bit small system where I've had different releases of NetBSD over the years, starting with spinning disk and now with solid state. After going through more hurdles than I expected, that "Atom" powered machine is on 10-Beta and being tested ("h1"). Then I remembered a donated PC that would support NetBSD, although I either fumbled the boot tracks or the BIOS is just not BSD-friendly. That machine ("h2") got a fresh install from the ISO image.

H1 BEFORE:

Jan  1 20:16:19 h1 /netbsd: [   1.0000000] NetBSD 9.3 (GENERIC) #0: Thu Aug  4 15:30:37 UTC 2022

H1 AFTER:

Jan  1 21:22:51 h1 /netbsd: [   1.0000000] NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 31 04:55:53 UTC 2022

NetBSD h1 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 31 04:55:53 UTC 2022  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386

H2 AFTER:

Jan 29 16:08:18 h2 /netbsd: [   1.0000000] NetBSD 10.0_BETA (GENERIC) #0: Mon Jan 23 16:02:49 UTC 2023

NetBSD h2 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Mon Jan 23 16:02:49 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/i386/compile/GENERIC i386

The first install took longer than I wanted because I first tried to use a CD-ROM disk for the install image, but unfortunately the standard distribution size has now grown pas 640MB for i386. Then I installed to a clean external drive via a USB adapter before managing to lay down the correct OS image and all of the parts.

One self-imposed problem was, using sysupgrade as a novice, making the fstab file empty or wrong. Fortunately, somehow got around this using the other boot drive to repair the goof.

$ ls -trl fst*
-rw-r--r--  1 root  wheel   348 Jan  2 13:48 fstab-10-fail
-rw-r--r--  1 root  wheel  1100 Jan  2 14:05 fstab-10-almost
-rw-r--r--  1 root  wheel   953 Jan  2 14:18 fstab

About 3 weeks elapsed between the first i386 kernel build to the second, so some slight changes have occurred it seems.

-rwxr-xr-x  1 root  wheel  25611864 Dec 31 04:55 /netbsd
-rwxr-xr-x  1 root  wheel  25611940 Jan 23 16:02 /netbsd


Typically the first package I'd install not in the base distribution is the bash shell, requiring steps to initiate a package source/executable tree. To get up-to-date, I refreshed pkgsrc via CVS on both; the first box has almost 2000 top-level distfile items and the second just 100 so far. For more complex packages with extensive dependencies I've ended up with pre-built packages generally. More below.
 

Entropy


Jan 11 18:03:44 h1 /netbsd: [ 176217.1118082] entropy: pid 19289 (python) blocking due to lack of entropy

Stuck on this for a while until I noticed the screen message. Repaired. Didn't happen on the second, newer, machine, with a quite different CPU (Intel Atom versus AMD A8).

Packages


The most glaring omissions below the first tier (64-bit) systems on NetBSD I've run into are web browsers, audio software, and the LibreOffice package. Thunderbird is another with a massive compile time is you do it yourself.

Zabbix 6.2 server and agents weren't there in pkgsrc and I've installed them from source across the NetBSD architectures I have. Most common hurdle has been the Perl regular expression library pcre which seems like it should have trivial build or run requirements (i.e. ldd works).

The net.c file needed to be pulled in from the pkgsrc "work in progress"

$ ls -l  zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c*
-rw-r--r--  1 me  us  9245 Jan 27 00:57 zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c
-rw-r--r--  1 me  us  8129 Dec  1 07:47 zabbix-6.2.6/src/libs/zbxsysinfo/netbsd/net.c.orig

Alas, pkgsrc.se went away quite recently, so I cant backtrack to the source; maybe this will help:


"zabbix62-*: Update Zabbix to 6.2.7"

Oh, right, this:


lynx sometimes shows

SSL error:unable to get local issuer certificate-Continue? (n) 

then works:

    Linkname:LYNX - The Text Web-Browser                                                       URL:https://lynx.invisible-island.net/
     Charset:us-ascii                                                                      Server:Sucuri/Cloudproxy                                                                Date:Sat, 04 Feb 2023 14:37:40 GMT                                                Last Mod:Wed, 03 Apr 2019 08:25:24 GMT 

Firefox loads and runs.

Thunderbird doesn't like 32-bit on my attempts.
no such installed package thunderbird-78.12.0nb9

For fill-ins of lost regexp parts:

Benchmarks


I should have run times from a lot of historic tests, except those under NDA (nondisclosure agreements) or other reasons. The BYTE benchmark is one of the oldest, and I probably ran "sieve" tests in the 1980s for C compilers.
The package source runs the older 4.x version (5.x is in the FreeBSD ports tree).

$ ls -l /var/bytebench/report 
-rw-r--r--  1 root  wheel  3792 Jan 31 14:28 /var/bytebench/report

Dhrystones: 2,285,582 h1
            4,820,219 h2

AMD:
           64,565,735 (ryzen 6-core)

Old times, good times:

Dhrystones:   570,441 (pentium ~2005)
            3,437,974 (pentium ~2005)
            3,484,307 (pentium ~2007)
            7,126,944 (opteron ~2007)


Benchmark Run: Sat Jun 26 2010 20:06:15 - 20:06:15
192 CPUs in system; running 192 parallel copies of tests

Even older:

-rwxr--r--  1 377  wheel   63570 Mar 11  1997 DHRY1ND.EXE
-rwxr--r--  1 377  wheel   59570 Mar 11  1997 DHRY1OD.EXE
-rwxr--r--  1 377  wheel   67466 Mar 11  1997 DHRY2ND.EXE
-rwxr--r--  1 377  wheel   67598 Mar 11  1997 DHRY2OD.EXE

hbench


I began collecting results from the hbench-os package because I've used it in the past and had rough ideas what values might be expected on high-end processors. Lower end modules like the Pi Zero 2W show degradation in the results.

I snagged on remote OS calls in the run sequence, and commented them out in a rough way.

$ cat  /usr/pkg/share/hbench/Results/netbsdelf10.0-i386/h1.1/lat_syscall_sbrk
0.0259
0.0257
0.0275

I guess Byte bench should write to /usr/pkg instead of /var.

$ cat  /usr/pkg/share/hbench/Results/netbsdelf10.0-i386/h2.1/lat_syscall_sbrk
0.0160
0.0157
0.0156
0.0148
0.0160

(the 9.x amd system shows):

$ cat  /usr/pkg/share/hbench/Results/netbsd9.2-x86_64/amd.3/*sbrk
0.0024
0.0024
0.0024

test suite

For whatever reason I had not previously looked at the /usr/tests suite of system tests that have been included in NetBSD for far longer than I would have guessed. I can't find a lot of cross references so just dug into the docs and tried things out. Many tests failed when /usr/sbin or /sbin wasn't in the PATH, and a few others I have checked work if logged in as root. Though I don't have enough experience with this suite, getting values from all the systems I could access made sense in trying to help isolate either test or system faults.

i386
Failed test cases:
    include/t_paths:paths, kernel/kqueue/t_empty:sock_tcp, 
    kernel/t_magic_symlinks:realpath, 

spurious artifacts

[ 104957.413584] WARNING: pid 19263 (t_futex_robust) lwp 7568: exhausted robust futex limit


ARM

NetBSD a1 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC evbarm

NetBSD a2 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC evbarm

NetBSD a3 10.0_BETA NetBSD 10.0_BETA (GENERIC64) #0: Fri Jan 13 19:15:32 UTC 2023  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC64 evbarm

Like one of the i386 upgrades, I made errors using sysupgrade, with the worst problem being removal of necessary security files. When normal groups came up as "not found" I began to worry. With some backups, the majority of issues were fixed eventually. The workaround was again, mounting the flawed install elsewhere and updating the gaps.


One fix didn't catch immediately; setting the last touched time to "now" did the right thing.

(not supposed to be zero:)
-rw-------  1 root  wheel     0 Jan 17 16:38 

Clean install on a Pi Zero 2W






One oddity I found was date stamps on the upgrade sets. On the website the dates were nearly identical but the cached versions differ.

$ ls -l /var/cache/sysupgrade/

-rw-r--r-- 1 root wheel 37548220 Jan 16 10:22 base.tar.xz
-rw-r--r-- 1 root wheel 54971904 Jan 16 10:26 comp.tar.xz
-rw-r--r-- 1 root wheel   447912 Dec 20 00:56 dtb.tar.xz
-rw-r--r-- 1 root wheel   495604 Dec 23 00:04 etc.tar.xz
-rw-r--r-- 1 root wheel  2614024 Jan 16 10:20 games.tar.xz
-rw-r--r-- 1 root wheel  1103864 Jan 14 22:43 gpufw.tar.xz
-rw-r--r-- 1 root wheel  7566312 Jan 16 10:21 man.tar.xz
-rw-r--r-- 1 root wheel  4135072 Jan 16 10:20 misc.tar.xz
-rw-r--r-- 1 root wheel  6185508 Jan 16 10:20 modules.tar.xz
-rw-r--r-- 1 root wheel  7852529 Jan 14 22:37 netbsd-GENERIC64.gz
-rw-r--r-- 1 root wheel  2694436 Jan 16 10:20 rescue.tar.xz
-rw-r--r-- 1 root wheel 10290016 Jan 16 10:20 tests.tar.xz
-rw-r--r-- 1 root wheel  1900648 Jan 16 10:20 text.tar.xz
-rw-r--r-- 1 root wheel  6021116 Dec 31 21:03 xbase.tar.xz
-rw-r--r-- 1 root wheel  6026656 Jan 14 22:44 xcomp.tar.xz
-rw-r--r-- 1 root wheel    27840 Jan  4 15:11 xetc.tar.xz
-rw-r--r-- 1 root wheel 28997008 Jan 16 10:21 xfont.tar.xz
-rw-r--r-- 1 root wheel 21325496 Jan 16 10:21 xserver.tar.xz



Packages

Firefox isn't entirely working for me on the arm systems. Still checking options.
LibreOffice, PostgreSQL and Thunderbird work in all tests so far. 

Thunderbird email header:

User-Agent: Mozilla/5.0 (X11; NetBSD evbarm; rv:78.0) Gecko/20100101 Thunderbird/78.12.0

Zabbix agents are at 6.2 with local compiles. I had a Zabbix server working on NetBSD current which is not set up at the moment.

catclock: yes

One of the cooler packages to get running is minidlna, which can share out media from MP3s to MP4s. Well, that's not many types. Probably M4A and WAV and more too. FreeBSD shines here as well.

[2023/01/13 04:28:01] minidlna.c:1121: warn: Starting MiniDLNA version 1.2.1.

VLC

pkgin search vlc
cleaning database from https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/9.0/All entries...
reading local summary...
processing local summary...
processing remote summary (https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/aarch64/10.0/All)...

pkgin search vlc-3
vlc-3.0.17.4nb8 <    VideoLAN media player and streaming server

=: package is installed and up-to-date
<: package is installed but newer version is available
>: installed package has a greater version than available package

ctwm

I made local "dot" ctwm rc files after seeing the NetBSD X standard change. If I use the system defaults with no local dot file, I get application menus from pkgsrc. But with my local tweaks I can't find a way to run that menu. The CTWM app doesn't make a lot of noise, which is generally good.


Benchmarks


Pi0
Dhrystones: 2087553 a1
Dhrystones: 2207961 a2
Pi4
Dhrystones: 8777214 a3



hbench


sbrk pi 4
0.4160
0.4468
0.4161
0.4132


pi0:
pkgin search hbench
No results found for hbench

Still trying to run this particular flavor


bash-5.1$ file  bin/netbsd10.0-evbarm/lat_pipe
bin/netbsd10.0-evbarm/lat_pipe: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 10.0, compiled for: earmv7hf, not stripped

These will make more sense with legends. The old way I created these was with ploticus, which meanwhile has evolved such that my old configurations do not work.

test suite


Failed test cases:
    include/t_paths:paths, kernel/kqueue/t_empty:sock_tcp,
    kernel/t_magic_symlinks:realpath,
[...]



Network adapter failure(s)

I opened a PR after getting repeated reboots when using an external ethernet adapter with a Pi 02W. After several days/weeks, I installed a second system. That had no problems with a second adapter. Then I switched the hardware, and surprise, no problems. Oof.


Jan 16 09:06:50 a1 /netbsd: [ 12626.1912013] Skipping crash dump on recursive
 panic
Jan 16 09:06:50 a1 /netbsd: [ 12626.1912013] panic: lock error: Mutex: mutex_
vector_enter,516: assertion failed: !cpu_intr_p(): lock 0x90d7087c cpu 0 lwp 0x9
0af8040

(last one seen as of 04-Feb-2023)


messages:
Jan 16 05:37:46 a1 /netbsd: [   3.9488398] ure0: Realtek (0x0bda) USB 10/100/1000 LAN (0x8153), rev 2.10/31.00, addr 4
Jan 27 20:16:13 a1 /netbsd: [   2.9264624] ure0: Realtek (0x0bda) USB 10/100 LAN (0x8152), rev 2.10/20.00, addr 3





End notes


In getting ancient "Byte Benches" to run, I found a few obstacles and retreaded some old territory. In reviewing the test conditions, I realized only the Linux variants on the Pi4 were modifying the CPU clock dynamically. NetBSD had the clock set to 1500 MHz, and FreeBSD 600 MHz.

Also wondering which tests used all 4 cores, given the evolved benchmarks that run 4 parallel tests when 4 cores are detected.






Found the kernel control for CPU clock throttling on FreeBSD with powerd, while NetBSD has a package called estd. I have both running, though the FB activity seems to be a see-saw from high to low, while the NB box just keeps running at top speed. In future tests I will add more load and also watch the CPU temperatures (only the Raspbian box has no fan right now).

I opened problem reports for kernel crashes and the entropy hangs. The other issue, if I can document it, is the audio faults after a period of time. It's great to have music work out the headphone jack, as well as HDMI, as earlier builds of non-Raspberry OS had issues.

AMD64 - future path to upgrade will either be a live USB or a CD-ROM. Probably I'll test with the latter first since the i386 image outgrew ye olde 640MB disk size.



No comments: