Saturday, September 28, 2024

Flipping Zabbix from NetBSD to FreeBSD

 I wanted to upgrade my Zabbix server running on a NetBSD amd64 host and at almost the same time install NetBSD on an NVME card instead of SSD. The mainboard has no graphics, so I played hit-or-miss with cheap-ish cards, getting NetBSD 9 to run but then not, and using FreeBSD under low-res VGA mode. Anyway, to avoid missing data in case the reinstall had surprises I decided to clone the running Zabbix system to a FreeBSD host on a recycled laptop.

NetBSD pkgsrc has Zabbix 6.0, which I'm running, with 6.4 in the work-in-progress prep area, that I have tested but not deployed. FreeBSD has 6.4 in ports, with the modules I've used so the copy would also test the auto-upgrade.

Export and import with PostgreSQL was quick and tidy once I set up the target with an empty database and the necessary glue. The export file was just over 1GB, and imported with no (visible) errors. Starting up the 6.4 server triggered upgrade steps, also showing no noticeable gaffes.

 75396:20240925:142052.283 hosts_name_upper_update trigger for table "hosts" already exists, skipping patch of adding "name_upper" column to "hosts" table


 75396:20240925:142052.877 completed 99% of database upgrade

 75396:20240925:142052.890 completed 100% of database upgrade

 75396:20240925:142052.891 database upgrade fully completed

 75406:20240925:142053.025 starting HA manager

 75406:20240925:142053.056 HA manager started in active mode

Then a bunch of agent connection errors since I was going to configure them after rather than before. Next, install of the front end pieces, and http server, and the PHP bits and pieces to access the console. Happily no unsurmountable hurdles, made easier by having a running system to compare against.

IMAGE 1


Yay, history moved over intact. The dashboard shown was defined to show NetBSD Rasberry Pi wi-fi and that definition also survived the transfer.

IMAGE 2



The down-time, which wasn't optimized as I was not in a hurry. A few hours max. 

IMAGE 3


Because this was a new server, the out-of-the box poller configuration needed a little tweaking, handled over the shakedown phase as agent loads increased.

IMAGE 4


The first unexpected result was having empty Zabbix Server dashboard panels. There should be charts on each of those sub-frames, and also a log of messages near the top.

IMAGE 5

The prior widget definitions had pointed to "Zabbix Server" or a specific hostname. A definition distinction I didn't fathom until forced to repair the above. Doing a web search for terms "zabbix", "server" and "dashboard" was a time-waster given their generality.


The other built-in dashboard revealed the source hostname instead of the target. I scratched my head, read the docs, and left un-illuminated. I forced a fix by updating the various panels to point to the new server, knowing that the next move will trap me the same way.

IMAGE 6


Once I have set up a valid "Zabbix Server" target I can go back to the red loading zone and re-patch.

IMAGE 7

Once the gap was ID'd I set up a couple then completed the remaining sub-panels.



Initially, a valid working alert came out, leading to a side-bar investigation of kernel process limits.

IMAGE 8


Zabbix predefines macros that appear tilted to modern system eyes.

IMAGE 9



Colors and numbers and pop-ups oh my! The next question is which NVME board for NetBSD 10?