Saturday, January 21, 2023

Another Zabbix chapter: software upgrade, hardware downgrade.

I started using Zabbix about a year ago, trying to get it running on the Raspberry Pi "platform." I wanted to deploy a different OS than the standard Raspbian, and went through tests on NetBSD, FreeBSD, and another Linux flavor before settling on FreeBSD and Zabbix 5.4.

Building out an intra-home data aggregator

After running that system for some time, I decided a second system would be nice, for redundancy and for some development in parallel. I built the most recent version that I could find on the NetBSD package source, ending up with Zabbix 4 on a Pi 4 and 5.0 on an i386 (not really Pi). Minor version differences such as XML file content for templates and hosts were small enough to ignore or workaround.

Then I found an available Raspberry Pi 3A, causing a domino effect where I ended up putting FreeBSD 13.1 on a Pi 3B (the working 5.2 is on a Pi 4).

OS, database, application deploy

  • FreeBSD 13.1 ELF 64-bit LSB executable, ARM aarch64
  • Postgres 14.5 (I missed 15.x by 1 or 2 dependent patch levels)
  • Zabbix 6.2.6

The first thing I tried, even though it hasn't worked in the past, was to use only FreeBSD packages, not compile anything. I was hoping for PostgreSQL support on the Zabbix server rather than MySQL as I can do either but prefer the former (for reasons).

pkg install zabbix62-frontend-php81
pkg install mod_php81

When I started, indications were Zabbix 6.2.3 was the packaged version, which supported PostgreSQL server 14 but not 15. So I deployed version 14. As it turned out, I needed to compile a newer version which should work with 15. I didn't feel like dropping and reloading the data yet; maybe a later upgrade will be feasible.

Traces from /var/log/messages of my package journey:

DateTimePackage information
May 1211:01:06zabbix5-agent-5.0.28 installed
May 1215:19:38zabbix62-frontend-php81-6.2.3 installed
May 1214:55:37zabbix62-server-6.2.3 installed
May 1215:49:31zabbix62-frontend-php81-6.2.3 deinstalled
May 1215:50:41php82-pgsql-8.2.0.r2 installed
May 1215:52:34mod_php82-8.2.0.r2_1 installed
Jan 1016:12:16php82-pgsql-8.2.0.r2 deinstalled
Jan 1003:29:27zabbix62-frontend-php81-6.2.3 installed
Jan 1003:48:51mod_php82-8.2.0.r2_1 deinstalled
Jan 1003:49:25mod_php81-8.1.12 installed
Jan 1018:48:59zabbix62-server-6.2.3 deinstalled
Jan 1018:49:39zabbix62-server-6.2.6 installed
Jan 1019:19:52zabbix62-frontend-php81-6.2.6 installed
Jan 1020:20:05php81-pgsql-8.1.14 installed

The date flip occurred once I noticed the ntp daemon was not running so the system date was from the image build time. Besides Zabbix server not supporting PGSQL as packaged, the PHP version needed to be adjusted to match the workable Zabbix and database parts. Hence the PHP 8.2 install then deinstall, for the httpd "mod" as well as the PHP<>PGSQL shim.

Too bad about the downgrade from PHP 8.2 to 8.1. On the plus side, earlier Zabbix servers were incompatible with PHP 8, and any system where PHP 7 was unavailable meant Zabbix was blocked.

Screen shot of the critical database choice moment: 

Screenshot of Zabbix install, with PostgreSQL highlighted. Other databases not highlighted.

Voila (I particularly like the new built-in map feature)


Out of the box, I already had a local agent (v5) running on the server, but it did not connect cleanly to the server. Minor adjustments and then the "Zabbix server values per second" went from under 1 to around 10, which is what I've seen in a home grid with roughly a dozen nodes.

The primary imports from running Zabbix systems included templates and hosts, in the required dependency order. I'd already set up shared folders with prior configuration migrations among other systems, making this exercise more familiar thus quicker.

Probably an opportunity for some refactoring.

The Zabbix server emitted messages warning about poller usage, indicating tuning needed on process counts in the zabbix_server.conf file.

> # Wed Jan 11 13:00:29 UTC 2023
> # 1:
> # StartPollers=10
> # 2:
> StartPollers=20
> # Wed Jan 11 13:00:55 UTC 2023
> StartPollersUnreachable=2

I bumped StartPollers first to 10, and then to 20, which at least made the messages stop.


pi slices

The message "mmcblk0: Disk read/write request responses are too high" is back. I found that on earlier installs as a result of running on SD cards rather than spinning disk or SSD. Funny enough, when I looked for the fix, I found my earlier post with the suggestion of where these macros are found.
A "CPU is throttling" message is also back on the Pi 3 (no fan); not yet on a Pi 4 (has fan). I researched this error message and believe it is misleading. The code "0x80000" from what I've read means the CPU had been throttled but is not now. I think I have 3 choices: (1) ignore this error; (2) correct the code; (3) add a fan to the Pi.

Red light/green light. Or grey light/green light.


Above, a 3 piece snapshot of the Zabbix dashboard after loading a dozen or so nodes. A primary devolution is that host connections do not show as green except where there are no active checks. In that case, the "ZBX" icon is grey; hovering over it produces a pop-up with two status boxes. Active checks shows status "Unknown" and the host check shows "Available". Which is less helpful than earlier versions that simply showed green or red for connected or not.

The funny part here is that highlight the 3 letters triggered goog to display completely unrelated search results on screen.

One dollar gets you two dollars.

"Error $2 on $1"

Known error, "The $1, $2 macros were deprecated,"

Slower hardware (the downgrade)

"slow query"

slow query: 3.138252 sec, 
"delete from history where itemid=10073 and clock<1673580394"

The first working Zabbix server I set up that stayed running was on FreeBSD using a Raspberry Pi 4 with 4GB RAM. This build is on a Pi 3B with 1GB. The former runs of an SSD while the latter has a micro-SD and a USB-3 for the database files. The 3 only has USB-2 speed.
My user experience is that the new platform feels slower but not maddeningly. If I can avoid adding a plugin interface and drive that would be preferable for look and feel.



Would like to do more with the supplied features like dashboards, and new features to be discovered. This view of "room light" shows we're in winter with the short days. A health aide to avoid seasonal disorders perhaps?

history data moves

I've cobbled a Perl script to read from one Zabbix server and insert trending for one item into a second server, using a control table of "from" and "to" record IDs. Not bullet-proof but efficient enough even with row level commits to be a breeze. 

benchmark - Two Pi Zeros running NetBSD 10 beta:

There is one visible gap on the node with the troublesome USB to ethernet connectors.

The spike is from an install; working on setting up parallel benchmark tests on these for comparisons.

The bad ethernet behavior is noticeable as gaps in the red line...