I've had Zabbix running on PostgreSQL for over a year now it seems, and during that time Zabbix has released a couple major updates, and I've tried to build the newest servers and agents I could on a variety of BSD and Linux flavors. My go-to platform, NetBSD, doesn't have the latest and greatest of every app out there, so I moved on to FreeBSD as the primary Zabbix home base, er at home.
I've had 2 levels of the Zabbix server on NetBSD, once on i386 and once on Arm but for this post I'll ignore those branches and focus on 2 FreeBSD directions for data and applications with the history ramifications.
When I started Zabbix 5 server was the best I could manage; as I tried other variations I had 6.2 up and running eventually. I figure out that exporting and importing history/trends from one Zabbix box to another is simple but time consuming as valuable data feeds increase.
Platform 1: Upgrade database in place; conform to schema changes
In early 2022, I started monitoring with Zabbix en masse, as they say.
-rw-r--r-- 1 root wheel 20385599 Aug 8 2022 postgresql-11.17.tar.bz2
-rw-r--r-- 1 root wheel 22132996 Aug 8 2022 postgresql-14.5.tar.bz2
While upgrading, I needed both versions 11 and 14; FreeBSD would not install one if the other was already. Hence, skipping the "make clean" part meant slightly quicker re-installs.
Installing zabbix62-server-6.2.3...
pkg-static: zabbix62-server-6.2.3 conflicts with zabbix54-server-5.4.9 (installs files into the same place).
===> zabbix62-frontend-php74-6.2.3 conflicts with installed package(s):
zabbix54-frontend-php74-5.4.9_1
===> Installing for zabbix62-server-6.2.3
===> Checking if zabbix62-server is already installed
===> zabbix62-server-6.2.3 is already installed
Installed packages to be REMOVED:
zabbix62-server: 6.2.3
Starting version (2022):
- Jan 30 21:58:37 freebsd1 pkg-static[69272]: postgresql11-server-11.14 installed
Upgrade phases (2023):
- Feb 20 15:46:38 freebsd1 pkg[11293]: postgresql11-server-11.17 deinstalled
- Feb 20 16:59:17 freebsd1 pkg-static[65464]: postgresql11-server-11.17 installed
- Feb 20 17:14:14 freebsd1 pkg[69391]: postgresql11-server-11.17 deinstalled
- Feb 20 17:15:10 freebsd1 pkg-static[69814]: postgresql14-server-14.5 installed
The "deinstalls" probably include the pre-packaged server version that only supports mysql/mariadb.
FAIL:
Configuration file error
| DB type "POSTGRESQL" is not supported by current setup. Possible values MYSQL. |
PHP
Feb 3 23:17:58 freebsd1 pkg-static[89086]: php80-pgsql-8.0.15 installed
Feb 3 23:57:55 freebsd1 pkg-static[15828]: php80-pgsql-8.0.15 deinstalled
Feb 4 02:51:20 freebsd1 pkg-static[52566]: php74-pgsql-7.4.27 installed
This sequence shows the installed package mix failed to include PHP 8.0; this limit was something I searched for in early installs. Zabbix didn't work with PHP 8 on the systems I could muster.
Data changes
Along the way, Zabbix changed history table keys so newer versions may not deal with older versions if there is redundancy (as I understand it). So history needed to be dumped and reloaded, being renamed along the way so that there was a fall back. If it worked.
38636366 Jan 17 03:14 backup_zabbix_history_text.sql
302137397 Jan 17 03:15 backup_zabbix_history_uint.sql
302137397 Jan 17 03:15 backup_zabbix_history_uint.sql
174197724 Jan 17 03:16 backup_zabbix_trends.sql
101438770 Jan 17 03:16 backup_zabbix_trends_uint.sql
101438770 Jan 17 03:16 backup_zabbix_trends_uint.sql
=> select count(*) from history_old;
count
---------
9281308
Platform 2: New database, import hosts and templates, manually import history.
Database Server versions
(2022)
May 12 15:13:26 freebsd2 pkg[11749]: postgresql14-server-14.5 installed
May 12 15:49:32 freebsd2 pkg[14127]: postgresql14-server-14.5 deinstalled
May 12 16:13:38 freebsd2 pkg[15371]: postgresql14-server-14.5 installed
My notes aren't clear on why 2 tries, but at least that's still correct on the Zabbix application server side. I moved the database in 2023 to a different system meaning no need to upgrade the original host unless I need to do a refresh there sometime (mmm, backups).
Feb 18 23:17:41 freebsd3 pkg-static[33301]: postgresql15-server-15.1_1 installed
Feb 18 23:26:00 freebsd3 pkg[42270]: postgresql15-server reinstalled: 15.1_1 -> 15.1_1
Feb 18 23:40:46 freebsd3 pkg[45979]: postgresql15-server-15.1_1 deinstalled
Feb 18 23:47:06 freebsd3 pkg-static[64934]: postgresql15-server-15.1_1 installed
I struggled a bit here with getting the package/port to include PostgreSQL. If I slipped, the make had to be re-done with particular incantations to forget my previous mistake. So I won't do it again, it's [ make rmconfig ].
Hammer time
20230220:143137.337 Unable to start Zabbix server due to unsupported PostgreSQL database version (11.17).
20230220:143137.337 Must be at least (13.0).
20230220:143137.337 Use of supported database version is highly recommended.
20230220:143137.337 Override by setting AllowUnsupportedDBVersions=1 in Zabbix server configuration file at your own risk.
=# INSERT INTO history SELECT * FROM history_old ON CONFLICT (itemid,clock,ns) DO NOTHING;
INSERT 0 9281304
[...]
COPY 1357
COPY 3828
COPY 0
COPY 18
COPY 1
COPY 9311373
COPY 5024
COPY 0
COPY 0
=> select count(*) from history;
count
---------
9311373
(1 row)
=> select count(*) from history_old;
count
---------
9281308
PHP
Here, unlike the legacy system, the PHP 8 version is working with Zabbix.
(2022)
May 12 15:50:41 freebsd2 pkg[14127]: php82-pgsql-8.2.0.r2 installed
May 12 16:12:16 freebsd2 pkg[15371]: php82-pgsql-8.2.0.r2 deinstalled
(2023)
Jan 10 20:20:05 freebsd2 pkg-static[50148]: php81-pgsql-8.1.14 installed
Data migration
To have a faster response time, I used system 3 as a database server, and installed PostgreSQL 15 over the previously running version 14.
Basic export import steps follow, along with dump size.
- Shutdown Zabbix application server
- Run a database export
- Set up target system at least the same database version
- Create database and schema as needed
- Run database import
- Shutdown original database
- Alter Zabbix server configuration
- Viola (ha)
2023-02-19 06:25:02.997 UTC [9256] FATAL: terminating connection due to administrator command
-rw-r--r-- 1 postgres postgres 562198532 Feb 19 06:01 /dump/zabbix_freebsd2.dump
This is the wholly grail; if I miss this screen or fail to check the right box, it's not show time, it's dump and reload time.
-rw-r--r-- 1 freebsd freebsd 24382685 Dec 23 2021 zabbix-5.4.9.tar.gz
-rw-r--r-- 1 freebsd freebsd 24510838 Jan 31 2022 zabbix-5.4.10.tar.gz
-rw-r--r-- 1 freebsd freebsd 41038757 Dec 5 08:44 zabbix-6.2.6.tar.gz
-rw-r--r-- 1 freebsd freebsd 24510838 Jan 31 2022 zabbix-5.4.10.tar.gz
-rw-r--r-- 1 freebsd freebsd 41038757 Dec 5 08:44 zabbix-6.2.6.tar.gz
Meanwhile 6.2.7 and 6.2.8 are probably out, on some platforms at least.
History results (from platform 1):
The gaps are due to sensors or test systems going offline or elsewhere.
Deltas
What did I find most different after trying the upgraded platform and the fresh platform? One obvious nicety is the delivery of open street maps in the base server. This only shows up for me in the fresh install, though.
Next big difference is the change from a single agent status level to seeing two of them, squeezed into one lamp.
Perhaps multiple interfaces may be configured to show here, though my first tries to add second adapters didn't pan out.
Option: HeartbeatFrequency is available on later agent versions, so check if this needs to be set (missing means an earlier configuration file).
This dash display works simply enough:
Groups and other metadata containers/tags have changed somewhat from the earliest versions I've used (5).
Which way for future upgrades? It depends, as usual. If the keys change, that's doable. Larger data collections are unlikely at my pace; for others trends would need to be analyzed. Avoiding "database unsupported" is always a good idea.
Ephemera
Along the way, I found this bug (hit it myself):
ZBX_NOTSUPPORTED: Cannot obtain a descriptor to access kernel virtual memory.
Although the question refers to NetBSD 8 and Zabbix 4 (both outdated in 2023), the error may still hit if you use an older agent software base. NetBSD patches to get an agent the correct system internal metrics are out there, maybe on package source work-in-progress, and maybe in a distribution near you. I went through each system I test to get a Zabbix 6.2.6 agent running, so they'd match the server version and not have quirks with features absent or deformed.
The FreeBSD kernel limited the import speed, I presume.
Feb 20 17:40:48 freebsd1 kernel: Limiting open port RST response from 223 to 200 packets/sec
Feb 20 17:40:50 freebsd1 kernel: Limiting open port RST response from 224 to 200 packets/sec
Feb 20 17:41:24 freebsd1 kernel: Limiting open port RST response from 220 to 200 packets/sec
References:
No comments:
Post a Comment