Native Zabbix systemd monitoring

Zabbix module that enables Zabbix to query the systemd D-Bus API for native and granular system state monitoring + relative cgroup (CPU, MEM, IO, ...) metrics.

Download

The following packages are available:

• Sources
• Zabbix 3.2 - EL7 x86_64 RPM

Install

# Required CentOS/RHEL apps:   yum install -y libtool make autoconf automake dbus-devel
# Required Debian/Ubuntu apps: apt-get install -y libtool make autoconf automake libdbus-1-dev
# Required Fedora apps:        dnf install -y libtool make autoconf automake dbus-devel
# Required openSUSE apps:      zypper install -y libtool make autoconf automake dbus-1-devel gcc
# Required Gentoo apps 1:      emerge sys-devel/libtool sys-devel/make sys-devel/autoconf
# Required Gentoo apps 2:      emerge sys-devel/automake sys-devel/gcc
# Source, use your version:    
# git clone https://git.zabbix.com/scm/zbx/zabbix.git --depth 1 --single-branch --branch 4.2.6 /usr/src/zabbix
# Generate config.h: bash -c "cd /usr/src/zabbix; ./bootstrap.sh; ./configure"
./autogen.sh
./configure --with-zabbix=/usr/src/zabbix
make
sudo make install

If you are using a packaged version of Zabbix, you may wish to redirect the installation directories as follows:

sudo make prefix=/usr sysconfdir=/etc libdir=/usr/lib64 install

Configure Zabbix agent to load module libzbxsystemd.so.

Available keys

Note: systemd.cgroup.* keys require the cgroup accounting. The system default for this setting may be controlled with Default*Accounting settings in systemd-system.conf.

Example how to enable cgroup accounting for Zabbix systemd monitoring:

sed -i -e "s/.*DefaultCPUAccounting=.*/DefaultCPUAccounting=yes/g" /etc/systemd/system.conf
sed -i -e "s/.*DefaultBlockIOAccounting=.*/DefaultBlockIOAccounting=yes/g" /etc/systemd/system.conf
sed -i -e "s/.*DefaultMemoryAccounting=.*/DefaultMemoryAccounting=yes/g" /etc/systemd/system.conf
systemctl daemon-reexec
systemctl restart zabbix-agent

Templates

Available examples of monitoring templates:

• Template App systemd services.xml - the discovery of enabled systemd services with True condition result

Examples

# return the system architecture
$ zabbix_get -k systemd[Architecture]
x86-64

# discover all units - filtering for sockets
$ zabbix_get -k systemd.unit.discovery[socket]
{
  "data": [
    {
      "{#UNIT.NAME}": "dbus.socket",
      "{#UNIT.DESCRIPTION}": "D-Bus System Message Bus Socket",
      "{#UNIT.LOADSTATE}": "loaded",
      "{#UNIT.ACTIVESTATE}": "active",
      "{#UNIT.SUBSTATE}": "running",
      "{#UNIT.OBJECTPATH}": "/org/freedesktop/systemd1/unit/dbus_2esocket",
      "{#UNIT.FRAGMENTPATH}": "/usr/lib/systemd/system/dbus.socket",
      "{#UNIT.UNITFILESTATE}": "static",
      "{#UNIT.CONDITIONRESULT}": "yes"
    },
    ...
  ]
}

# return the location of a mount unit
$ zabbix_get -k systemd.unit[dev-mqueue.mount,Mount,Where]
/dev/mqueue

# return the number of open connections on a socket unit
$ zabbix_get -k systemd.unit[dbus.socket,Socket,NConnections]
1

# discover all services
$ zabbix_get -k systemd.service.discovery[service]
{
  "data": [
    {
      "{#SERVICE.TYPE}": "service",
      "{#SERVICE.NAME}": "dbus.service",
      "{#SERVICE.DISPLAYNAME}": "D-Bus System Message Bus",
      "{#SERVICE.PATH}": "/usr/lib/systemd/system/dbus.service",
      "{#SERVICE.STARTUPNAME}": "static",
      "{#SERVICE.CONDITIONRESULT}": "yes"
    },
    ...
  ]
}

# return the state of a service as an integer
$ zabbix_get -k systemd.service.info[sshd]
0

# return the startup state of a service as an integer
$ zabbix_get -k systemd.service.info[sshd,startup]
0

# total cpu usage of dbus.service
$ zabbix_get -k systemd.cgroup.cpu[dbus.service,total]
44

# resident set size (RSS) mem usage of dbus.service
$ zabbix_get -k systemd.cgroup.mem[dbus.service,rss]
663552

# total queued iops of dbus.service
$ zabbix_get -k systemd.cgroup.dev[dbus.service,blkio.io_queued,Total]
0

Debugging

Please use systemctl, gdbus, busctl, zabbix_get utilities for debugging systemd unit properties and their values. For example debugging of ConditionResult property value for sshd service:

$ systemctl show sshd.service | grep ConditionResult
ConditionResult=yes
$ gdbus introspect --system --dest org.freedesktop.systemd1 --object-path \
  /org/freedesktop/systemd1/unit/sshd_2eservice \
  /org/freedesktop/systemd1/unit/sshd_2eservice | grep ConditionResult
      readonly b ConditionResult = true;
$ busctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/sshd_2eservice \
  | grep ConditionResult
.ConditionResult                    property  b              true               emits-change
$ zabbix_get -k systemd.unit[sshd.service,Unit,ConditionResult]
1

Enable DebugLevel=5 in the Zabbix agent config to see systemd module debug output in the zabbix-agent log file.

SELinux

If you have configured SELinux in enforcing mode, you might see the following error in your Zabbix logs, when attempting to use item keys from this module:

[systemd] org.freedesktop.DBus.Error.AccessDenied: SELinux policy denies access

This is because the SELinux policy that ships with RedHat/CentOS does not explicitly allow the Zabbix agent to communicate with D-Bus. This package includes an extension module to grant Zabbix only the permissions it requires for read-only access.

To build the SELinux module, add --enable-semodule to the build configuration:

./configure --with-zabbix=/usr/src/zabbix-3.2.6 --enable-semodule
make
sudo make prefix=/usr sysconfdir=/etc libdir=/usr/lib64 install

After installing this package, the SELinux module libzbxsystemd.pp can be enabled by running:

semodule -v -i /usr/share/selinux/packages/zabbix-module-systemd/libzbxsystemd.pp

Built-in Zabbix systemd support

Zabbix agent 2 supports systemd from version 4.4+ (ZBXNEXT-2871). Keys: systemd.unit.discovery, systemd.unit.info. However this module still offers more options for systemd monitoring.