Zabbix

This page has information about how to install, configure and run the system monitoring solution Zabbix.

Debian packages

Install the following packages from Debian backports

zabbix-agent
zabbix-server-pgsql
zabbix-frontend-php

Zabbix makes use of fping to perform ICMP pings. This package therefore must also be installed:

fping

Server setup

Create database and user

This is documented in

/usr/share/doc/zabbix-server-pgsql/README.Debian

Run the following commands to create the PostgreSQL database and the database user/role:

sudo -u postgres createdb zabbix
sudo -u postgres createuser -SDRP zabbix

Discussion

• -D = --no-createdb
• -P = --pwprompt
• -R = --no-createrole
• -S = --no-superuser

Finally, this command populates the database:

zcat /usr/share/zabbix-server-pgsql/{schema,images,data}.sql.gz | psql -h localhost zabbix zabbix

Server configuration

The Zabbix server configuration is stored in

/etc/zabbix/zabbix_server.conf

Local configuration snippets that override the main server configuration can be placed in the folder

/etc/zabbix/zabbix_server.conf.d

Here are the settings I use on pelargir:

root@pelargir:~# cat /etc/zabbix/zabbix_server.conf.d/pelargir.conf
# Listen for 1 trapper items on port 10051, otherwise the web frontend
# displays the error message 
# "Zabbix server is not running: the information displayed may not be current."
# Note from the documentation:
#       At least one trapper process must be running to display server availability and view queue
#       in the frontend.
# I don't know yet what "trapper items" are.
StartTrappers=1

# If listening for trapper items is required for any reason, at least
# do not listen on the public network interface.
ListenIP=127.0.0.1

DBPassword=secret

Because the configuration snippet contains a password, its file system permissions must be restricted:

root@pelargir:~# ls -l /etc/zabbix/zabbix_server.conf.d/pelargir.conf
-r--r----- 1 root zabbix 27 Jul 28 02:45 /etc/zabbix/zabbix_server.conf.d/pelargir.conf

Note: If the Zabbix Server cannot read any of its configuration files on startup, it will not start up at all - the error is only visible in the syslog.

Server start

The usual

/etc/init.d/zabbix-server start

Agent setup

Agent configuration

The Zabbix agent configuration is stored in

/etc/zabbix/zabbix_agentd.conf

Local configuration snippets that override the main agent configuration can be placed in the folder

/etc/zabbix/zabbix_agentd.conf.d

Here are the settings I use on pelargir:

# Do not listen on the public network interface
ListenIP=127.0.0.1

Agent start

The usual

/etc/init.d/zabbix-agent start

Web frontend setup

Apache configuration

The Zabbix packages provide an Apache configuration snippet, but it remains disabled by default. To enable it manually:

a2enconf zabbix-frontend-php

Also, add this to pelargir.herzbube.ch.conf to make Zabbix accessible under https://pelargir.herzbube.ch/zabbix/:

<Directory /usr/share/zabbix/>
  php_admin_flag engine on
</Directory>

PHP configuration

The Zabbix web frontend setup requires that the PHP timezone is set up correctly. See this wiki page for details.

Finish setup in web browser

Point the browser at this URL:

https://pelargir.herzbube.ch/zabbix/

This starts the setup wizard. Enter the following details:

• Database = Enter database name, user and password
• Server = localhost, port = 10051
• Installation name = pelargir.herzbube.ch

The setup process tries to create the config file

/etc/zabbix/zabbix.conf.php

which of course fails because the web server does not have write access to this directory. The configuration file can be manually downloaded and installed in the path shown above. Here is the content:

root@pelargir:~# cat /etc/zabbix/zabbix.conf.php
<?php
// Zabbix GUI configuration file.
global $DB;

$DB['TYPE']     = 'POSTGRESQL';
$DB['SERVER']   = 'localhost';
$DB['PORT']     = '0';
$DB['DATABASE'] = 'zabbix';
$DB['USER']     = 'zabbix';
$DB['PASSWORD'] = 'secret';

// Schema name. Used for IBM DB2 and PostgreSQL.
$DB['SCHEMA'] = '';

$ZBX_SERVER      = 'localhost';
$ZBX_SERVER_PORT = '10051';
$ZBX_SERVER_NAME = 'pelargir.herzbube.ch';

$IMAGE_FORMAT_DEFAULT = IMAGE_FORMAT_PNG;

Because the configuration file contains a password, its file system permissions must be restricted:

root@pelargir:~# ls -l /etc/zabbix/zabbix.conf.php
-r--r----- 1 root www-data 449 Jul 28 03:04 /etc/zabbix/zabbix.conf.php

Once the configuration file is in place, the setup wizard can be finished. Important: Immediately log in with the default username/password (Admin/zabbix) and change the password to something non-standard!

Configuration

Remove guest login

Navigate to "Administration > User groups > Guests", then disable the user group to remove the guest login option from the login screen.

Configure media types

Navigate to "Administration > Media types > Email". Configure the media type like this:

• SMTP server = localhost
• SMTP helo = zabbix
• SMTP email = zabbix@herzbube.ch
• Connection security = None
• Authentication = None

Disable the other media types

• Administration > Media types > Jabber
• Administration > Media types > SMS

Enable email alerts

Navigate to "Administration > Users > Admin > Media". Select "Add", then enter the following details:

• Type = Email
• Send to = herzbube@herzbube.ch

Enable server monitoring

Navigate to "Configuration > Hosts > Zabbix server", then enable the host.

Monitor internet connection

TODO: The following stuff doesn't work. I have not yet found out how to ping an external host that does not run a Zabbix agent.

Navigate to "Configuration > Host groups", then select "Create host group" and enter the following details:

• Group name = Internet servers

Navigate to "Configuration > Hosts", then select "Create host" and enter the following details:

• Host name = www.google.ch
• Groups = Internet servers
• Agent interfaces = Remove the default interface

Navigate to "Configuration > Hosts > Zabbix server > Items", then select "Create item" and enter the following details:

• Name = Internet connection
• Type = Simple check
• Key = icmpping[www.google.ch,4]

Service monitoring

Navigate to "Configuration > Hosts > Zabbix server > Templates". In the section "Link new templates", search and add the following templates:

• Template App SMTP Service
• Template App IMAP Service
• Template App LDAP Service
• Template App HTTP Service
• Template App HTTPS Service
• Template App MySQL Service
• Template App SSH Service

TODO: PostgreSQL

Disk I/O overload

The default check for disk I/O overload, which is named "Disk I/O is overloaded on {HOST.NAME}" and is provided by the Zabbix template "OS Linux", uses this trigger:

{Zabbix server:system.cpu.util[,iowait].avg(5m)}>20

which means that it checks that the iowait value, when averaged over the last 5 minutes, does not rise above 20%.

This trigger is too sensitive for pelargir: With the above "average over 5 minutes" trigger, the nightly backup generates an iowait spike of about 55% and thus causes a warning email to be sent. To prevent the backup process from triggering the warning email, I am increasing the average time to 15 minutes. Usually the iowait value is around 1%, and the backup duration is around 5 minutes, so tripling the time span should cause the average to drop well below the 20% mark.

Because the default trigger is provided by a template it cannot be edited. In order to change it you have to create a clone:

• Log in to the Zabbix web interface
• Select "Configuration > Hosts > Zabbix server > Triggers"
• Find the trigger for iowait
• Click on the link in the "Name" column (should be labeled "Disk I/O is overloaded on Zabbix server")
• Click the button "Clone" at the bottom
• Change the expression so that it checks the iowait average over 15 minutes: {Zabbix server:system.cpu.util[,iowait].avg(15m)}>20
• Change the name to "Avg(15m) Disk I/O is overloaded on {HOST.NAME}"
• Click the button "Add" at the bottom
• The new trigger should now appear in the list, and it should be marked as "Enabled"

The old trigger is still around and enabled. Click on the link in the "Status" column - it should change from "Enabled" to "Disabled".

See also this useful SO post.

Custom Graph

If you want to see the data for an item for which there is no pre-configured graph, you have to create a custom graph like this:

• Log in to the Zabbix web interface
• Select "Configuration > Hosts > Zabbix server > Graphs"
• Click the button "Create graph" at the bottom
• Add whatever items you need and configure the graph however you like

For details check out the Zabbix Manual.

Certificate validity checks

TODO

Diagnostics

Testing whether email works

Add a dummy trigger that is guaranteed to fire to the "Zabbix server" host:

• Name = root fs has more than 10% free space
• Expression = {Zabbix server:vfs.fs.size[/,pfree].last()}>10