Monitorando serviços com SNMP e Zabbix

Inicialização do servidor SNMP e coleta da rede

Entre no terminal da máquina virtual e inicialize o serviço SNMP nela através do comando:

# service snmpd restart

Ainda no terminal, execute os comandos snmpget, snmpgetnext e snmpwalk a fim de testar o cliente e servidor SNMP fornecido pela implementação do Net-SNMP:

# snmpget -v 1 -c public localhost sysDescr.0

# snmpget -v 1 -c public localhost system.sysName.0

# snmpget -v 1 -c public localhost ifDescr.2

# snmpgetnext -v 1 -c public localhost interfaces

# snmpgetnext -v 1 -c public localhost host

# snmpgetnext -v 1 -c public localhost snmpV2

# snmpwalk -v 1 -c public localhost system

# snmpwalk -v 1 -c public localhost interfaces

Repita os comandos anteriores utilizando os parâmetros -On, -Os e -Of.

Volte ao frontend do Zabbix e inclua um item com as seguintes configurações:

• Description: eth0 in/s

• Type: SNMPv1 agent

• SNMP OID: interfaces.ifTable.ifEntry.ifInOctets.2

• SNMP community: public

• SNMP port: 161

• Key: eth0.in

• Type of information: Numeric (unsigned)

• Store value: Delta (speed per second)

Repita a operação para incluir este outro indicador:

• Description: eth0 out/s

• Type: SNMPv1 agent

• SNMP OID: ifOutOctets.2

• SNMP community: public

• SNMP port: 161

• Key: eth0.out

• Type of information: Numeric (unsigned)

• Store value: Delta (speed per second)

Aguarde alguns minutos e então acesse o menu Monitoring > Latest data e abra os gráficos correspondente as coletas dos indicadores recém-criados:

Coleta da carga da CPU e gráficos customizados

Entre no terminal da máquina virtual e explore a UCD-MIB executando os comandos a seguir:

# snmpwalk -v 1 -c public localhost ucdavis | less

# snmpbulkget -v 2c -c public localhost -C r3 laNames laLoadInt -Os

Observe a carga média dos processadores executando as instruções a seguir:

# uptime

# snmpwalk -v 1 -c public localhost laLoadInt -Os

Provoque o aumento no consumo de CPU executando diversas vezes o comando abaixo:

# cat /dev/urandom | md5sum &

Dica: para cancelar a execução desses processos, utilize os comandos jobs e kill do Linux.

De volta ao frontend do Zabbix, cadastre o seguinte indicador para o host:

• Description: cpu load-1

• Type: SNMPv1 agent

• SNMP OID: laLoadInt.1 (pode ser OID completo – textual ou numérico)

• SNMP community: public

• SNMP port: 161

• Key: laLoadInt.1

• Type of information: Numeric (float)

• Use multiplier: Custom multiplier

• Custom multiplier: 0.01

Utilizando a funcionalidade de clonagem de indicadores, crie mais dois indicadores alterando apenas os campos indicados abaixo:

• Description: cpu load-5

• SNMP OID: laLoadInt.2

• Key: laLoadInt.2

• Description: cpu load-15

• SNMP OID: laLoadInt.3

• Key: laLoadInt.3

Acesse o menu Configuration > Hosts e crie um novo gráfico contendo os três indicadores de carga de CPU recém incluídos para o host:

Acesse o menu Monitoring > Latest data e o menu Monitoring > Graphs para acompanhar a evolução da coleta destes indicadores:

Referências:

• VirtualBox: http://www.virtualbox.org/

• Net-SNMP: http://www.net-snmp.org/

• Zabbix: http://www.zabbix.org/
  

• Ubuntu VirtualBox: https://help.ubuntu.com/community/VirtualBox

• Zabbix Appliance: http://www.zabbix.com/documentation/1.8/manual/installation/appliance

Testando Zabbix Appliance com VirtualBox

Baixe a partir do endereço a seguir o pacote Debian adequado a sua distribuição Linux:

http://www.virtualbox.org/wiki/Linux_Downloads

Instale o pacote via DPKG utilizando as instruções abaixo:

# dpkg -i virtualbox-*.deb

# apt-get -f install

Baixe a imagem ISO do Zabbix Appliance a partir do endereço a seguir:

http://www.zabbix.com/download.php

Execute o VirtualBox e crie uma nova máquina utilizando os parâmetros seguintes:

• Nome: Zabbix 1.8

• Sistema Operacional: Linux

• Versão: openSUSE

• Memória: 512 MB

• Disco rígido: (nenhum)

Clique na máquina recém-criada e associe a imagem ISO do Zabbix Appliance no drive de CD:

Configure a rede para utilizar a opção “host-only” e a interface “vboxnet0”:

Inicie a nova máquina virtual e aguarde o completo carregamento do sistema operacional.

No prompt que se abrirá, entre com o usuário “root” e a senha “zabbix”.

Carregue o mapa de teclados adequado e confira o endereço IP associado à máquina virtual através dos comandos a seguir:

# loadkeys br-abnt2

# ifconfig eth0

Anote o endereço IP exibido (ex: 192.168.56.101).

Abra no navegador o endereço HTTP correspondente ao frontend do Zabbix. Ex:

http://192.168.56.101/zabbix/

Entre na aplicação utilizando o login “Admin” e a password “zabbix”:

Acesse o menu Configuration > Hosts e crie um novo host utilizando essas configurações:

• Name: Local

• Groups: Linux servers

• IP address: 127.0.0.1

• Connect to: IP address

Acesse os indicadores do host e inclua um novo conforme indicado a seguir:

• Description: cpu load

• Type: Zabbix agent

• Key: system.cpu.load

• Type of information: Numeric (float)

Aguarde alguns minutos e então acesse o menu Monitoring > Latest data, procurando pelos dados históricos e gráficos gerados pela coleta do indicador “cpu load”.

Referências:

• VirtualBox: http://www.virtualbox.org/

• Zabbix: http://www.zabbix.org/
  

• Ubuntu VirtualBox: https://help.ubuntu.com/community/VirtualBox

• Zabbix Appliance: http://www.zabbix.com/documentation/1.8/manual/installation/appliance

Reading stdin on scripts for handling SNMP traps

In a shell, sometimes data are read through standard input (i.e STDIN) rather than arguments, which are most commonly used. For that reading purpose there is an instruction named "read", whose syntax is exemplified below:

$ read var
123

$ echo "var: $var"
var: 123

Today I was wondering how could I retrieve all values passed to a given Shell script considering that one does not know how many items are available. There are a lot of script samples of reading regular files, line by line, using "while" loop along with "read" command. But what if one needs to read the standard input?

Fortunately, there is a special device on Linux called /dev/stdin, which promised to give the expected outcome. And here's a testing script based on that assertion:

#!/bin/bash
while read s
do
    echo "s = $s"
done < /dev/stdin

And it works perfectly! Well, but.. is there any real application for such trick?

Yes, indeed! My real motivation was to debug and understand why the integration between Net-SNMP (i.e. snmptrapd service) and ZABBIX was not working when SNMP traps were received by the server. Those traps were supposed to be forwarded by snmptrapd to a Shell script called snmptrap.sh, which invoked zabbix_sender along with the received arguments.

Thus, in order to fine-grained debbuging, I created another script, named snmptrapdebug.sh with the contents below:

#!/bin/bash

echo "Request received: $0"
echo "--------------------------------------------------"

while read param
do
    echo -e "$param "
done < /dev/stdin

echo "--------------------------------------------------"
echo

exit 0

On the configuration file snmptrapd.conf the following line was appended:

traphandle default /bin/bash /home/zabbix/snmptrapdebug.sh

In order to send a simple SNMP trap the following command was issued:

$ snmptrap -v 1 -c public localhost .1.3.6.1.4.1.24.0 localhost 6 12345678 8 .1.3.6.1.4.1.24.12.10.22.64 s "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"

So, here are the LOG results after receiving a SNMP trap by the server:

2010-02-05 11:48:32 localhost [127.0.0.1] (via UDP: [127.0.0.1]:40338) TRAP, SNMP v1, community public
SNMPv2-SMI::enterprises.24.0 Enterprise Specific Trap (12345678) Uptime: 0:00:00.08
SNMPv2-SMI::enterprises.24.12.10.22.64 = STRING: "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"
Request received: /home/zabbix/snmptrapdebug.sh
--------------------------------------------------
localhost
UDP: [127.0.0.1]:40338
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:00:00.08
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-SMI::enterprises.24.0.0.12345678
SNMPv2-SMI::enterprises.24.12.10.22.64 "2010-02-04 17:03:18,352,DEBUG,main,SimpleTest,This is a Debug Message"
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 127.0.0.1
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 SNMPv2-SMI::enterprises.24.0
--------------------------------------------------

That's it! I hope that tip is worthful for someone else. :)

References:

• Tutorial on SNMP traps - Net-SNMP Wiki
• "How do I handle traps and notifications?" - Net_SNMP FAQ
• "Sending Traps" - Essential SNMP
• Canivete Suíço do Shell (Bash)

PostgreSQL monitoring on ZABBIX

I've recently started to get to know ZABBIX [1] a little deeper, especially regarding PostgreSQL database servers monitoring. At first sight I thought it an incredible monitoring and notification system as it fulfills most of the requirements I wondered to have in Cedrus [2].

After setting up some basic OS-related items to be monitored, I was searching for PostgreSQL specific configurations and then I found a wiki [3] on ZABBIX UserParameters. It is indeed very simple! You just need to create SQL statements and then invoke them with psql. If successful, you could append the corresponding lines into ZABBIX agent configuration file and restart it. Then, in the front-end application, what is left to do is to properly set these PostgreSQL parameters to a given host.

I've created some additional parameters I always use in PostgreSQL instances in order to monitor the server health. In this case, I previously created a user called "zabbix" and a database with same name on PostgreSQL.

Here are the included lines on /etc/zabbix/zabbix_agentd.conf:

# PostgreSQL custom parameters

# instance version
UserParameter=pgsql.version,psql -U zabbix zabbix -Atc 'select version()'

# instance databases summary
UserParameter=pgsql.db.summary,psql -c "select a.datname, pg_size_pretty(pg_database_size(a.datid)) as size, cast(blks_hit/(blks_read+blks_hit+0.000001)*100.0 as numeric(5,2)) as cache, cast(xact_commit/(xact_rollback+xact_commit+0.000001)*100.0 as numeric(5,2)) as success from pg_stat_database a order by a.datname"

# total databases size
UserParameter=pgsql.db.totalsize,psql -Atc "select sum(pg_database_size(datid)) as total_size from pg_stat_database"

# specific database size (in bytes)
UserParameter=pgsql.db.size[*],psql -Atc "select pg_database_size('$1') as size"

# database cache hit ratio (percentage)
UserParameter=pgsql.db.cache[*],psql -Atc "select cast(blks_hit/(blks_read+blks_hit+0.000001)*100.0 as numeric(5,2)) as cache from pg_stat_database where datname = '$1'"

# database success rate (percentage)
UserParameter=pgsql.db.success[*],psql -Atc "select cast(xact_commit/(xact_rollback+xact_commit+0.000001)*100.0 as numeric(5,2)) as success from pg_stat_database where datname = '$1'"

After restarting ZABBIX Agent, you can check whether the added parameters are valid by issuing zabbix_get command in a shell. For instance, to query PostgreSQL instance version, type this:

$ zabbix_get -s localhost -k pgsql.version
PostgreSQL 8.3.3 on i486-pc-linux-gnu, compiled by GCC cc (GCC) 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

This other script returns an overview of existing databases in the instance, highlighting their names, size in disk, cache hit ratio and percentage of successful transactions:

$ zabbix_get -s localhost -k pgsql.db.summary
   datname   |  size   | cache | success 
-------------+---------+-------+---------
 auction5    | 4400 kB |  0.00 |    0.00
 auditing    | 4512 kB |  0.00 |    0.00
 escola      | 4656 kB |  0.00 |    0.00
 postgres    | 4223 kB | 99.81 |  100.00
 rodrigo     | 4144 kB |  0.00 |    0.00
 template0   | 4144 kB |  0.00 |    0.00
 template1   | 4144 kB |  0.00 |    0.00
 zabbix      | 10 MB   | 99.99 |  100.00
 zahle       | 86 MB   |  0.00 |    0.00
(9 registros)

In order to measure total space in disk occupied by the entire instance, this script should be used:

$ zabbix_get -s localhost -k pgsql.db.totalsize
1507326452

On the other hand, to retrieve the space (in bytes) occupied by a single database, you just need to specify its name in the parameter key, as exemplified below:

$ zabbix_get -s localhost -k pgsql.db.size[auction5] 
4505604

Likewise, to recover cache hit ratio (in percentage) of a single database, use this key:

$ zabbix_get -s localhost -k pgsql.db.cache[zabbix]
99.99

The percentage of successful transactions in relation to all attempts is given by this parameter:

$ zabbix_get -s localhost -k pgsql.db.success[postgres]
100.00

After testing these parameters, it is time to set them up onto ZABBIX Frontend as illustrated below:



It is very interesting to further add some traps and actions based on the configured items. For example, to send an email to the DBA every time a given database grows up faster than expected or whether its cache ratio starts to lower significantly.

At Joe Uhl's blog there is a post [4] concerning using ZABBIX to monitor PostgreSQL TPS (Transactions per Second). It is an interesting source as it explains how to configure deltas and graphs in ZABBIX.

References:

[1] ZABBIX Monitoring Solution
[2] Cedrus: PostgreSQL Manager
[3] ZABBIX Wiki - PostgreSQL UserParameters
[4] Monitoring PostgreSQL TPS with Zabbix