Matejunkie » nagios

How to find a proper monitoring solution

Mike Adolphs — Tue, 20 Oct 2009 22:03:54 +0000

As in our own environment at nugg.ad every startup/HA environment needs a proper monitoring solution to fit the minimum requirements to fullfill the high availability demands of your customers. Without a proper solution you can trust on you’re screwed. It’s simple as that.
You just need to know first whether your application servers are melting, your database/nosql backend begins to burst or someone accidentally the whole thing (you know the meme, do you?). Not only to inform customers before they call you, but also to be able to plan further growth of your environment.

Since I’m currently rewriting our whole monitoring environment based on 3 1/2 years experience with Nagios and its competitors I thought to share that knowledge with you. Just in case someone has a use for this.

Choosing a monitoring solution

First rule: Stick with the monitoring solution you’re at least a bit experienced with and which fits to your environment. In huge environments Zabbix is capable of doing the monitoring work for you and so is Zenoss or Nagios as well. In smaller environments MRTG or Munin might also do the job.
All big Open Source monitoring solutions are highly customizable and extendable. You just need to know how to find the right plugins or to ask properly within the community how something can be achieved.
If you’re not experienced with Open Source monitoring solutions at all, get a first look on the feature set of various solutions at Wikipedia. Choose wisely afterwards and most important: Stick to that solution for quite a while to explore its advantages and to get better at anger management when facing its disadvantages as well. Sooner or later you’ll get the big picture.

It’s not that important which software you choose. It’s more important what you make out of it for your environment!

The user’s demands

What I’ve expierienced within the last years is that the demands are quite comprehensive:

The operations team needs to get informed almost instantly in case of a real emergency via various contact possibilities
… and it’ll let you stay longer at the office to fix this when they get woken up each night due to false alarms
The CTO needs proper escalation methods to get informed when something’s broken and not taken care of
The executives and its board need a nice and shiny visualization of the platform to present the company’s growth and its state
The consulting or support team needs a simple read only web-interface to get a proper impression whether everything’s allright in case of a unexpected customer call.

All of these demands have something in common: The basis of all operations is trust.

The operations team has to trust your monitoring solution to fix problems in a fast, but advised way instead of ignoring problems after the fourth false alarm during a week. The CTO needs to know that your escalation strategies are working and that you don’t screw him. The executives need proper graphs without spikes or even downtimes which they’re not able to explain within meetings and the consulting or support team’s need is to get an overview of an almost real time state of your environment without getting confused.

My monitoring environment

I’m currently using Nagios as the basic monitoring solution for our environment with several plugins attached to it.

To reuse all the data provided by Nagios I let it write its information to a MySQL database via ndoutils. This enables you to use almost any software which understands the ndoutils database layout, e.g. nice and shiny web-interfaces or visualization tools like nagvis.
For the graphing I’m currently using two solutions. pnp4nagios 0.6x with its highly recommended NPCD daemon acts as the basis for proper graphing of various system information. Since the pnp4nagios web-interface is only recommended for Unix/Linux system administrators I’m reusing the rrd databases within Cacti to provide a better overview of the whole platform. Cacti again is mostly being used for SNMP based checks which needs no alerting.

You’ll now see the difference: System metrics which needs a proper alerting are handled by Nagios itself and metrics that only needs to be monitored for its statistics are realized via Cacti (and therefore mostly SNMP). This eases your configuration work, keeps system resources in balance and avoids misunderstandings within the team that needs to take care of your environment while enjoying your Martini on the Keys.

To enable the consultants to get a good overview of the platform’s health, I’m using two different tools. At first NagVis provides a graphic overview of the system’s health without providing too detailed data. A traffic light based graphic with three states (green, yellow and red) might be too few information, but a Google Maps based view of your various datacenter locations with green, yellow or red icons will do the job. The supporters then are able to explain to the customer that datacenter 123 is down due to a failure within the system which is enough in most cases. All other cases with demand of a clearer view will be redirected to you, no worries. They know their job.
For the consultants which are more experienced with the system itself I provide a nice and shiny Nagios interface within Cacti itself. They’re able then to access Cacti’s performance graphs and the metrics provided by Nagios as well.

Conclusion

Maintaining an Open Source monitoring solution means a lot of work and you have to grapple with your favorite monitoring solution for quite a while before achieving your goals. But if you do, you’re the peacekeeper that lets your colleagues sleep during the night, the magician that casts nice and shiny graphs instantly and the master who has a global overview of your platform.
I hope, this gave you a basic view on how to find the right monitoring solution for you without going into the technical details. If you’ve got any questions let me know.

http://www.youtube.com/watch?v=JmS0Kjxs2v4

Database fix for Nagios plugin for Cacti

admin — Mon, 12 Oct 2009 15:30:37 +0000

If you’re interested in running Nagios within the Cacti interface you may be interested in the Nagios plugin for Cacti (NPC) project which does a good job overall in combining these two powerful tools within the monitoring/graphing world. Nevertheless, I had a database related problem during the initial setup due to the fact that NPC brings along its own schema of the ndo database which is outdated since the last release of ndoutils 1.4b8 (they added a new column called long_output).

Therefore, you want do the changes below via your mysql console to the following tables if you’d like to use the latest versions of ndoutils (1.4b8) with npc (2.0.4):

use databasename ;
alter table npc_servicechecks add column long_output VARCHAR(8192) NOT NULL default '' AFTER output;
alter table npc_servicestatus add column long_output VARCHAR(8192) NOT NULL default '' AFTER output;
alter table npc_systemcommands add column long_output VARCHAR(8192) NOT NULL default '' AFTER output;
alter table npc_statehistory add column long_output VARCHAR(8192) NOT NULL default '' AFTER output;

Afterwards NPC should work like a charm.

PNP template for check_apache2.sh

Mike Adolphs — Tue, 08 Sep 2009 20:07:52 +0000

This is a PNP template which works quite well for me when there’s need to identify possible bottlenecks or unexpected behavior of the Apache webserver. It shows the CPU usage, req/sec, byte/req and the amount of busy/idle workers in a single graph. Feel free to copy the template from below or head over to Nagios Exchange.

#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

#   PNP Template for check_apache2.sh
#   Author: Mike Adolphs (http://www.matejunkie.com/)

$opt[1] = "--vertical-label \"Various stats \" -l 0 -r --title \"Apache metrics for $hostname / $servicedesc\" ";
$def[1]  = "DEF:cpu=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .= "DEF:reqpsec=$rrdfile:$DS[2]:AVERAGE " ;
$def[1] .= "DEF:bytepreq=$rrdfile:$DS[4]:AVERAGE " ;
$def[1] .= "DEF:wbusy=$rrdfile:$DS[5]:AVERAGE " ;
$def[1] .= "DEF:widle=$rrdfile:$DS[6]:AVERAGE " ;
 
$opt[2] .= "--vertical-label \"Byte/sec \" -l 0 -r --title \"Apache metrics for $hostname / $servicedesc\" ";
$def[2] .= "DEF:bytepsec=$rrdfile:$DS[3]:AVERAGE " ;
 
$def[1] .= "COMMENT:\"\\t\\t\\t\\tLAST\\t\\tAVERAGE\\t\\tMAX\\n\" " ;
 
$def[1] .= "LINE2:cpu#008000:\"CPU usage [%]\\t \" " ;
$def[1] .= "GPRINT:cpu:LAST:\"%6.0lf\\t\" " ;
$def[1] .= "GPRINT:cpu:AVERAGE:\" %6.0lf\\t\" " ;
$def[1] .= "GPRINT:cpu:MAX:\" %6.0lf\\n\" " ;
 
$def[1] .= "LINE2:reqpsec#0C64E8:\"Requests/sec\\t \" " ;
$def[1] .= "GPRINT:reqpsec:LAST:\"%6.0lf\\t\" " ;
$def[1] .= "GPRINT:reqpsec:AVERAGE:\" %6.0lf\\t\" " ;
$def[1] .= "GPRINT:reqpsec:MAX:\" %6.0lf\\n\" " ;
 
$def[1] .= "LINE2:bytepreq#FFA500:\"Byte/req\\t\\t \" " ;
$def[1] .= "GPRINT:bytepreq:LAST:\"%6.0lf\\t\" " ;
$def[1] .= "GPRINT:bytepreq:AVERAGE:\" %6.0lf\\t\" " ;
$def[1] .= "GPRINT:bytepreq:MAX:\" %6.0lf\\n\" " ;
 
$def[1] .= "LINE2:wbusy#1CC8E8:\"Busy workers\\t \" " ;
$def[1] .= "GPRINT:wbusy:LAST:\"%6.0lf\\t\" " ;
$def[1] .= "GPRINT:wbusy:AVERAGE:\" %6.0lf\\t\" " ;
$def[1] .= "GPRINT:wbusy:MAX:\" %6.0lf\\n\" " ;
 
$def[1] .= "LINE2:widle#E80C8C:\"Idle workers\\t \" " ;
$def[1] .= "GPRINT:widle:LAST:\"%6.0lf\\t\" " ;
$def[1] .= "GPRINT:widle:AVERAGE:\" %6.0lf\\t\" " ;
$def[1] .= "GPRINT:widle:MAX:\" %6.0lf\\n\" " ;
 
$def[2] .= "COMMENT:\"\\t\\t\\t\\tLAST\\t\\tAVERAGE\\t\\tMAX\\n\" " ;
 
$def[2] .= "LINE2:bytepsec#E80C3E:\"Byte/sec\\t\\t \" " ;
$def[2] .= "GPRINT:bytepsec:LAST:\"%6.0lf\\t\" " ;
$def[2] .= "GPRINT:bytepsec:AVERAGE:\" %6.0lf\\t\" " ;
$def[2] .= "GPRINT:bytepsec:MAX:\" %6.0lf\\n\" " ;

Update 1.3: Apache check plugin for Nagios

Mike Adolphs — Mon, 07 Sep 2009 22:49:16 +0000

Long time no see. Sorry for the long downtime, but I’ve been on vacation for quite some time and spent my time out of reach of any electronical device except my phone. Anyway, strength and motivation for work during spare time is back again! Expect a few updates of various plugins I’ve written in the past within the next weeks.

For the beginning here’s the revised version of the Apache check plugin for Nagios. It now doesn’t write temporary files anymore, the code is cleaner and it should be a bit faster. Note that I haven’t done much testing for now (will be done tomorrow and on Wednesday). Therefore it might won’t work out of the box for you. At least in my environments it works well. Let me know if you have any problems or if you stumble upon something which is completely nonsense.

Copy’n'paste below or svn co via svn.matejunkie.com/nagios-plugins/trunk/check_apache/.

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.3,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "Description:"
    echo "$PROGNAME is a Nagios plugin to check the Apache's server status."
    echo "It monitors requests per second, bytes per second/request, "
    echo "amount of busy/idle workers and its CPU load."
    echo ""
    echo "Example call:"
    echo "./$PROGNAME -H localhost -P 80 -t 3 -b /usr/sbin -p /var/run \\"
    echo "-n apache2.pid -s status_page [-S] [-R] [-wr] 100 [-cr] 250"
    echo ""
    echo "Options:"
    echo "  -H|--hostname)"
    echo "    Sets the hostname. Default is: localhost"
    echo "  -P|--port)"
    echo "    Sets the port. Default is: 80"
    echo "  -t|--timeout)"
    echo "    Sets a timeout within the server's status page must've been"
    echo "    accessed. Otherwise the check will go into an error state."
    echo "    Default is: 3"
    echo "  -b|--binary-path)"
    echo "    Sets the path to the apache binary. Used for getting Apache's"
    echo "    CPU load. Default is: /usr/sbin"
    echo "  -p|--pid-path)"
    echo "    Path to Apache's pid file. Default is: /var/run"
    echo "  -n|--pid-name)"
    echo "    Name of Apache's pid file. Default is: apache2.pid"
    echo "  -s|--status-page)"
    echo "    Defines the name of the status page. Default is: server-status"
    echo "  -R|--remote-server)"
    echo "    Disabled the pid check so that remote Apaches can be queried."
    echo "    Default is: off"
    echo "  -S|--secure)"
    echo "    Enables HTTPS (no certificate check though). Default is: off"
    echo "  -wr|--warning-req)"
    echo "    Sets a warning level for requests per second. Default is: off"
    echo "  -cr|--critical-req)"
    echo "    Sets a critical level for requests per second. Default is: off"
    exit $ST_UK
}
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
hostname="localhost"
port=80
remote_srv=0
path_binary="/usr/sbin"
path_pid="/var/run"
name_pid="apache2.pid"
status_page="server-status"
timeout=3
secure=0
running=0
 
wcdiff_req=0
wclvls_req=0
 
while test -n "$1"; do
    case "$1" in
        --help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --timeout|-t)
            timeout=$2
            shift
            ;;
        --remote-server|-R)
            remote_srv=1
            ;;
        --binary_path|-b)
            path_binary=$2
            shift
            ;;
        --pid_path|-p)
            path_pid=$2
            shift
            ;;
        --pid_name|-n)
            name_pid=$2
            shift
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning-req|-wr)
            warn_req=$2
            shift
            ;;
        --critical-req|-cr)
            crit_req=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
    esac
    shift
done
 
# check functions
val_wcdiff_req() {
    if [ ! -z "$warn_req" -a ! -z "$crit_req" ]
    then
        wclvls_req=1
 
        if [ ${warn_req} -gt ${crit_req} ]
        then
            wcdiff_req=1
        fi
    elif [ ! -z "$warn_req" -a -z "$crit_req" ]
    then
        wcdiff_req=2
    elif [ -z "$warn_req" -a ! -z "$crit_req" ]
    then
        wcdiff_req=3
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
check_processes() {
    if [ $1 -lt 1 ]
    then
        echo "UNKNOWN - Your Apache server seems not to run. Is your Nagios \
privileged to run 'ps ax' and is the Apache2 binary really located in \
$path_binary?"
        exit $ST_UK
    fi
}
 
check_output() {
    stat_output=`stat -c %s ${output_dir}/server-status`
    if [ "$stat_output" = 0 ]
    then
        echo "UNKNOWN - Local copy of server-status is empty. Are we \
allowed to access http://${hostname}:${port}/server-status?"
        exit $ST_UK
    fi
}
 
# get functions
get_status() {
    if [ "$secure" = 1 ]
    then
        server_status1=`wget -qO- --no-check-certificate -t 3 \
-T ${timeout} https://${hostname}:${port}/${status_page}?auto`
    sleep 1
        server_status2=`wget -qO- --no-check-certificate -t 3 \
-T ${timeout} https://${hostname}:${port}/${status_page}?auto`
    else
        server_status1=`wget -qO- -t 3 -T ${timeout} \
http://${hostname}:${port}/${status_page}?auto`
        sleep 1
        server_status2=`wget -qO- -t 3 -T ${timeout} \
http://${hostname}:${port}/${status_page}?auto`
    fi
}
 
get_vals() {
    cpu_load="$(cpu_load=0; ps -Ao pcpu,args | grep "$path_binary/apache2" \
| awk '{print $1}' | while read line
    do
        cpu_load=`echo "scale=3; $cpu_load + $line" | bc -l`
    echo $cpu_load
    done)"
    cpu_load=`echo $cpu_load | awk '{print $NF}' | sed 's/^\./0./'`
 
    tmp1_req_psec=`echo ${server_status1} | awk '{print $3}'`
    tmp2_req_psec=`echo ${server_status2} | awk '{print $3}'`
    req_psec=`echo "scale=2; ${tmp2_req_psec} - ${tmp1_req_psec}" | bc -l \
| sed 's/^\./0./'`
 
    bytes_psec=`echo ${server_status1} | awk '{print $14}' | sed 's/^\./0./'`
    bytes_preq=`echo ${server_status1} | awk '{print $16}' | sed 's/^\./0./'`
    wkrs_busy=`echo ${server_status1} | awk '{print $18}' | sed 's/^\./0./'`
    wkrs_idle=`echo ${server_status1} | awk '{print $20}' | sed 's/^\./0./'`
}
 
do_output() {
    output="Apache serves $req_psec Requests per second with an average CPU \
utilization of $cpu_load%. Busy workers: $wkrs_busy, idle: $wkrs_idle"
}
 
do_perfdata() {
    perfdata="'cpu_load'=$cpu_load 'req_psec'=$req_psec \
'bytes_psec'=$bytes_psec 'bytes_preq'=$bytes_preq 'workers_busy'=$wkrs_busy \
'workers_idle'=$wkrs_idle"
}
 
# Let's go
val_wcdiff_req
 
if [ "$wcdiff_req" = 1 ]
then
    echo "Please adjust your warning/critical thresholds. The warning must \
be lower than the critical level!"
    exit $ST_UK
elif [ "$wcdiff_req" = 2 ]
then
    echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
    exit $ST_UK
elif [ "$wcdiff_req" = 3 ]
then
    echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
    exit $ST_UK
else
    if [ "$remote_srv" = 0 ]
    then
        running=`check_pid`
        check_pid $running
    fi
 
    get_status
    get_vals
 
    do_output
    do_perfdata
 
    if [ ${wclvls_req} = 1 ]
    then
        if [ ${req_psec} -ge ${warn_req} -a ${req_psec} -lt ${crit_req} ]
        then
            echo "WARNING - ${output} | ${perfdata}"
            exit $ST_WR
        elif [ ${req_psec} -ge ${crit_req} ]
        then
            echo "CRITICAL - ${output} | ${perfdata}"
        exit $ST_CR
        else
            echo "OK - ${output} | ${perfdata}"
            exit $ST_OK
        fi
    else
        echo "OK - ${output} | ${perfdata}"
        exit $ST_OK
    fi
fi

Update 1.1: nginx Check Plugin for Nagios

Mike Adolphs — Tue, 16 Jun 2009 07:51:24 +0000

Last week I’ve got an E-Mail by Jordi, who’s blogging at terraquis.net, with a few suggestions to the nginx check plugin I’ve recently released. He made two significant changes to the plugin:

His version runs with bash only, but except bc all external calls have been replaced by bash built-ins
There’s no temporary file anymore which has to be written to a temporary directory, nginx_status page output is saved into a variable

The plugin therefore runs much faster than my initial version of the plugin: 14-16ms plus the 1 second sleep interval between the two nginx_status checks.

And then I realized that writing output to a temporary file is just one thing: plain stupid. Why writing files when an information can be stored in a variable where it stays in memory?
Well, I can’t really say why I walked this way when I’ve written the first plugins, but I eventually had the right idea when I’ve written the Hadoop DFS check. There I already store the information within variables. The only guess why I did this is that I’ve assumed that storing longer output in variables just would be too much.

But anyway, there has to be room for improvements, right? Therefore, expect a few reworks within the next days, …

The script

Below is the revised script I’ve already uploaded yesterday to Nagios and MonitoringExchange. It runs much faster than the initial version and doesn’t write files anymore. You may also stick to the output below or do the SVN way.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_nginx/ check_nginx/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.1,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
hostname="localhost"
port=80
path_pid=/var/run
name_pid="nginx.pid"
status_page="nginx_status"
pid_check=1
secure=0
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check whether nginx is running."
    echo "It also parses the nginx's status page to get requests and"
    echo "connections per second as well as requests per connection. You"
    echo "may have to alter your nginx configuration so that the plugin"
    echo "can access the server's status page."
    echo "The plugin is highly configurable for this reason. See below for"
    echo "available options."
    echo ""
    echo "$PROGNAME -H localhost -P 80 -p /var/run -n nginx.pid "
	echo "  -s nginx_statut -o /tmp [-w INT] [-c INT] [-S] [-N]"
    echo ""
    echo "Options:"
    echo "  -H/--hostname)"
    echo "     Defines the hostname. Default is: localhost"
    echo "  -P/--port)"
    echo "     Defines the port. Default is: 80"
    echo "  -p/--path-pid)"
    echo "     Path where nginx's pid file is being stored. You might need"
    echo "     to alter this path according to your distribution. Default"
    echo "     is: /var/run"
    echo "  -n/--name_pid)"
    echo "     Name of the pid file. Default is: nginx.pid"
    echo "  -N/--no-pid-check)"
    echo "     Turn this on, if you don't want to check for a pid file"
    echo "     whether nginx is running, e.g. when you're checking a"
    echo "     remote server. Default is: off"
    echo "  -s/--status-page)"
    echo "     Name of the server's status page defined in the location"
    echo "     directive of your nginx configuration. Default is:"
    echo "     nginx_status"
    echo "  -S/--secure)"
    echo "     In case your server is only reachable via SSL, use this"
    echo "     this switch to use HTTPS instead of HTTP. Default is: off"
    echo "  -w/--warning)"
    echo "     Sets a warning level for requests per second. Default is: off"
    echo "  -c/--critical)"
    echo "     Sets a critical level for requests per second. Default is:"
	echo "     off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --path-pid|-p)
            path_pid=$2
            shift
            ;;
        --name-pid|-n)
            name_pid=$2
            shift
            ;;
        --no-pid-check|-N)
            pid_check=0
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -ge ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
get_status() {
    if [ "$secure" = 1 ]
    then
        wget_opts="-O- -q -t 3 -T 3 --no-check-certificate"
        out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
        sleep 1
        out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
    else        
        wget_opts="-O- -q -t 3 -T 3"
        out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
        sleep 1
        out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
    fi
 
    if [ -z "$out1" -o -z "$out2" ]
    then
        echo "UNKNOWN - Local copy/copies of $status_page is empty."
        exit $ST_UK
    fi
}
 
get_vals() {
    tmp1_reqpsec=`echo ${out1}|awk '{print $10}'`
    tmp2_reqpsec=`echo ${out2}|awk '{print $10}'`
    reqpsec=`expr $tmp2_reqpsec - $tmp1_reqpsec`
 
    tmp1_conpsec=`echo ${out1}|awk '{print $9}'`
    tmp2_conpsec=`echo ${out2}|awk '{print $9}'`
    conpsec=`expr $tmp2_conpsec - $tmp1_conpsec`
 
    reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
    if [ "$reqpcon" = ".99" ]
    then
        reqpcon="1.00"
    fi
}
 
do_output() {
    output="nginx is running. $reqpsec requests per second, $conpsec \
connections per second ($reqpcon requests per connection)"
}
 
do_perfdata() {
    perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
if [ ${pid_check} = 1 ]
then
    check_pid
    if [ "$retval" = 1 ]
    then
        echo "There's no pid file for nginx. Is nginx running? Please \
also make sure whether your pid path and name is correct."
        exit $ST_CR
    fi
fi
 
get_status
get_vals
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$reqpsec" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} ]"
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

Jordi’s script

This is Jordi’s bash built-in (plus bc) version which runs even faster. Feel free to copy the output below.

#!/bin/bash
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
hostname="localhost"
port=80
path_pid=/var/run
name_pid="nginx.pid"
status_page="nginx_status"
pid_check=1
secure=0
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check whether nginx is running."
    echo "It also parses the nginx's status page to get requests and"
    echo "connections per second as well as requests per connection. You"
    echo "may have to alter your nginx configuration so that the plugin"
    echo "can access the server's status page."
    echo "The plugin is highly configurable for this reason. See below for"
    echo "available options."
    echo ""
    echo "$PROGNAME -H localhost -P 80 -p /var/run -n nginx.pid "
	echo "  -s nginx_statut -o /tmp [-w INT] [-c INT] [-S] [-N]"
    echo ""
    echo "Options:"
    echo "  -H/--hostname)"
    echo "     Defines the hostname. Default is: localhost"
    echo "  -P/--port)"
    echo "     Defines the port. Default is: 80"
    echo "  -p/--path-pid)"
    echo "     Path where nginx's pid file is being stored. You might need"
    echo "     to alter this path according to your distribution. Default"
    echo "     is: /var/run"
    echo "  -n/--name_pid)"
    echo "     Name of the pid file. Default is: nginx.pid"
    echo "  -N/--no-pid-check)"
    echo "     Turn this on, if you don't want to check for a pid file"
    echo "     whether nginx is running, e.g. when you're checking a"
    echo "     remote server. Default is: off"
    echo "  -s/--status-page)"
    echo "     Name of the server's status page defined in the location"
    echo "     directive of your nginx configuration. Default is:"
    echo "     nginx_status"
    echo "  -o/--output-directory)"
    echo "     Unneeded. The option is preserved for backwards compatibility."
    echo "  -S/--secure)"
    echo "     In case your server is only reachable via SSL, use this"
    echo "     this switch to use HTTPS instead of HTTP. Default is: off"
    echo "  -w/--warning)"
    echo "     Sets a warning level for requests per second. Default is: off"
    echo "  -c/--critical)"
    echo "     Sets a critical level for requests per second. Default is:"
	echo "     off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --path-pid|-p)
            path_pid=$2
            shift
            ;;
        --name-pid|-n)
            name_pid=$2
            shift
            ;;
        --no-pid-check|-N)
            pid_check=0
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --output-directory|-o)
            output_dir=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -gt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
get_status() {
    url="http://${hostname}:${port}/${status_page}"
 
    if [ "$secure" = 1 ] ; then
        wget_opts="-O- -q -t 3 -t 3 --no-check-certificate"
    else
        wget_opts="-O- -q -t 3 -t 3"
    fi
 
    out1=$(wget ${wget_opts} ${url})
    sleep 1
    out2=$(wget ${wget_opts} ${url})
 
    # Both commands must return some content.
    if [ -z "${out1}" -o -z "${out2}" ] ; then
        echo "UNKNOWN - Local copy/copies of $status_page is empty."
        exit $ST_UK
    fi
}
 
get_vals() {
    # Convert into an array.
    values1=(${out1})
    values2=(${out2})
 
    reqpsec=$(( ${values2[9]} - ${values1[9]} ))
    conpsec=$(( ${values2[8]} - ${values1[8]} ))
 
    reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
    if [ "$reqpcon" = ".99" ]
    then
        reqpcon="1.00"
    fi
}
 
do_output() {
    output="nginx is running. $reqpsec requests per second, $conpsec \
connections per second ($reqpcon requests per connection)"
}
 
do_perfdata() {
    perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
if [ ${pid_check} = 1 ]
then
    check_pid
    if [ "$retval" = 1 ]
    then
        echo "There's no pid file for nginx. Is nginx running? Please \
also make sure whether your pid path and name is correct."
        exit $ST_CR
    fi
fi
 
get_status
get_vals
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$reqpsec" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} ]"
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

License

As always this little script is ment to be sh-compliant and released under the terms of the GPL Version 2 only. Feel free to subscribe via rss to get updates on this one. More options will be added in the future.

Nagios Exchange opens up

Mike Adolphs — Tue, 09 Jun 2009 11:45:46 +0000

Just a short note from my side:
A week ago Ethan launched Nagios Exchange, an addon and plugin repository similar to MonitoringExchange, which is part of the whole Nagios website framework. I began to add all of my plugins to it as well, but due to the approval process it might take a while till they’re downloadable. I’ll let you know when it’s done.

Nevertheless, from now on you’re even more free from where to download the plugins I’ve written. From now on they’re available via this blog, via SVN, via MonitoringExchange and via Nagios Exchange. I’ll update them on all sources.

Hadoop DFS Check Plugin for Nagios

Mike Adolphs — Tue, 02 Jun 2009 22:46:45 +0000

Since I’m currently occupied with kicking off a several terabytes large Hadoop cluster I thought it’d be a good idea to provide a few Nagios plugins related to Hadoop.

Just in case you don’t know: Hadoop is a free Java software framework that supports data intensive distributed applications. It got its own filesystem called HDFS which scales up to petabytes of storage running on top of the operating system’s filesystem. Hadoop is inspired by Google’s MapReduce and GFS, the Google File System.

Yahoo for example uses several Hadoop Clusters with 100.000 CPUs total in 20.000 boxes (2.000 boxes per cluster) according to the official Hadoop website. If you haven’t checked it out yet, do so! This is – sorry for that – crazy shit and it’s working!
For the beginning I’d recommend reading the Wikipedia article, the offical documentation and, of course, watching the Hadoop related lectures over at Cloudera.

Anyway, the first plugin is almost finished and checks the amount of running DataNodes and the capacity of the DFS. It’s working, but I want to add a few more features so that the plugin become more flexible. Nevertheless, feel free to give it a try if you’re in the mood.

The Script

Hadoop is completely operated by the hadoop user and since it got its own filesystem it got its own permissions as well. To keep things clean and safe we won’t give hadoop related permissions to the Nagios user. We’ll rather enable the Nagios user to run a small shell script as root which contains a command ran by the hadoop user to get information about the cluster.

Add the following small script to a directory of your choice, e.g. /usr/local/sbin:

#!/bin/sh
su -s /bin/bash - hadoop -c 'hadoop dfsadmin -report'

Afterwards alter its permissions so that it’s only read-, write and accessible by root:

user@host: ~ $ sudo chmod 700 /usr/local/sbin/get-dfsreport.sh

Then enable the Nagios user to sudo run the script via /etc/sudoers (or better visudo):

nagios          ALL=(ALL)       NOPASSWD: /usr/local/sbin/get-dfsreport.sh

This is it for the prerequisites. You may then run the script provided for copy’n'pasting below (or svn co, your choice).
If you’ve chosen another directory than /usr/local/sbin you have to provide the path via -s/–path-sh when running the script.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_hadoop-dfs/ check_hadoop-dfs/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
path_sh="/usr/local/sbin"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check the status of HDFS, Hadoop's"
    echo "underlying, redundant, distributed file system."
    echo ""
    echo "$PROGNAME -s /usr/local/sbin [-w 10] [-c 5]"
    echo ""
    echo "Options:"
    echo "  -s|--path-sh)"
    echo "     Path to the shell script that is mentioned in the"
    echo "     documentation. Default is: /usr/local/sbin"
    echo "  -w|--warning)"
    echo "     Defines the warning level for available datanodes. Default"
    echo "     is: off"
    echo "  -c|--critical)"
    echo "     Defines the critical level for available datanodes. Default"
    echo "     is: off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --path-sh|-s)
            path_sh=$2
            shift
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -lt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be higher than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
get_vals() {
    tmp_vals=`sudo ${path_sh}/get-dfsreport.sh`
    dn_avail=`echo -e "$tmp_vals" | grep -m1 "Datanodes available:" | awk '{print $3}'`
    dfs_used=`echo -e "$tmp_vals" | grep -m1 "Used raw bytes:" | awk '{print $4}'`
    dfs_used=`expr ${dfs_used} / 1024 / 1024`
    dfs_used_p=`echo -e "$tmp_vals" | grep -m1 "% used:" | awk '{print $3}'`
    dfs_total=`echo -e "$tmp_vals" | grep -m1 "Total raw bytes:" | awk '{print $4}'`
    dfs_total=`expr ${dfs_total} / 1024 / 1024`
}
 
do_output() {
    output="Datanodes up and running: ${dn_avail}, DFS total: \
${dfs_total} MB, DFS used: ${dfs_used} MB (${dfs_used_p})"
}
 
do_perfdata() {
    perfdata="'datanodes_available'=${dn_avail} 'dfs_total'=${dfs_total} \
'dfs_used'=${dfs_used}"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
get_vals
 
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$dn_avail" -le "$warning" -a "$dn_avail" -gt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$dn_avail" -le "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} "
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

Output example:

If everything went fine you should see output like the following:

user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 40 -c 30
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 60 -c 40
WARNING - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 70-c 60
CRITICAL - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 20 -c 40
Please adjust your warning/critical thresholds. The warning must be higher than the critical level!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 30
Please also set a critical value when you want to use warning/critical thresholds!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -c 30
Please also set a warning value when you want to use warning/critical thresholds!

The License

PNP Template for Process Check Plugin

Mike Adolphs — Tue, 02 Jun 2009 12:35:07 +0000

Just a short note:
Here’s the appropriate PNP Template to the process check plugin which I’ve recently written to create nice and shiny graphs:

#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

#   PNP Template for check_ps.sh
#   Author: Mike Adolphs (http://www.matejunkie.com/

$opt[1] = "--vertical-label \"percent\" -u 100 -l 0 -r --title \"CPU/Memory Usage for $hostname / $servicedesc\" ";
$opt[2] = "--vertical-label \"minutes\" -u 100 -l 0 -r --title \"cputime for $hostname / $servicedesc\" ";
 
$def[1] =  "DEF:cpu=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .=  "DEF:memory=$rrdfile:$DS[2]:AVERAGE " ;
$def[2] .=  "DEF:cputime=$rrdfile:$DS[3]:AVERAGE " ;
 
$def[1] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
$def[2] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
 
$def[1] .= "LINE2:cpu#E80C3E:\"CPU\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:cpu:MAX:\"%6.2lf \\n\" " ;
 
$def[1] .= "LINE2:memory#008000:\"Memory\\t\" " ;
$def[1] .= "GPRINT:memory:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:memory:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:memory:MAX:\"%6.2lf \\n\" " ;
 
$def[2] .= "AREA:cputime#E80C3E:\"CPUTime\\t\" " ;
$def[2] .= "GPRINT:cputime:LAST:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:AVERAGE:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:MAX:\"%6.2lf min\\n\" " ;

Process Check Plugin for Nagios

Mike Adolphs — Sat, 30 May 2009 17:07:24 +0000

This is a sh-compliant check plugin for Nagios which enables you to monitor a specific process whether it’s running or not and to see how much cpu/memory it’s wasting. This can be accomplished with the check_procs plugin which is included in the official Nagios-Plugins package as well, but it doesn’t generate any performance output. At least that’s what I’ve experienced lately (I’d have written the plugin anyway for training purposes).

The script

No special needs, just the basic stuff. The script parses the output of ps aux in different ways. You need to provide a specific process via -p/–process, if warning/critical thresholds are wanted you may also choose between cpu or memory thresholds via -t/–target (default is memory).
As always, you may grab the plugin at MonitoringExchange or directly from here via SVN and the copy’n'paste way below.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_ps/ check_ps/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
process="cron"
target="mem"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check a specific process via ps."
    echo "You may provide any string as an argument to match a specific"
    echo "process. Please note that the output could be distorted if the"
    echo "argument matches various processes, so please make sure to use"
    echo "unique strings to match a process."
    echo ""
    echo "$PROGNAME -p firefox [-w 10] [-c 20] [-t cpu]"
    echo ""
    echo "Options:"
    echo "  -p/--process)"
    echo "     You need to provide a string for which the ps output is then"
    echo "     then \"greped\"."
    echo "  -w/--warning)"
    echo "     Defines a warning level for a target which is explained"
    echo "     below. Default is: off"
    echo "  -c/--critical)"
    echo "     Defines a critical level for a target which is explained"
    echo "     below. Default is: off"
    echo "  -t/--target)"
    echo "     A target can be defined via -t. Choose between cpu and mem."
    echo "     Default is: mem"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --process|-p)
            process=$2
            shift
            ;;
        --target|-t)
            target=$2
            shift
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
        if [ ${warning} -gt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
get_vals() {
    process=`echo ${process} | sed 's/^.\|[a-z][A-Z] /\[&]/g'`
 
    tmp_output=`ps aux | grep "$process" | grep -v $0`
 
    if [ -z "$tmp_output" ]
    then
        echo "CRITICAL - Process is not running!"
        exit $ST_CR
    fi
 
    ps_user=`echo ${tmp_output} | awk '{print $1}'`
    ps_pid=`echo ${tmp_output} | awk '{print $2}' `
    ps_cpu=`echo ${tmp_output} | awk '{print $3}'`
    ps_mem=`echo ${tmp_output} | awk '{print $4}' `
    ps_start=`echo ${tmp_output} | awk '{print $9}' `
 
    tmp_ps_cputime=`echo ${tmp_output} | awk '{print $10}'`
    tmp_ps_cpuhours=`echo ${tmp_ps_cputime} | awk -F \: '{print $1}'`
    tmp_ps_cpumin=`echo ${tmp_ps_cputime} | awk -F \: '{print $2}'`
    ps_cputime=`echo "scale=0; (${tmp_ps_cpuhours} * 60) + \
${tmp_ps_cpumin}" | bc -l`
}
 
do_wccalc() {
    if [ -n "$warning" -a -n "$critical" ]
    then
        if [ "$target" = "cpu" ]
        then
            tmp_wc_target=`echo ${ps_cpu} | awk -F \. '{print $2}'`
            if [ "$tmp_wc_target" -ge 5 ]
            then
                wc_target=`echo ${ps_cpu} | awk -F \. '{print $1}'`
                wc_target=`expr ${wc_target} + 1`
            else
                wc_target=`echo ${ps_cpu} | awk -F \. '{print $1}'`
            fi
                elif [ "$target" = "mem" ]
                then
            tmp_wc_target=`echo ${ps_mem} | awk -F \. '{print $2}'`
            if [ "$tmp_wc_target" -ge 5 ]
            then
                wc_target=`echo ${ps_mem} | awk -F \. '{print $1}'`
                wc_target=`expr ${wc_target} + 1`
            else
                wc_target=`echo ${ps_mem} | awk -F \. '{print $1}'`
            fi
        fi
    fi
}
 
do_output() {
    process=`echo ${process} | sed 's/\[//g' |  sed 's/\]//g'`
    output="Process: ${process}, User: ${ps_user}, CPU: ${ps_cpu}%, \
RAM: ${ps_mem}%, Start: ${ps_start}, CPU Time: ${ps_cputime} min"
}
 
do_perfdata() {
    perfdata="'cpu'=${ps_cpu} 'memory'=${ps_mem} 'cputime'=${ps_cputime}"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
get_vals
do_wccalc
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$wc_target" -ge "$warning" -a "$wc_target" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
        exit $ST_WR
    elif [ "$wc_target" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
        exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata}"
        exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

PNP Template

#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

#   PNP Template for check_ps.sh
#   Author: Mike Adolphs (http://www.matejunkie.com/

$opt[1] = "--vertical-label \"percent\" -u 100 -l 0 -r --title \"CPU/Memory Usage for $hostname / $servicedesc\" ";
$opt[2] = "--vertical-label \"minutes\" -u 100 -l 0 -r --title \"cputime for $hostname / $servicedesc\" ";
 
$def[1] =  "DEF:cpu=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .=  "DEF:memory=$rrdfile:$DS[2]:AVERAGE " ;
$def[2] .=  "DEF:cputime=$rrdfile:$DS[3]:AVERAGE " ;
 
$def[1] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
$def[2] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
 
$def[1] .= "LINE2:cpu#E80C3E:\"CPU\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:cpu:MAX:\"%6.2lf \\n\" " ;
 
$def[1] .= "LINE2:memory#008000:\"Memory\\t\" " ;
$def[1] .= "GPRINT:memory:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:memory:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:memory:MAX:\"%6.2lf \\n\" " ;
 
$def[2] .= "AREA:cputime#E80C3E:\"CPUTime\\t\" " ;
$def[2] .= "GPRINT:cputime:LAST:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:AVERAGE:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:MAX:\"%6.2lf min\\n\" " ;

The License

nginx Check Plugin for Nagios

Mike Adolphs — Sun, 24 May 2009 16:34:14 +0000

Update: Please also consider reading the article regarding Update 1.1 for check_nginx.sh.

Ok, this was obvious. Here’s a Nagios plugin I’ve just finished to check whether nginx is running and to get some performance data like requests and connections per second as well as requests per connection.

Please note that the StubStatus module for nginx must be compiled into it, this isn’t done by default! And please make sure that you’ve added a location directive to your nginx configuration like the following as well. The script must be able to access it!

location /nginx_status {
    stub_status on;
    access_log off;
    allow 127.0.0.1;
}

The script

Feel free to grab the script from below, at MonitoringExchange or via svn checkout. Let me know if something’s not working for you and I’ll improve it. If you have any questsions do so as well.

user@host: ~ $ svn co http://svn.matejunkie.com/nagios-plugins/stable/check_nginx/ check_nginx/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
hostname="localhost"
port=80
path_pid=/var/run
name_pid="nginx.pid"
status_page="nginx_status"
output_dir=/tmp
pid_check=1
secure=0
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check whether nginx is running."
    echo "It also parses the nginx's status page to get requests and"
    echo "connections per second as well as requests per connection. You"
    echo "may have to alter your nginx configuration so that the plugin"
    echo "can access the server's status page."
    echo "The plugin is highly configurable for this reason. See below for"
    echo "available options."
    echo ""
    echo "$PROGNAME -H localhost -P 80 -p /var/run -n nginx.pid "
	echo "  -s nginx_statut -o /tmp [-w INT] [-c INT] [-S] [-N]"
    echo ""
    echo "Options:"
    echo "  -H/--hostname)"
    echo "     Defines the hostname. Default is: localhost"
    echo "  -P/--port)"
    echo "     Defines the port. Default is: 80"
    echo "  -p/--path-pid)"
    echo "     Path where nginx's pid file is being stored. You might need"
    echo "     to alter this path according to your distribution. Default"
    echo "     is: /var/run"
    echo "  -n/--name_pid)"
    echo "     Name of the pid file. Default is: nginx.pid"
    echo "  -N/--no-pid-check)"
    echo "     Turn this on, if you don't want to check for a pid file"
    echo "     whether nginx is running, e.g. when you're checking a"
    echo "     remote server. Default is: off"
    echo "  -s/--status-page)"
    echo "     Name of the server's status page defined in the location"
    echo "     directive of your nginx configuration. Default is:"
    echo "     nginx_status"
    echo "  -o/--output-directory)"
    echo "     Specifies where to write the tmp-file that the check creates."
    echo "     Default is: /tmp"
    echo "  -S/--secure)"
    echo "     In case your server is only reachable via SSL, use this"
    echo "     this switch to use HTTPS instead of HTTP. Default is: off"
    echo "  -w/--warning)"
    echo "     Sets a warning level for requests per second. Default is: off"
    echo "  -c/--critical)"
    echo "     Sets a critical level for requests per second. Default is:"
	echo "     off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --path-pid|-p)
            path_pid=$2
            shift
            ;;
        --name-pid|-n)
            name_pid=$2
            shift
            ;;
        --no-pid-check|-N)
            pid_check=0
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --output-directory|-o)
            output_dir=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
    esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -gt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
get_status() {
    if [ "$secure" = 1 ]
    then
        wget --no-check-certificate -q -t 3 -T 3 \
http://${hostname}:${port}/${status_page} -O ${output_dir}/nginx-status.1
        sleep 1
        wget --no-check-certificate -q -t 3 -T 3 \
http://${hostname}:${port}/${status_page} -O ${output_dir}/nginx-status.2
    else
        wget -q -t 3 -T 3 http://${hostname}:${port}/${status_page} \
-O ${output_dir}/nginx-status.1
        sleep 1
        wget -q -t 3 -T 3 http://${hostname}:${port}/${status_page} \
-O ${output_dir}/nginx-status.2
    fi
 
    stat_output1=`stat -c %s ${output_dir}/nginx-status.1`
    stat_output2=`stat -c %s ${output_dir}/nginx-status.2`
    if [ "$stat_output1" = 0 -o "$stat_output2" = 0 ]
    then
        echo "UNKNOWN - Local copy/copies of $status_page is empty."
    exit $ST_UK
    fi
}
 
get_vals() {
    tmp1_reqpsec=`grep '^ ' ${output_dir}/nginx-status.1|awk '{print $3}'`
    tmp2_reqpsec=`grep '^ ' ${output_dir}/nginx-status.2|awk '{print $3}'`
    reqpsec=`expr $tmp2_reqpsec - $tmp1_reqpsec`
 
    tmp1_conpsec=`grep '^ ' ${output_dir}/nginx-status.1|awk '{print $2}'`
    tmp2_conpsec=`grep '^ ' ${output_dir}/nginx-status.2|awk '{print $2}'`
    conpsec=`expr $tmp2_conpsec - $tmp1_conpsec`
 
    reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
    if [ "$reqpcon" = ".99" ]
    then
        reqpcon="1.00"
    fi
}
 
do_output() {
    output="nginx is running. $reqpsec requests per second, $conpsec \
connections per second ($reqpcon requests per connection)"
}
 
do_perfdata() {
    perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
if [ ${pid_check} = 1 ]
then
    check_pid
    if [ "$retval" = 1 ]
    then
        echo "There's no pid file for nginx. Is nginx running? Please \
also make sure whether your pid path and name is correct."
        exit $ST_CR
    fi
fi
 
get_status
get_vals
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$reqpsec" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} ]"
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi