Matejunkie » linux

How to find a proper monitoring solution

Mike Adolphs — Tue, 20 Oct 2009 22:03:54 +0000

As in our own environment at nugg.ad every startup/HA environment needs a proper monitoring solution to fit the minimum requirements to fullfill the high availability demands of your customers. Without a proper solution you can trust on you’re screwed. It’s simple as that.
You just need to know first whether your application servers are melting, your database/nosql backend begins to burst or someone accidentally the whole thing (you know the meme, do you?). Not only to inform customers before they call you, but also to be able to plan further growth of your environment.

Since I’m currently rewriting our whole monitoring environment based on 3 1/2 years experience with Nagios and its competitors I thought to share that knowledge with you. Just in case someone has a use for this.

Choosing a monitoring solution

First rule: Stick with the monitoring solution you’re at least a bit experienced with and which fits to your environment. In huge environments Zabbix is capable of doing the monitoring work for you and so is Zenoss or Nagios as well. In smaller environments MRTG or Munin might also do the job.
All big Open Source monitoring solutions are highly customizable and extendable. You just need to know how to find the right plugins or to ask properly within the community how something can be achieved.
If you’re not experienced with Open Source monitoring solutions at all, get a first look on the feature set of various solutions at Wikipedia. Choose wisely afterwards and most important: Stick to that solution for quite a while to explore its advantages and to get better at anger management when facing its disadvantages as well. Sooner or later you’ll get the big picture.

It’s not that important which software you choose. It’s more important what you make out of it for your environment!

The user’s demands

What I’ve expierienced within the last years is that the demands are quite comprehensive:

The operations team needs to get informed almost instantly in case of a real emergency via various contact possibilities
… and it’ll let you stay longer at the office to fix this when they get woken up each night due to false alarms
The CTO needs proper escalation methods to get informed when something’s broken and not taken care of
The executives and its board need a nice and shiny visualization of the platform to present the company’s growth and its state
The consulting or support team needs a simple read only web-interface to get a proper impression whether everything’s allright in case of a unexpected customer call.

All of these demands have something in common: The basis of all operations is trust.

The operations team has to trust your monitoring solution to fix problems in a fast, but advised way instead of ignoring problems after the fourth false alarm during a week. The CTO needs to know that your escalation strategies are working and that you don’t screw him. The executives need proper graphs without spikes or even downtimes which they’re not able to explain within meetings and the consulting or support team’s need is to get an overview of an almost real time state of your environment without getting confused.

My monitoring environment

I’m currently using Nagios as the basic monitoring solution for our environment with several plugins attached to it.

To reuse all the data provided by Nagios I let it write its information to a MySQL database via ndoutils. This enables you to use almost any software which understands the ndoutils database layout, e.g. nice and shiny web-interfaces or visualization tools like nagvis.
For the graphing I’m currently using two solutions. pnp4nagios 0.6x with its highly recommended NPCD daemon acts as the basis for proper graphing of various system information. Since the pnp4nagios web-interface is only recommended for Unix/Linux system administrators I’m reusing the rrd databases within Cacti to provide a better overview of the whole platform. Cacti again is mostly being used for SNMP based checks which needs no alerting.

You’ll now see the difference: System metrics which needs a proper alerting are handled by Nagios itself and metrics that only needs to be monitored for its statistics are realized via Cacti (and therefore mostly SNMP). This eases your configuration work, keeps system resources in balance and avoids misunderstandings within the team that needs to take care of your environment while enjoying your Martini on the Keys.

To enable the consultants to get a good overview of the platform’s health, I’m using two different tools. At first NagVis provides a graphic overview of the system’s health without providing too detailed data. A traffic light based graphic with three states (green, yellow and red) might be too few information, but a Google Maps based view of your various datacenter locations with green, yellow or red icons will do the job. The supporters then are able to explain to the customer that datacenter 123 is down due to a failure within the system which is enough in most cases. All other cases with demand of a clearer view will be redirected to you, no worries. They know their job.
For the consultants which are more experienced with the system itself I provide a nice and shiny Nagios interface within Cacti itself. They’re able then to access Cacti’s performance graphs and the metrics provided by Nagios as well.

Conclusion

Maintaining an Open Source monitoring solution means a lot of work and you have to grapple with your favorite monitoring solution for quite a while before achieving your goals. But if you do, you’re the peacekeeper that lets your colleagues sleep during the night, the magician that casts nice and shiny graphs instantly and the master who has a global overview of your platform.
I hope, this gave you a basic view on how to find the right monitoring solution for you without going into the technical details. If you’ve got any questions let me know.

http://www.youtube.com/watch?v=JmS0Kjxs2v4

Update 1.3: Apache check plugin for Nagios

Mike Adolphs — Mon, 07 Sep 2009 22:49:16 +0000

Long time no see. Sorry for the long downtime, but I’ve been on vacation for quite some time and spent my time out of reach of any electronical device except my phone. Anyway, strength and motivation for work during spare time is back again! Expect a few updates of various plugins I’ve written in the past within the next weeks.

For the beginning here’s the revised version of the Apache check plugin for Nagios. It now doesn’t write temporary files anymore, the code is cleaner and it should be a bit faster. Note that I haven’t done much testing for now (will be done tomorrow and on Wednesday). Therefore it might won’t work out of the box for you. At least in my environments it works well. Let me know if you have any problems or if you stumble upon something which is completely nonsense.

Copy’n'paste below or svn co via svn.matejunkie.com/nagios-plugins/trunk/check_apache/.

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.3,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "Description:"
    echo "$PROGNAME is a Nagios plugin to check the Apache's server status."
    echo "It monitors requests per second, bytes per second/request, "
    echo "amount of busy/idle workers and its CPU load."
    echo ""
    echo "Example call:"
    echo "./$PROGNAME -H localhost -P 80 -t 3 -b /usr/sbin -p /var/run \\"
    echo "-n apache2.pid -s status_page [-S] [-R] [-wr] 100 [-cr] 250"
    echo ""
    echo "Options:"
    echo "  -H|--hostname)"
    echo "    Sets the hostname. Default is: localhost"
    echo "  -P|--port)"
    echo "    Sets the port. Default is: 80"
    echo "  -t|--timeout)"
    echo "    Sets a timeout within the server's status page must've been"
    echo "    accessed. Otherwise the check will go into an error state."
    echo "    Default is: 3"
    echo "  -b|--binary-path)"
    echo "    Sets the path to the apache binary. Used for getting Apache's"
    echo "    CPU load. Default is: /usr/sbin"
    echo "  -p|--pid-path)"
    echo "    Path to Apache's pid file. Default is: /var/run"
    echo "  -n|--pid-name)"
    echo "    Name of Apache's pid file. Default is: apache2.pid"
    echo "  -s|--status-page)"
    echo "    Defines the name of the status page. Default is: server-status"
    echo "  -R|--remote-server)"
    echo "    Disabled the pid check so that remote Apaches can be queried."
    echo "    Default is: off"
    echo "  -S|--secure)"
    echo "    Enables HTTPS (no certificate check though). Default is: off"
    echo "  -wr|--warning-req)"
    echo "    Sets a warning level for requests per second. Default is: off"
    echo "  -cr|--critical-req)"
    echo "    Sets a critical level for requests per second. Default is: off"
    exit $ST_UK
}
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
hostname="localhost"
port=80
remote_srv=0
path_binary="/usr/sbin"
path_pid="/var/run"
name_pid="apache2.pid"
status_page="server-status"
timeout=3
secure=0
running=0
 
wcdiff_req=0
wclvls_req=0
 
while test -n "$1"; do
    case "$1" in
        --help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --timeout|-t)
            timeout=$2
            shift
            ;;
        --remote-server|-R)
            remote_srv=1
            ;;
        --binary_path|-b)
            path_binary=$2
            shift
            ;;
        --pid_path|-p)
            path_pid=$2
            shift
            ;;
        --pid_name|-n)
            name_pid=$2
            shift
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning-req|-wr)
            warn_req=$2
            shift
            ;;
        --critical-req|-cr)
            crit_req=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
    esac
    shift
done
 
# check functions
val_wcdiff_req() {
    if [ ! -z "$warn_req" -a ! -z "$crit_req" ]
    then
        wclvls_req=1
 
        if [ ${warn_req} -gt ${crit_req} ]
        then
            wcdiff_req=1
        fi
    elif [ ! -z "$warn_req" -a -z "$crit_req" ]
    then
        wcdiff_req=2
    elif [ -z "$warn_req" -a ! -z "$crit_req" ]
    then
        wcdiff_req=3
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
check_processes() {
    if [ $1 -lt 1 ]
    then
        echo "UNKNOWN - Your Apache server seems not to run. Is your Nagios \
privileged to run 'ps ax' and is the Apache2 binary really located in \
$path_binary?"
        exit $ST_UK
    fi
}
 
check_output() {
    stat_output=`stat -c %s ${output_dir}/server-status`
    if [ "$stat_output" = 0 ]
    then
        echo "UNKNOWN - Local copy of server-status is empty. Are we \
allowed to access http://${hostname}:${port}/server-status?"
        exit $ST_UK
    fi
}
 
# get functions
get_status() {
    if [ "$secure" = 1 ]
    then
        server_status1=`wget -qO- --no-check-certificate -t 3 \
-T ${timeout} https://${hostname}:${port}/${status_page}?auto`
    sleep 1
        server_status2=`wget -qO- --no-check-certificate -t 3 \
-T ${timeout} https://${hostname}:${port}/${status_page}?auto`
    else
        server_status1=`wget -qO- -t 3 -T ${timeout} \
http://${hostname}:${port}/${status_page}?auto`
        sleep 1
        server_status2=`wget -qO- -t 3 -T ${timeout} \
http://${hostname}:${port}/${status_page}?auto`
    fi
}
 
get_vals() {
    cpu_load="$(cpu_load=0; ps -Ao pcpu,args | grep "$path_binary/apache2" \
| awk '{print $1}' | while read line
    do
        cpu_load=`echo "scale=3; $cpu_load + $line" | bc -l`
    echo $cpu_load
    done)"
    cpu_load=`echo $cpu_load | awk '{print $NF}' | sed 's/^\./0./'`
 
    tmp1_req_psec=`echo ${server_status1} | awk '{print $3}'`
    tmp2_req_psec=`echo ${server_status2} | awk '{print $3}'`
    req_psec=`echo "scale=2; ${tmp2_req_psec} - ${tmp1_req_psec}" | bc -l \
| sed 's/^\./0./'`
 
    bytes_psec=`echo ${server_status1} | awk '{print $14}' | sed 's/^\./0./'`
    bytes_preq=`echo ${server_status1} | awk '{print $16}' | sed 's/^\./0./'`
    wkrs_busy=`echo ${server_status1} | awk '{print $18}' | sed 's/^\./0./'`
    wkrs_idle=`echo ${server_status1} | awk '{print $20}' | sed 's/^\./0./'`
}
 
do_output() {
    output="Apache serves $req_psec Requests per second with an average CPU \
utilization of $cpu_load%. Busy workers: $wkrs_busy, idle: $wkrs_idle"
}
 
do_perfdata() {
    perfdata="'cpu_load'=$cpu_load 'req_psec'=$req_psec \
'bytes_psec'=$bytes_psec 'bytes_preq'=$bytes_preq 'workers_busy'=$wkrs_busy \
'workers_idle'=$wkrs_idle"
}
 
# Let's go
val_wcdiff_req
 
if [ "$wcdiff_req" = 1 ]
then
    echo "Please adjust your warning/critical thresholds. The warning must \
be lower than the critical level!"
    exit $ST_UK
elif [ "$wcdiff_req" = 2 ]
then
    echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
    exit $ST_UK
elif [ "$wcdiff_req" = 3 ]
then
    echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
    exit $ST_UK
else
    if [ "$remote_srv" = 0 ]
    then
        running=`check_pid`
        check_pid $running
    fi
 
    get_status
    get_vals
 
    do_output
    do_perfdata
 
    if [ ${wclvls_req} = 1 ]
    then
        if [ ${req_psec} -ge ${warn_req} -a ${req_psec} -lt ${crit_req} ]
        then
            echo "WARNING - ${output} | ${perfdata}"
            exit $ST_WR
        elif [ ${req_psec} -ge ${crit_req} ]
        then
            echo "CRITICAL - ${output} | ${perfdata}"
        exit $ST_CR
        else
            echo "OK - ${output} | ${perfdata}"
            exit $ST_OK
        fi
    else
        echo "OK - ${output} | ${perfdata}"
        exit $ST_OK
    fi
fi

The Top 10 of your most used UNIX commands

Mike Adolphs — Tue, 07 Jul 2009 09:30:30 +0000

Almost three years ago Uwe Hermann posted a one-liner in his blog which was mentioned in an article from IBM called UNIX Productivity tips. It shows you the ten most used commands on your local machine and I think it’s time to revive this habit.

Do the following on your console and see what you’re frequently using:

user@host: ~ $ history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -rn|head -10

This is the result of my workstation which now runs for about 1 1/2 months with a current total of 8.000 commands:

   1987 ssh
    767 ls
    746 sudo
    349 for
    267 cd
    259 exit
    231 vim
    156 svn
    154 ping
    130 pwd

To get Songbird 1.2 up and running under Ubuntu 9.04

Mike Adolphs — Fri, 19 Jun 2009 08:44:04 +0000

For those about to rock: Songbird is an Open Source iTunes alternative which aims to be interoperable between Windows, Mac and Linux operating systems.

When I checked out Songbird 1.1 a few weeks ago I was quite astonished in the first place. It comes with a nice and shiny interface, it’s fast and quite stable as well and, of course, there’re various web-services like Shoutcast or Last.fm integrated. The only thing that Songbird 1.1 really lacked off was the non-existing equalizer, which is indispensable in my opinion. Therefore, I switched back to Amarok 1.4 on my Ubuntu desktop (Gnome) and to audacious on my Gentoo desktop (wmii).

Yesterday Songbird 1.2 has been released and besides a lot of other improvements… hell yes, it now has an equalizer onboard which works well at least for my sense of hearing.

To get it running under Ubuntu simply download the tar.gz from the Songbird website, extract it and remove two libvisual plugins that otherwise prevent Songbird from starting up.

user@host: ~ $ sudo aptitude remove libvisual-0.4-plugins libvisual-projectm

If you haven’t checked it out yet, give it a go!

Update 1.1: nginx Check Plugin for Nagios

Mike Adolphs — Tue, 16 Jun 2009 07:51:24 +0000

Last week I’ve got an E-Mail by Jordi, who’s blogging at terraquis.net, with a few suggestions to the nginx check plugin I’ve recently released. He made two significant changes to the plugin:

His version runs with bash only, but except bc all external calls have been replaced by bash built-ins
There’s no temporary file anymore which has to be written to a temporary directory, nginx_status page output is saved into a variable

The plugin therefore runs much faster than my initial version of the plugin: 14-16ms plus the 1 second sleep interval between the two nginx_status checks.

And then I realized that writing output to a temporary file is just one thing: plain stupid. Why writing files when an information can be stored in a variable where it stays in memory?
Well, I can’t really say why I walked this way when I’ve written the first plugins, but I eventually had the right idea when I’ve written the Hadoop DFS check. There I already store the information within variables. The only guess why I did this is that I’ve assumed that storing longer output in variables just would be too much.

But anyway, there has to be room for improvements, right? Therefore, expect a few reworks within the next days, …

The script

Below is the revised script I’ve already uploaded yesterday to Nagios and MonitoringExchange. It runs much faster than the initial version and doesn’t write files anymore. You may also stick to the output below or do the SVN way.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_nginx/ check_nginx/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.1,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
hostname="localhost"
port=80
path_pid=/var/run
name_pid="nginx.pid"
status_page="nginx_status"
pid_check=1
secure=0
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check whether nginx is running."
    echo "It also parses the nginx's status page to get requests and"
    echo "connections per second as well as requests per connection. You"
    echo "may have to alter your nginx configuration so that the plugin"
    echo "can access the server's status page."
    echo "The plugin is highly configurable for this reason. See below for"
    echo "available options."
    echo ""
    echo "$PROGNAME -H localhost -P 80 -p /var/run -n nginx.pid "
	echo "  -s nginx_statut -o /tmp [-w INT] [-c INT] [-S] [-N]"
    echo ""
    echo "Options:"
    echo "  -H/--hostname)"
    echo "     Defines the hostname. Default is: localhost"
    echo "  -P/--port)"
    echo "     Defines the port. Default is: 80"
    echo "  -p/--path-pid)"
    echo "     Path where nginx's pid file is being stored. You might need"
    echo "     to alter this path according to your distribution. Default"
    echo "     is: /var/run"
    echo "  -n/--name_pid)"
    echo "     Name of the pid file. Default is: nginx.pid"
    echo "  -N/--no-pid-check)"
    echo "     Turn this on, if you don't want to check for a pid file"
    echo "     whether nginx is running, e.g. when you're checking a"
    echo "     remote server. Default is: off"
    echo "  -s/--status-page)"
    echo "     Name of the server's status page defined in the location"
    echo "     directive of your nginx configuration. Default is:"
    echo "     nginx_status"
    echo "  -S/--secure)"
    echo "     In case your server is only reachable via SSL, use this"
    echo "     this switch to use HTTPS instead of HTTP. Default is: off"
    echo "  -w/--warning)"
    echo "     Sets a warning level for requests per second. Default is: off"
    echo "  -c/--critical)"
    echo "     Sets a critical level for requests per second. Default is:"
	echo "     off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --path-pid|-p)
            path_pid=$2
            shift
            ;;
        --name-pid|-n)
            name_pid=$2
            shift
            ;;
        --no-pid-check|-N)
            pid_check=0
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -ge ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
get_status() {
    if [ "$secure" = 1 ]
    then
        wget_opts="-O- -q -t 3 -T 3 --no-check-certificate"
        out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
        sleep 1
        out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
    else        
        wget_opts="-O- -q -t 3 -T 3"
        out1=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
        sleep 1
        out2=`wget ${wget_opts} http://${hostname}:${port}/${status_page}`
    fi
 
    if [ -z "$out1" -o -z "$out2" ]
    then
        echo "UNKNOWN - Local copy/copies of $status_page is empty."
        exit $ST_UK
    fi
}
 
get_vals() {
    tmp1_reqpsec=`echo ${out1}|awk '{print $10}'`
    tmp2_reqpsec=`echo ${out2}|awk '{print $10}'`
    reqpsec=`expr $tmp2_reqpsec - $tmp1_reqpsec`
 
    tmp1_conpsec=`echo ${out1}|awk '{print $9}'`
    tmp2_conpsec=`echo ${out2}|awk '{print $9}'`
    conpsec=`expr $tmp2_conpsec - $tmp1_conpsec`
 
    reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
    if [ "$reqpcon" = ".99" ]
    then
        reqpcon="1.00"
    fi
}
 
do_output() {
    output="nginx is running. $reqpsec requests per second, $conpsec \
connections per second ($reqpcon requests per connection)"
}
 
do_perfdata() {
    perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
if [ ${pid_check} = 1 ]
then
    check_pid
    if [ "$retval" = 1 ]
    then
        echo "There's no pid file for nginx. Is nginx running? Please \
also make sure whether your pid path and name is correct."
        exit $ST_CR
    fi
fi
 
get_status
get_vals
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$reqpsec" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} ]"
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

Jordi’s script

This is Jordi’s bash built-in (plus bc) version which runs even faster. Feel free to copy the output below.

#!/bin/bash
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
hostname="localhost"
port=80
path_pid=/var/run
name_pid="nginx.pid"
status_page="nginx_status"
pid_check=1
secure=0
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check whether nginx is running."
    echo "It also parses the nginx's status page to get requests and"
    echo "connections per second as well as requests per connection. You"
    echo "may have to alter your nginx configuration so that the plugin"
    echo "can access the server's status page."
    echo "The plugin is highly configurable for this reason. See below for"
    echo "available options."
    echo ""
    echo "$PROGNAME -H localhost -P 80 -p /var/run -n nginx.pid "
	echo "  -s nginx_statut -o /tmp [-w INT] [-c INT] [-S] [-N]"
    echo ""
    echo "Options:"
    echo "  -H/--hostname)"
    echo "     Defines the hostname. Default is: localhost"
    echo "  -P/--port)"
    echo "     Defines the port. Default is: 80"
    echo "  -p/--path-pid)"
    echo "     Path where nginx's pid file is being stored. You might need"
    echo "     to alter this path according to your distribution. Default"
    echo "     is: /var/run"
    echo "  -n/--name_pid)"
    echo "     Name of the pid file. Default is: nginx.pid"
    echo "  -N/--no-pid-check)"
    echo "     Turn this on, if you don't want to check for a pid file"
    echo "     whether nginx is running, e.g. when you're checking a"
    echo "     remote server. Default is: off"
    echo "  -s/--status-page)"
    echo "     Name of the server's status page defined in the location"
    echo "     directive of your nginx configuration. Default is:"
    echo "     nginx_status"
    echo "  -o/--output-directory)"
    echo "     Unneeded. The option is preserved for backwards compatibility."
    echo "  -S/--secure)"
    echo "     In case your server is only reachable via SSL, use this"
    echo "     this switch to use HTTPS instead of HTTP. Default is: off"
    echo "  -w/--warning)"
    echo "     Sets a warning level for requests per second. Default is: off"
    echo "  -c/--critical)"
    echo "     Sets a critical level for requests per second. Default is:"
	echo "     off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --hostname|-H)
            hostname=$2
            shift
            ;;
        --port|-P)
            port=$2
            shift
            ;;
        --path-pid|-p)
            path_pid=$2
            shift
            ;;
        --name-pid|-n)
            name_pid=$2
            shift
            ;;
        --no-pid-check|-N)
            pid_check=0
            ;;
        --status-page|-s)
            status_page=$2
            shift
            ;;
        --output-directory|-o)
            output_dir=$2
            shift
            ;;
        --secure|-S)
            secure=1
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -gt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
check_pid() {
    if [ -f "$path_pid/$name_pid" ]
    then
        retval=0
    else
        retval=1
    fi
}
 
get_status() {
    url="http://${hostname}:${port}/${status_page}"
 
    if [ "$secure" = 1 ] ; then
        wget_opts="-O- -q -t 3 -t 3 --no-check-certificate"
    else
        wget_opts="-O- -q -t 3 -t 3"
    fi
 
    out1=$(wget ${wget_opts} ${url})
    sleep 1
    out2=$(wget ${wget_opts} ${url})
 
    # Both commands must return some content.
    if [ -z "${out1}" -o -z "${out2}" ] ; then
        echo "UNKNOWN - Local copy/copies of $status_page is empty."
        exit $ST_UK
    fi
}
 
get_vals() {
    # Convert into an array.
    values1=(${out1})
    values2=(${out2})
 
    reqpsec=$(( ${values2[9]} - ${values1[9]} ))
    conpsec=$(( ${values2[8]} - ${values1[8]} ))
 
    reqpcon=`echo "scale=2; $reqpsec / $conpsec" | bc -l`
    if [ "$reqpcon" = ".99" ]
    then
        reqpcon="1.00"
    fi
}
 
do_output() {
    output="nginx is running. $reqpsec requests per second, $conpsec \
connections per second ($reqpcon requests per connection)"
}
 
do_perfdata() {
    perfdata="'reqpsec'=$reqpsec 'conpsec'=$conpsec 'conpreq'=$reqpcon"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
if [ ${pid_check} = 1 ]
then
    check_pid
    if [ "$retval" = 1 ]
    then
        echo "There's no pid file for nginx. Is nginx running? Please \
also make sure whether your pid path and name is correct."
        exit $ST_CR
    fi
fi
 
get_status
get_vals
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$reqpsec" -ge "$warning" -a "$reqpsec" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$reqpsec" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} ]"
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

License

As always this little script is ment to be sh-compliant and released under the terms of the GPL Version 2 only. Feel free to subscribe via rss to get updates on this one. More options will be added in the future.

How to fix NetworkManager’s OpenVPN plugin in Ubuntu 9.04

Mike Adolphs — Wed, 10 Jun 2009 13:07:06 +0000

The OpenVPN plugin for Gnome’s NetworkManager is somewhat broken, at least in various Ubuntu releases. Even with proper configuration it runs into an “NeedSecrets” error although the same command with the same configuration from shell via sudo work’s just fine.
But fear not mighty roadwarrior! Here’s a fix to let the NetworkManager get back to work.

Needless to say that this is just a temporary solution (a fix is in the making), but the only thing you have to do is to add an at_console policy to /etc/dbus-1/system.d/nm-openvpn-service.conf right between the two already existing policies.

	 user="at_console">
		 own="org.freedesktop.NetworkManager.vpnc"/>
		 send_destination="org.freedesktop.NetworkManager.vpnc"/>
	>

Then the file should look like the following:


 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
>
	 user="root">
		 own="org.freedesktop.NetworkManager.openvpn"/>
		 send_destination="org.freedesktop.NetworkManager.openvpn"/>
	>
	 user="at_console">
		 own="org.freedesktop.NetworkManager.vpnc"/>
		 send_destination="org.freedesktop.NetworkManager.vpnc"/>
	>
	 context="default">
		 own="org.freedesktop.NetworkManager.openvpn"/>
		 send_destination="org.freedesktop.NetworkManager.openvpn"/>
	>
>

Afterwards simply restart the NetworkManager via “sudo /etc/init.d/NetworkManager restart” and it’ll work. For Solutions for other VPN technologies you may also consider Ubuntu’s Launchpad.

Hadoop DFS Check Plugin for Nagios

Mike Adolphs — Tue, 02 Jun 2009 22:46:45 +0000

Since I’m currently occupied with kicking off a several terabytes large Hadoop cluster I thought it’d be a good idea to provide a few Nagios plugins related to Hadoop.

Just in case you don’t know: Hadoop is a free Java software framework that supports data intensive distributed applications. It got its own filesystem called HDFS which scales up to petabytes of storage running on top of the operating system’s filesystem. Hadoop is inspired by Google’s MapReduce and GFS, the Google File System.

Yahoo for example uses several Hadoop Clusters with 100.000 CPUs total in 20.000 boxes (2.000 boxes per cluster) according to the official Hadoop website. If you haven’t checked it out yet, do so! This is – sorry for that – crazy shit and it’s working!
For the beginning I’d recommend reading the Wikipedia article, the offical documentation and, of course, watching the Hadoop related lectures over at Cloudera.

Anyway, the first plugin is almost finished and checks the amount of running DataNodes and the capacity of the DFS. It’s working, but I want to add a few more features so that the plugin become more flexible. Nevertheless, feel free to give it a try if you’re in the mood.

The Script

Hadoop is completely operated by the hadoop user and since it got its own filesystem it got its own permissions as well. To keep things clean and safe we won’t give hadoop related permissions to the Nagios user. We’ll rather enable the Nagios user to run a small shell script as root which contains a command ran by the hadoop user to get information about the cluster.

Add the following small script to a directory of your choice, e.g. /usr/local/sbin:

#!/bin/sh
su -s /bin/bash - hadoop -c 'hadoop dfsadmin -report'

Afterwards alter its permissions so that it’s only read-, write and accessible by root:

user@host: ~ $ sudo chmod 700 /usr/local/sbin/get-dfsreport.sh

Then enable the Nagios user to sudo run the script via /etc/sudoers (or better visudo):

nagios          ALL=(ALL)       NOPASSWD: /usr/local/sbin/get-dfsreport.sh

This is it for the prerequisites. You may then run the script provided for copy’n'pasting below (or svn co, your choice).
If you’ve chosen another directory than /usr/local/sbin you have to provide the path via -s/–path-sh when running the script.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_hadoop-dfs/ check_hadoop-dfs/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
path_sh="/usr/local/sbin"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check the status of HDFS, Hadoop's"
    echo "underlying, redundant, distributed file system."
    echo ""
    echo "$PROGNAME -s /usr/local/sbin [-w 10] [-c 5]"
    echo ""
    echo "Options:"
    echo "  -s|--path-sh)"
    echo "     Path to the shell script that is mentioned in the"
    echo "     documentation. Default is: /usr/local/sbin"
    echo "  -w|--warning)"
    echo "     Defines the warning level for available datanodes. Default"
    echo "     is: off"
    echo "  -c|--critical)"
    echo "     Defines the critical level for available datanodes. Default"
    echo "     is: off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --path-sh|-s)
            path_sh=$2
            shift
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -lt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be higher than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
get_vals() {
    tmp_vals=`sudo ${path_sh}/get-dfsreport.sh`
    dn_avail=`echo -e "$tmp_vals" | grep -m1 "Datanodes available:" | awk '{print $3}'`
    dfs_used=`echo -e "$tmp_vals" | grep -m1 "Used raw bytes:" | awk '{print $4}'`
    dfs_used=`expr ${dfs_used} / 1024 / 1024`
    dfs_used_p=`echo -e "$tmp_vals" | grep -m1 "% used:" | awk '{print $3}'`
    dfs_total=`echo -e "$tmp_vals" | grep -m1 "Total raw bytes:" | awk '{print $4}'`
    dfs_total=`expr ${dfs_total} / 1024 / 1024`
}
 
do_output() {
    output="Datanodes up and running: ${dn_avail}, DFS total: \
${dfs_total} MB, DFS used: ${dfs_used} MB (${dfs_used_p})"
}
 
do_perfdata() {
    perfdata="'datanodes_available'=${dn_avail} 'dfs_total'=${dfs_total} \
'dfs_used'=${dfs_used}"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
get_vals
 
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$dn_avail" -le "$warning" -a "$dn_avail" -gt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$dn_avail" -le "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} "
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

Output example:

If everything went fine you should see output like the following:

user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 40 -c 30
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 60 -c 40
WARNING - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 70-c 60
CRITICAL - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 20 -c 40
Please adjust your warning/critical thresholds. The warning must be higher than the critical level!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 30
Please also set a critical value when you want to use warning/critical thresholds!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -c 30
Please also set a warning value when you want to use warning/critical thresholds!

The License

PNP Template for Process Check Plugin

Mike Adolphs — Tue, 02 Jun 2009 12:35:07 +0000

Just a short note:
Here’s the appropriate PNP Template to the process check plugin which I’ve recently written to create nice and shiny graphs:

#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

#   PNP Template for check_ps.sh
#   Author: Mike Adolphs (http://www.matejunkie.com/

$opt[1] = "--vertical-label \"percent\" -u 100 -l 0 -r --title \"CPU/Memory Usage for $hostname / $servicedesc\" ";
$opt[2] = "--vertical-label \"minutes\" -u 100 -l 0 -r --title \"cputime for $hostname / $servicedesc\" ";
 
$def[1] =  "DEF:cpu=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .=  "DEF:memory=$rrdfile:$DS[2]:AVERAGE " ;
$def[2] .=  "DEF:cputime=$rrdfile:$DS[3]:AVERAGE " ;
 
$def[1] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
$def[2] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
 
$def[1] .= "LINE2:cpu#E80C3E:\"CPU\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:cpu:MAX:\"%6.2lf \\n\" " ;
 
$def[1] .= "LINE2:memory#008000:\"Memory\\t\" " ;
$def[1] .= "GPRINT:memory:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:memory:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:memory:MAX:\"%6.2lf \\n\" " ;
 
$def[2] .= "AREA:cputime#E80C3E:\"CPUTime\\t\" " ;
$def[2] .= "GPRINT:cputime:LAST:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:AVERAGE:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:MAX:\"%6.2lf min\\n\" " ;

Process Check Plugin for Nagios

Mike Adolphs — Sat, 30 May 2009 17:07:24 +0000

This is a sh-compliant check plugin for Nagios which enables you to monitor a specific process whether it’s running or not and to see how much cpu/memory it’s wasting. This can be accomplished with the check_procs plugin which is included in the official Nagios-Plugins package as well, but it doesn’t generate any performance output. At least that’s what I’ve experienced lately (I’d have written the plugin anyway for training purposes).

The script

No special needs, just the basic stuff. The script parses the output of ps aux in different ways. You need to provide a specific process via -p/–process, if warning/critical thresholds are wanted you may also choose between cpu or memory thresholds via -t/–target (default is memory).
As always, you may grab the plugin at MonitoringExchange or directly from here via SVN and the copy’n'paste way below.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_ps/ check_ps/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
process="cron"
target="mem"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check a specific process via ps."
    echo "You may provide any string as an argument to match a specific"
    echo "process. Please note that the output could be distorted if the"
    echo "argument matches various processes, so please make sure to use"
    echo "unique strings to match a process."
    echo ""
    echo "$PROGNAME -p firefox [-w 10] [-c 20] [-t cpu]"
    echo ""
    echo "Options:"
    echo "  -p/--process)"
    echo "     You need to provide a string for which the ps output is then"
    echo "     then \"greped\"."
    echo "  -w/--warning)"
    echo "     Defines a warning level for a target which is explained"
    echo "     below. Default is: off"
    echo "  -c/--critical)"
    echo "     Defines a critical level for a target which is explained"
    echo "     below. Default is: off"
    echo "  -t/--target)"
    echo "     A target can be defined via -t. Choose between cpu and mem."
    echo "     Default is: mem"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --process|-p)
            process=$2
            shift
            ;;
        --target|-t)
            target=$2
            shift
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
        if [ ${warning} -gt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be lower than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
get_vals() {
    process=`echo ${process} | sed 's/^.\|[a-z][A-Z] /\[&]/g'`
 
    tmp_output=`ps aux | grep "$process" | grep -v $0`
 
    if [ -z "$tmp_output" ]
    then
        echo "CRITICAL - Process is not running!"
        exit $ST_CR
    fi
 
    ps_user=`echo ${tmp_output} | awk '{print $1}'`
    ps_pid=`echo ${tmp_output} | awk '{print $2}' `
    ps_cpu=`echo ${tmp_output} | awk '{print $3}'`
    ps_mem=`echo ${tmp_output} | awk '{print $4}' `
    ps_start=`echo ${tmp_output} | awk '{print $9}' `
 
    tmp_ps_cputime=`echo ${tmp_output} | awk '{print $10}'`
    tmp_ps_cpuhours=`echo ${tmp_ps_cputime} | awk -F \: '{print $1}'`
    tmp_ps_cpumin=`echo ${tmp_ps_cputime} | awk -F \: '{print $2}'`
    ps_cputime=`echo "scale=0; (${tmp_ps_cpuhours} * 60) + \
${tmp_ps_cpumin}" | bc -l`
}
 
do_wccalc() {
    if [ -n "$warning" -a -n "$critical" ]
    then
        if [ "$target" = "cpu" ]
        then
            tmp_wc_target=`echo ${ps_cpu} | awk -F \. '{print $2}'`
            if [ "$tmp_wc_target" -ge 5 ]
            then
                wc_target=`echo ${ps_cpu} | awk -F \. '{print $1}'`
                wc_target=`expr ${wc_target} + 1`
            else
                wc_target=`echo ${ps_cpu} | awk -F \. '{print $1}'`
            fi
                elif [ "$target" = "mem" ]
                then
            tmp_wc_target=`echo ${ps_mem} | awk -F \. '{print $2}'`
            if [ "$tmp_wc_target" -ge 5 ]
            then
                wc_target=`echo ${ps_mem} | awk -F \. '{print $1}'`
                wc_target=`expr ${wc_target} + 1`
            else
                wc_target=`echo ${ps_mem} | awk -F \. '{print $1}'`
            fi
        fi
    fi
}
 
do_output() {
    process=`echo ${process} | sed 's/\[//g' |  sed 's/\]//g'`
    output="Process: ${process}, User: ${ps_user}, CPU: ${ps_cpu}%, \
RAM: ${ps_mem}%, Start: ${ps_start}, CPU Time: ${ps_cputime} min"
}
 
do_perfdata() {
    perfdata="'cpu'=${ps_cpu} 'memory'=${ps_mem} 'cputime'=${ps_cputime}"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
get_vals
do_wccalc
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$wc_target" -ge "$warning" -a "$wc_target" -lt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
        exit $ST_WR
    elif [ "$wc_target" -ge "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
        exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata}"
        exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

PNP Template

#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

#   PNP Template for check_ps.sh
#   Author: Mike Adolphs (http://www.matejunkie.com/

$opt[1] = "--vertical-label \"percent\" -u 100 -l 0 -r --title \"CPU/Memory Usage for $hostname / $servicedesc\" ";
$opt[2] = "--vertical-label \"minutes\" -u 100 -l 0 -r --title \"cputime for $hostname / $servicedesc\" ";
 
$def[1] =  "DEF:cpu=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .=  "DEF:memory=$rrdfile:$DS[2]:AVERAGE " ;
$def[2] .=  "DEF:cputime=$rrdfile:$DS[3]:AVERAGE " ;
 
$def[1] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
$def[2] .= "COMMENT:\"\\t\\t\\tLAST\\t\\t\\tAVERAGE\\t\\t\\tMAX\\n\" " ;
 
$def[1] .= "LINE2:cpu#E80C3E:\"CPU\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:cpu:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:cpu:MAX:\"%6.2lf \\n\" " ;
 
$def[1] .= "LINE2:memory#008000:\"Memory\\t\" " ;
$def[1] .= "GPRINT:memory:LAST:\"%6.2lf %%\\t\\t\" " ;
$def[1] .= "GPRINT:memory:AVERAGE:\"%6.2lf \\t\\t\" " ;
$def[1] .= "GPRINT:memory:MAX:\"%6.2lf \\n\" " ;
 
$def[2] .= "AREA:cputime#E80C3E:\"CPUTime\\t\" " ;
$def[2] .= "GPRINT:cputime:LAST:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:AVERAGE:\"%6.2lf min\\t\\t\" " ;
$def[2] .= "GPRINT:cputime:MAX:\"%6.2lf min\\n\" " ;

The License

howto: Let nginx serve the Nagios web-interface

Mike Adolphs — Thu, 21 May 2009 15:33:45 +0000

For serving the Nagios web-interface it’s not necessary to run an Apache webserver. It also works quite well with others like lighttpd or nginx. Here’s how to configure nginx with basic auth so that Nagios knows who you are.

Prerequisites

Almost any package management system like apt pulls the Apache webserver as a dependency. I therefore strongly suggest you to compile and install Nagios manually since then you’ve got way more options to play with. Stick to the official documentation if you consider to do so. The following config files are all based on Nagios’ default configure options.

Get CGI going

nginx isn’t capable to serve dynamic content by default. You have to send this through a wrapper like we did in the last howto to get PHP and Passenger going. Therefore download a copy of the lighttpd source, unpack, configure and make it. Then copy the spawn-fcgi from the src directory to a local directory, e.g. /usr/bin:

user@host ~ $ wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
user@host ~ $ tar xvjf lighttpd-1.4.18.tar.bz2
user@host ~ $ cd lighttpd-1.4.18/
user@host ~/lighttpd-1.4.18 $ ./configure
user@host ~/lighttpd-1.4.18 $ make
user@host ~/lighttpd-1.4.18 $ sudo cp src/spawn-fcgi /usr/bin/spawn-fcgi

Then download the actual fcgiwrapper from github, unpack and make it. Afterwards copy the binary to a local directory, e.g. /usr/bin, as well:

user@host ~ $ wget http://github.com/gnosek/fcgiwrap/tarball/master
user@host ~ $ tar xzf gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec.tar.gz
user@host ~ $ cd gnosek-fcgiwrap-cdd6b8475744989940f9ce8322999e9ed4f0a6ec/
user@host ~ $ make
user@host ~ $ sudo mv fcgiwrap /usr/bin/
user@host ~ $ sudo chown root:root /usr/bin/fcgiwrap

Create a wrapper script to let the spawn-fcgi binary run a fcgiwrap instance and make it executable.

/usr/bin/c-fcgi.sh

#!/bin/sh
/usr/bin/spawn-fcgi -f /usr/bin/fcgiwrap -a 127.0.0.1 -p 49233 -P /var/run/fastcgi-c.pid -u www-data -g www-data

user@host: ~ $ sudo chmod +x /usr/bin/c-fcgi.sh

You may also create an init script to let the spawner run the fcgiwrap instance during boot.

/etc/init.d/c-fcgi

#!/bin/bash
C_SCRIPT=/usr/bin/c-fcgi
RETVAL=0
case "$1" in
	start)
		echo "Starting fastcgi"
		$C_SCRIPT
		RETVAL=$?
  ;;
	stop)
		echo "Stopping fastcgi"
		killall -9 fcgiwrap
		RETVAL=$?
  ;;
	restart)
		echo "Restarting fastcgi"
		killall -9 fcgiwrap
		$C_SCRIPT
		RETVAL=$?
  ;;
	*)
		echo "Usage: c-fastcgi {start|stop|restart}"
		exit 1
  ;;
esac
exit $RETVAL

Then start the spawner and put it in the default runlevel:

user@host ~ $ sudo update-rc.d c-fcgi defaults
user@host ~ $ sudo /etc/init.d/c-fcgi start

Configuring nginx

Assuming Nagios’ static html files reside in /usr/local/nagios/share and its cgi’s in /usr/local/nagios/sbin the configuration for an nginx vhost looks like the following:

server {
	listen   80;
	server_name  www.example.com;
 
	access_log  /var/www/example.com/logs/access.log;
 
	location / {
		root   /usr/local/nagios/share;
		index  index.html;
 
		rewrite	^/nagios/images/(.*)\.png /images/$1.png break;
 
		auth_basic		"Restricted";
		auth_basic_user_file	conf/htpasswd;
	}
 
	location ~ \.cgi$ {
		root	/usr/local/nagios/sbin;
		rewrite	^/nagios/cgi-bin/(.*)\.cgi /$1.cgi break;
 
		include /etc/nginx/fastcgi_params;
 
        	fastcgi_pass   127.0.0.1:49233;
	        fastcgi_param  SCRIPT_FILENAME  /usr/local/nagios$fastcgi_script_name;  # same path as above
 
		auth_basic		"Restricted";
		auth_basic_user_file	conf/htpasswd;
 
		fastcgi_param  AUTH_USER          $remote_user;
		fastcgi_param  REMOTE_USER	  $remote_user;
	}
}

Be sure to have the following fastcgi parameters configured in /etc/nginx/fastcgi_params.

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;
 
fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
 
fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
 
fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
 
# PHP only, required if PHP was built with --enable-force-cgi-redirect
#fastcgi_param  REDIRECT_STATUS    200;

Configuring Nagios

In /usr/local/nagios/etc/cgi.cfg you should turn use_authentication on:

use_authentication=1

Almost finished

After starting Nagios via its init script and a reload of your nginx webserver. it should show you an authentification dialogue window when you point your browser to the server’s url. To get authentication going, you need to provide credentials in /etc/nginx/conf/htpasswd in the following format:

user:password

Note that the password must be encrypted, plain text in this file wouldn’t work (for good reasons). You may want to have an Apache webserver around to use the htpasswd binary for that or you could use a solution from the web like the htpasswd generator, but keep in mind that using a service in the web may cause security troubles.

That should do the job. You’re now able to access your Nagios web-interface via nginx with proper authentication.

This guide also works for Icinga, a nice and shiny Nagios fork which came up two weeks ago. Simply change all names from nagios to icinga and you’re good to go.