Update: One invitation left.

Next Update: Again 10 invitations for your delight! ;-)

Just in case you don’t know: Hadoop is a free Java software framework that supports data intensive distributed applications. It got its own filesystem called HDFS which scales up to petabytes of storage running on top of the operating system’s filesystem. Hadoop is inspired by Google’s MapReduce and GFS, the Google File System.

Yahoo for example uses several Hadoop Clusters with 100.000 CPUs total in 20.000 boxes (2.000 boxes per cluster) according to the official Hadoop website. If you haven’t checked it out yet, do so! This is – sorry for that – crazy shit and it’s working!
For the beginning I’d recommend reading the Wikipedia article, the offical documentation and, of course, watching the Hadoop related lectures over at Cloudera.

Anyway, the first plugin is almost finished and checks the amount of running DataNodes and the capacity of the DFS. It’s working, but I want to add a few more features so that the plugin become more flexible. Nevertheless, feel free to give it a try if you’re in the mood.

The Script

Hadoop is completely operated by the hadoop user and since it got its own filesystem it got its own permissions as well. To keep things clean and safe we won’t give hadoop related permissions to the Nagios user. We’ll rather enable the Nagios user to run a small shell script as root which contains a command ran by the hadoop user to get information about the cluster.

Add the following small script to a directory of your choice, e.g. /usr/local/sbin:

#!/bin/sh
su -s /bin/bash - hadoop -c 'hadoop dfsadmin -report'

Afterwards alter its permissions so that it’s only read-, write and accessible by root:

user@host: ~ $ sudo chmod 700 /usr/local/sbin/get-dfsreport.sh

Then enable the Nagios user to sudo run the script via /etc/sudoers (or better visudo):

nagios          ALL=(ALL)       NOPASSWD: /usr/local/sbin/get-dfsreport.sh

This is it for the prerequisites. You may then run the script provided for copy’n'pasting below (or svn co, your choice).
If you’ve chosen another directory than /usr/local/sbin you have to provide the path via -s/–path-sh when running the script.

user@host: ~ $ svn co svn://svn.matejunkie.com/nagios-plugins/stable/check_hadoop-dfs/ check_hadoop-dfs/

#!/bin/sh
 
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
PROGNAME=`basename $0`
VERSION="Version 1.0,"
AUTHOR="2009, Mike Adolphs (http://www.matejunkie.com/)"
 
ST_OK=0
ST_WR=1
ST_CR=2
ST_UK=3
 
path_sh="/usr/local/sbin"
 
print_version() {
    echo "$VERSION $AUTHOR"
}
 
print_help() {
    print_version $PROGNAME $VERSION
    echo ""
    echo "$PROGNAME is a Nagios plugin to check the status of HDFS, Hadoop's"
    echo "underlying, redundant, distributed file system."
    echo ""
    echo "$PROGNAME -s /usr/local/sbin [-w 10] [-c 5]"
    echo ""
    echo "Options:"
    echo "  -s|--path-sh)"
    echo "     Path to the shell script that is mentioned in the"
    echo "     documentation. Default is: /usr/local/sbin"
    echo "  -w|--warning)"
    echo "     Defines the warning level for available datanodes. Default"
    echo "     is: off"
    echo "  -c|--critical)"
    echo "     Defines the critical level for available datanodes. Default"
    echo "     is: off"
    exit $ST_UK
}
 
while test -n "$1"; do
    case "$1" in
        -help|-h)
            print_help
            exit $ST_UK
            ;;
        --version|-v)
            print_version $PROGNAME $VERSION
            exit $ST_UK
            ;;
        --path-sh|-s)
            path_sh=$2
            shift
            ;;
        --warning|-w)
            warning=$2
            shift
            ;;
        --critical|-c)
            critical=$2
            shift
            ;;
        *)
            echo "Unknown argument: $1"
            print_help
            exit $ST_UK
            ;;
        esac
    shift
done
 
get_wcdiff() {
    if [ ! -z "$warning" -a ! -z "$critical" ]
    then
        wclvls=1
 
        if [ ${warning} -lt ${critical} ]
        then
            wcdiff=1
        fi
    elif [ ! -z "$warning" -a -z "$critical" ]
    then
        wcdiff=2
    elif [ -z "$warning" -a ! -z "$critical" ]
    then
        wcdiff=3
    fi
}
 
val_wcdiff() {
    if [ "$wcdiff" = 1 ]
    then
        echo "Please adjust your warning/critical thresholds. The warning \
must be higher than the critical level!"
        exit $ST_UK
    elif [ "$wcdiff" = 2 ]
    then
        echo "Please also set a critical value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    elif [ "$wcdiff" = 3 ]
    then
        echo "Please also set a warning value when you want to use \
warning/critical thresholds!"
        exit $ST_UK
    fi
}
 
get_vals() {
    tmp_vals=`sudo ${path_sh}/get-dfsreport.sh`
    dn_avail=`echo -e "$tmp_vals" | grep -m1 "Datanodes available:" | awk '{print $3}'`
    dfs_used=`echo -e "$tmp_vals" | grep -m1 "Used raw bytes:" | awk '{print $4}'`
    dfs_used=`expr ${dfs_used} / 1024 / 1024`
    dfs_used_p=`echo -e "$tmp_vals" | grep -m1 "% used:" | awk '{print $3}'`
    dfs_total=`echo -e "$tmp_vals" | grep -m1 "Total raw bytes:" | awk '{print $4}'`
    dfs_total=`expr ${dfs_total} / 1024 / 1024`
}
 
do_output() {
    output="Datanodes up and running: ${dn_avail}, DFS total: \
${dfs_total} MB, DFS used: ${dfs_used} MB (${dfs_used_p})"
}
 
do_perfdata() {
    perfdata="'datanodes_available'=${dn_avail} 'dfs_total'=${dfs_total} \
'dfs_used'=${dfs_used}"
}
 
# Here we go!
get_wcdiff
val_wcdiff
 
get_vals
 
do_output
do_perfdata
 
if [ -n "$warning" -a -n "$critical" ]
then
    if [ "$dn_avail" -le "$warning" -a "$dn_avail" -gt "$critical" ]
    then
        echo "WARNING - ${output} | ${perfdata}"
	exit $ST_WR
    elif [ "$dn_avail" -le "$critical" ]
    then
        echo "CRITICAL - ${output} | ${perfdata}"
	exit $ST_CR
    else
        echo "OK - ${output} | ${perfdata} "
	exit $ST_OK
    fi
else
    echo "OK - ${output} | ${perfdata}"
    exit $ST_OK
fi

Output example:

If everything went fine you should see output like the following:

user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 40 -c 30
OK - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 60 -c 40
WARNING - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 70-c 60
CRITICAL - Datanodes up and running: 50, DFS total: 20147365 MB, DFS used: 0 MB (0%) | 'datanodes_available'=50 'dfs_total'=20147365 'dfs_used'=0
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 20 -c 40
Please adjust your warning/critical thresholds. The warning must be higher than the critical level!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -w 30
Please also set a critical value when you want to use warning/critical thresholds!
user@host: ~ $ ./check_hadoop-dfs.sh -s /var/nagios/home/bin -c 30
Please also set a warning value when you want to use warning/critical thresholds!

The License

As always this little script is ment to be sh-compliant and released under the terms of the GPL Version 2 only. Feel free to subscribe via rss to get updates on this one. More options will be added in the future.

For example, searching for Linux in Bing shows Microsoft product placement within the search suggestions. Though this lets every IT professional grin from ear to ear, it could have major impact on “not so professional” people when they want to see what “this so called Linux” is about and rely on search suggestions.

Google instead shows suggestions that are pretty useful, when someone wants to get in touch with Linux.
In general it seems that Google tries to avoid product placement within the suggestions. Searching for “reader” for example doesn’t show any related content to Google Reader.

The same applies to the search for SQL. On the Bing hand it shows product placement for Microsoft’s SQL related products, Google on its hand offers way more universal suggestions.

Don’t get me wrong. Although I’m an open source addict, I’m not a Microsoft hater. I’d actually use Microsoft software in several areas if I have to. SQL Server 2008 for example is an awesome piece of software, but when it comes to the Internet they still seem to struggle.
If I’m searching for something I want to get to the result and not being bothered by subtle advertising and I definitely want to get a proper, almost universally valid search result, of course. Google makes this just right and I guess that’ll last a couple more years since there’s still no real competitor in the race. They still claim that search is their main product, of course.

Google’s power is simplicity, Microsoft makes it too complicated. It’s that easy.

The background picture at bing.com that changes every 24 hours and comes with several overlays to get additional information is a really nice idea to pass time and actually makes Bing more pleasant. How often do I browse the wikipedia clicking from one article to another to learn stuff I haven’t even heard of last week? Two times a week at least.
But why do I have to install Silverlight when I want to go back to last days picture? Couldn’t this have been done without the need to install additional software? This way it’s kinda obvious that Microsoft wants to push its alternative to Flash through their search engine and as I said, it’s all about the results and not about binding people to its own solutions.

The Internet is all about interoperability. Something that Microsoft obviously still don’t get and that’s sad because a little more competition in the search engine business would be quite nice.

Click here to view the embedded video.

Besides a lot of cool infrastructure features, I especially like the “Google provided personal transportation device” part, but that’s just because of my nerdiness. I apologize.

The advantages are obvious:

• You don’t have to care about setting up and maintaining your own mail infrastructure
  • that includes proper spam protection
• No need of a backup solution to archive E-Mails
• Access to a bunch of other features like Google Calender or Docs
• Easy setup for using E-Mail and other Google Aps on your mobile
• Reachable almost anytime, few downtimes
• Accessing E-Mails via Web-Interface, IMAP, POP3
• Sending E-Mails from anywhere via SMTP

But where’s light, there’s also shadow (the disadvantages):

• Less privacy due to the fact that E-Mails are getting parsed to provide ads that fit to the content
• You’re bound to the service provider
• Terms might get changed in the future to your disadvantage what might cause a migration for you
• Your data is stored externally
• If your domain is down, you won’t be able to receive E-Mails anymore (especially annoying when using low TTLs or really annoying when you configure bullshit with high TTLs as well)

Anyway, the important privacy part: As we all know, Google has access to a bunch of datasets from almost any of us. So if you’re in need of exchanging really important information, don’t do it the Google way!
For all other cases, e.g. using E-Mail to get informed when new comments have been made in your blog, trying out new web 2.0 services or reading several newsgroups, I’m fine with the fact, that Google parses every E-Mail to provide proper ads that fit to the content. This is done automatically as described in the privacy statement, but if you’re unsure whether it really fits your opinion of privacy, better take a look by yourself (and use other methods of communication to exchange really important information).

In general there’re two options available: A freely available standard and a charged premium one. The standard package includes all available Google applications like Mail, Calender, Sites or Doc, provides 7,3 GB of space for mail and another 10 GB of space for Google Sites storage. The premium package comes with 25 GB of space for mail, a guaranteed uptime of 99,9% per month, more support options, optional archiving and security solutions by Postini and API support for 50 US$ per year and user. Both packages are able to be used via SSL.
For my purposes the standard package was just fine. I don’t have tons of old E-Mails, don’t need archiving or additional spam utilities, but I’m really in need of solutions for mobile use via Blackberry or iPhone without spending evenings to maintain my own E-Mail infrastructure. That’s just not worth the time.

The sign up process and the initial configuration within the Google environment is pretty straight-forward. Be aware of the fact, that you must have the ability to alter your own DNS configuration. During the sign up you have to add a custom CNAME which is checked by Google to authorize the whole process. To be able to get E-Mails you also have to add Google’s 7 MX records to your zone file. If you’re unsure how to to that, I’d strongly suggest you to stick to a free webmailer …

To send E-Mails via your Google Mail account directly from your server, you’re in need of a MTA like SSMTP which is included in almost any Linux distribution today. Its configuration is pretty easy as you can see below:

user@host:~$ sudo cat /etc/ssmtp/ssmtp.conf
root=postmaster
mailhub=smtp.googlemail.com
hostname=hostname.domain.tld
 
FromLineOverride=YES
 
UseSTARTTLS=YES
AuthUser=name@domain.tld
AuthPass=password

user@host:~$ sudo cat /etc/ssmtp/revaliases
root:name@domain.tld:smtp.googlemail.com:587
user:name@domain.tld:smtp.googlemail.com:587

Therefore I took a look at the IX’s blacklist project NiX Spam and downloaded a snapshot of the blacklist’s content (save, right click as… you’ve been warned!) which includes a maximum of 40.000 IP addresses. The plain text file comes with a timestamp and the blacklisted IP addresses.
After some more or less simple batch-processing I’ve had a file including all the relevant GeoIP information, the longitude and latitude values of each IP. What was missing was a method to display them in a graphical way, e.g. on a world map.

Couldn’t be Google Maps a solution with its well documented and comprehensive API? So I digged into the Google Maps API and after an hour I was able to change the batch-processing in a way to create the proper output for creating markers on a map, including the well-know tooltip balloons including IP address and timestamp after clicking on a marker.

First tests with a few dozens data sets went just fine, but in this very special moment right before crossing the finish line, my browser decided to stop working for a couple of minutes.
It turned out that 40.000 markers, combined with a tooltip, was way too much data to be handled by the browser. Navigating or zooming the map just wasn’t possible. Even after I’ve reduced the amount of data sets to 10.000 it was slow as hell.

Besides the performance issues it turned out that the visualization is almost useless because of the many overlapping markers as you can see on the left. Spammer mostly use botnets,  thousands of Zombie PCs, distributed all over the world.
Sure, it is possible to see areas with a much higher density than other regions, but these are also the regions where internet is accessible from almost everywhere. It’s easily explainable that there are less Spammers in the australian outback or South America’s jungle.

But each drawback has also its advantages. Besides the fact that I gained more knowledge of vim, sed and awk once more, it won’t be too hard to complete this.
With a slightly different approach, e.g. abandoning the longitude/latitude data and prefering the city’s name, it’d be possible to present shrinking or growing markers based on the amount of machines in a specific city. This would drastically reduce the amount of markers and is not that hard to implement.