<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Matejunkie &#187; geoip</title>
	<atom:link href="http://www.matejunkie.com/tag/geoip/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.matejunkie.com</link>
	<description>"Look behind you, a Three-Headed Monkey!"</description>
	<lastBuildDate>Thu, 07 Jan 2010 14:26:10 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.0</generator>
		<item>
		<title>Visualizing spam</title>
		<link>http://www.matejunkie.com/visualizing-spam/</link>
		<comments>http://www.matejunkie.com/visualizing-spam/#comments</comments>
		<pubDate>Sun, 08 Feb 2009 03:10:39 +0000</pubDate>
		<dc:creator>Mike Adolphs</dc:creator>
				<category><![CDATA[Binary Talks]]></category>
		<category><![CDATA[api]]></category>
		<category><![CDATA[geoip]]></category>
		<category><![CDATA[google]]></category>
		<category><![CDATA[google maps]]></category>
		<category><![CDATA[performance]]></category>
		<category><![CDATA[spam]]></category>

		<guid isPermaLink="false">http://www.matejunkie.net//?p=686</guid>
		<description><![CDATA[After yesterday&#8217;s bauble with geoiplookup I was thinking about a more appropriate use of looking up the location of an IP. So what&#8217;s better than visualizing the areas of the world where most spam comes from, I guessed. Therefore I took a look at the IX&#8217;s blacklist project NiX Spam and downloaded a snapshot of [...]]]></description>
			<content:encoded><![CDATA[<p>After yesterday&#8217;s bauble with geoiplookup I was thinking about a more appropriate use of looking up the location of an IP. So what&#8217;s better than visualizing the areas of the world where most spam comes from, I guessed.</p>
<p>Therefore I took a look at the IX&#8217;s blacklist project <a title="NiX Spam" href="http://www.heise.de/ix/nixspam/dnsbl_en/" target="_blank">NiX Spam</a> and downloaded a snapshot of the <a title="NiX Spam Blacklist snapshot" href="http://www.ix.de/nixspam/nixspam.blackmatches" target="_blank">blacklist&#8217;s content</a> (save, right click as&#8230; you&#8217;ve been warned!) which includes a maximum of 40.000 IP addresses. The plain text file comes with a timestamp and the blacklisted IP addresses.<br />
After some more or less simple batch-processing I&#8217;ve had a file including all the relevant GeoIP information, the longitude and latitude values of each IP. What was missing was a method to display them in a graphical way, e.g. on a world map.</p>
<p>Couldn&#8217;t be Google Maps a solution with its well documented and comprehensive <a title="Google Maps API documentation" href="http://code.google.com/apis/maps/" target="_blank">API</a>? So I digged into the Google Maps API and after an hour I was able to change the batch-processing in a way to create the proper output for creating markers on a map, including the well-know tooltip balloons including IP address and timestamp after clicking on a marker.</p>
<p>First tests with a few dozens data sets went just fine, but in this very special moment right before crossing the finish line, my browser decided to stop working for a couple of minutes.<br />
It turned out that 40.000 markers, combined with a tooltip, was way too much data to be handled by the browser. Navigating or zooming the map just wasn&#8217;t possible. Even after I&#8217;ve reduced the amount of data sets to 10.000 it was slow as hell.</p>
<p><a href="http://www.matejunkie.com/wp-content/uploads/2009/02/googleearth_10000markers.png"><img class="size-thumbnail wp-image-688 alignleft" style="margin-left: 6px; margin-right: 6px;" title="Google Maps - 10.000 markers" src="http://www.matejunkie.com/wp-content/uploads/2009/02/googleearth_10000markers-150x150.png" alt="Google Maps - 10.000 markers" width="200" height="200" /></a>Besides the performance issues it turned out that the visualization is almost useless because of the many overlapping markers as you can see on the left. Spammer mostly use botnets,  thousands of Zombie PCs, distributed all over the world.<br />
Sure, it is possible to see areas with a much higher density than other regions, but these are also the regions where internet is accessible from almost everywhere. It&#8217;s easily explainable that there are less Spammers in the australian outback or South America&#8217;s jungle.</p>
<p>But each drawback has also its advantages. Besides the fact that I gained more knowledge of vim, sed and awk once more, it won&#8217;t be too hard to complete this.<br />
With a slightly different approach, e.g. abandoning the longitude/latitude data and prefering the city&#8217;s name, it&#8217;d be possible to present shrinking or growing markers based on the amount of machines in a specific city. This would drastically reduce the amount of markers and is not that hard to implement.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.matejunkie.com/visualizing-spam/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>GeoIP Lookups</title>
		<link>http://www.matejunkie.com/geoip-lookups/</link>
		<comments>http://www.matejunkie.com/geoip-lookups/#comments</comments>
		<pubDate>Sat, 07 Feb 2009 06:34:16 +0000</pubDate>
		<dc:creator>Mike Adolphs</dc:creator>
				<category><![CDATA[Binary Talks]]></category>
		<category><![CDATA[debian]]></category>
		<category><![CDATA[dns]]></category>
		<category><![CDATA[geoip]]></category>
		<category><![CDATA[linux]]></category>
		<category><![CDATA[maxmind]]></category>
		<category><![CDATA[sysadmin]]></category>
		<category><![CDATA[ubuntu]]></category>

		<guid isPermaLink="false">http://www.matejunkie.net//?p=667</guid>
		<description><![CDATA[If you want to know the location of a server connected to the Internet, you need a tiny, useful tool called geoiplookup. In Debian or Ubuntu the installation is quite easy: sudo aptitude install geoip-bin After installation you&#8217;re able to use geoiplookup from the command line: user@server:~$ geoiplookup www.heise.de GeoIP Country Edition: DE, Germany But [...]]]></description>
			<content:encoded><![CDATA[<p>If you want to know the location of a server connected to the Internet, you need a tiny, useful tool called geoiplookup. In Debian or Ubuntu the installation is quite easy:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #c20cb9; font-weight: bold;">sudo</span> <span style="color: #c20cb9; font-weight: bold;">aptitude</span> <span style="color: #c20cb9; font-weight: bold;">install</span> geoip-bin</pre></div></div>

<p>After installation you&#8217;re able to use geoiplookup from the command line:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;">user<span style="color: #000000; font-weight: bold;">@</span>server:~$ geoiplookup www.heise.de
GeoIP Country Edition: DE, Germany</pre></div></div>

<p>But that&#8217;ll only show you the country of the server&#8217;s location. For more precise information <a title="MaxMind" href="http://www.maxmind.com/" target="_blank">MaxMind</a>&#8216;s <a title="MaxMind's LGPL GeoIP database license" href="http://geolite.maxmind.com/download/geoip/database/LICENSE.txt" target="_blank">LGPL version</a> of their GeoIP database comes in handy. It&#8217;s being updated at the beginning of each month and available via <a title="MaxMind's LGPL GeoIP database" href="http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz" target="_blank">this link</a> (approx. 30 MB).</p>
<p>Simply download the archive, gunzip it and move it to the proper location:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #c20cb9; font-weight: bold;">gunzip</span> GeoLiteCity.dat.gz
<span style="color: #c20cb9; font-weight: bold;">mv</span> GeoLiteCity.dat <span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>share<span style="color: #000000; font-weight: bold;">/</span>GeoIP<span style="color: #000000; font-weight: bold;">/</span>GeoIPCity.dat</pre></div></div>

<p>After that, geoiplookup returns more accurate location details on a city-level:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;">user<span style="color: #000000; font-weight: bold;">@</span>server:~$ geoiplookup www.heise.de
GeoIP Country Edition: DE, Germany
GeoIP City Edition, Rev <span style="color: #000000;">1</span>: DE, 06, Hanover, <span style="color: #7a0874; font-weight: bold;">&#40;</span>null<span style="color: #7a0874; font-weight: bold;">&#41;</span>, <span style="color: #000000;">52.366699</span>, <span style="color: #000000;">9.716700</span>, <span style="color: #000000;">0</span>, <span style="color: #000000;">0</span>
GeoIP City Edition, Rev <span style="color: #000000;">0</span>: DE, 06, Hanover, <span style="color: #7a0874; font-weight: bold;">&#40;</span>null<span style="color: #7a0874; font-weight: bold;">&#41;</span>, <span style="color: #000000;">52.366699</span>, <span style="color: #000000;">9.716700</span></pre></div></div>

<p>Tossing longitude and latitude to Google Maps it&#8217;ll show the location of the Heise Verlag&#8217;s office, although the servers are running in Frankfurt/Main 350 kilometers south of Hannover.</p>
<div align="center"><iframe width="425" height="350" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" src="http://maps.google.com/maps?f=q&amp;source=s_q&amp;hl=en&amp;geocode=&amp;q=52.366699,+9.716700&amp;sll=51.206883,9.519653&amp;sspn=2.557072,4.053955&amp;ie=UTF8&amp;t=h&amp;s=AARTsJqtAGWWoTH8WVT5d-7AcAk-mwScew&amp;ll=52.372036,9.723501&amp;spn=0.018341,0.036478&amp;z=14&amp;iwloc=addr&amp;output=embed"></iframe><br /><small><a href="http://maps.google.com/maps?f=q&amp;source=embed&amp;hl=en&amp;geocode=&amp;q=52.366699,+9.716700&amp;sll=51.206883,9.519653&amp;sspn=2.557072,4.053955&amp;ie=UTF8&amp;t=h&amp;ll=52.372036,9.723501&amp;spn=0.018341,0.036478&amp;z=14&amp;iwloc=addr" style="color:#0000FF;text-align:left">View Larger Map</a></small></div>
<p>If you want to keep the information up to date you should create a small script, triggered by a cronjob running at 1:30am on the third day of each month.</p>
<p>The cronjob, assuming you define cronjobs via /etc/crontab:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #000000;">30</span> <span style="color: #000000;">1</span> <span style="color: #000000;">3</span> <span style="color: #000000; font-weight: bold;">*</span> <span style="color: #000000; font-weight: bold;">*</span> root <span style="color: #000000; font-weight: bold;">/</span>root<span style="color: #000000; font-weight: bold;">/</span>scripts<span style="color: #000000; font-weight: bold;">/</span>update_geoip_db.sh</pre></div></div>

<p>The &#8220;script&#8221;:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/bin/bash</span>
&nbsp;
<span style="color: #c20cb9; font-weight: bold;">wget</span> <span style="color: #660033;">-t</span> <span style="color: #000000;">5</span> <span style="color: #ff0000;">&quot;http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz&quot;</span>
<span style="color: #c20cb9; font-weight: bold;">gunzip</span> <span style="color: #660033;">-f</span> <span style="color: #000000; font-weight: bold;">/</span>root<span style="color: #000000; font-weight: bold;">/</span>scripts<span style="color: #000000; font-weight: bold;">/</span>GeoLiteCity.dat.gz
<span style="color: #c20cb9; font-weight: bold;">mv</span> <span style="color: #000000; font-weight: bold;">/</span>root<span style="color: #000000; font-weight: bold;">/</span>scripts<span style="color: #000000; font-weight: bold;">/</span>GeoLiteCity.dat <span style="color: #000000; font-weight: bold;">/</span>usr<span style="color: #000000; font-weight: bold;">/</span>share<span style="color: #000000; font-weight: bold;">/</span>GeoIP<span style="color: #000000; font-weight: bold;">/</span>GeoIPCity.dat</pre></div></div>

<p>On the console you usually won&#8217;t get more detailed information, because MaxMind doesn&#8217;t include all their knowledge in the LGPL version of their GeoIP databases. In case you need additional information you might want to use MaxMind&#8217;s freely accessible <a title="GeoIP lookup web-interface" href="http://www.maxmind.com/app/locate_ip" target="_blank">web-interface</a> which uses some of the purchasable databases as its backend.</p>
<p>But don&#8217;t expect to get information on a district, street or even house number level. That&#8217;s simply not possible due to the common IP address block allocation and the fact that most of the information is gathered from whois data, which mostly doesn&#8217;t include the &#8220;end user&#8217;s&#8221; location rather than the ISP&#8217;s office address.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.matejunkie.com/geoip-lookups/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
