Clickwatch Source

Intro


If you want to get up and running with Clickwatch on an ASP.NET server you should grab the download from the right column and skip down to the section that walks you through the installation process. If you're interested in the inner workings you should read on - it's probably simpler than you'd think.

To implement browser-based visualization of real-time HTTP traffic on a particular server, you need to solve the following problems:
  • 1) Gather event data in a web application
  • 2) Massage this data in your application so it is transformed into meaningful information - this step involves filtering the data gathered, translating IP addresses to geographic coordinates, etc.
  • 3) Deliver this data to a visualization client
  • 4) Display the data in an engaging fashion
Clickwatch has been implemented in ASP.NET but this doesn't mean it's not easy to port to other platforms. The backend code (steps 1-3 above) is very simple, and the same concepts would apply elsewhere.

The backend


Since we're talking about ASP.NET, the trivial way to intercept information about traffic as it's happening is to hook into IIS with an HttpModule. Ultimately we want to have the following information about every pageview:
  • Page being requested
  • Visitor IP address
  • Amount of time it took to process the request
  • The size of the server's response

ClickwatchHttpModule


An HttpModule provides a very versatile interface for post-processing (or pre-processing if you wish) requests that reach ASP.NET. There's a lot of documentation on the subject so I won't go into generic details; suffice to say that HttpModules are usually implemented in managed code (ours is a C# class library) and are registered with a simple entry in your web.config file. If you need more information on HttpModules, Google will help you.

The first two items (page URL and remote IP address) are readily available in the OnEndRequest event. To figure out the other two bits of information we need to hook into OnBeginRequest as well. In our OnBeginRequest handler, we store the current timestamp (so we can calculate the time taken once we get to OnEndRequest), and we also attach a simple passthrough filter to the Response object so we can count the number of bytes as they're being sent to the client. Unfortunately there's no way to simply query the size of the response in an OnEndRequest handler, or anywhere else in an HttpModule for that matter. Why Microsoft thought it wasn't important to implement Response.OutputStream.Length is beyond me - but the workaround is simple even though it involves about a hundred lines of mindless boilerplate code for the passthrough filter.

With the OnBeginRequest work done, once OnEndRequest is called we have almost all the information we need.

IpAddressToGeoInfoGeoIp


To translate the client's IP address to geographic information I chose to use a free database available from MaxMind. They also publish an open source library that can be used to query their binary databases, and it was easy to implement a COM wrapper around their C API, so the entire thing lives in IpAddressToGeoInfoGeoIp.dll. There's not much that's worth mentioning about the work done here: 99% of the code is boilerplate COM and the MaxMind library source.

It must be said that I am somewhat disappointed by the convoluted and less-than-optimal code that MaxMind makes available. I would have preferred to work with the database and API published by Ip2Location - their C library code isn't optimal either, but it's very simple and easy to change so it uses memory mapped files instead of fseek-ing a hojillion times per lookup. Ip2Location's DB is not free - but even if it were, their placenames are all capitalized which feels so 1980s, which gets us back to preferring MaxMind after all. But I digest.

The aforementioned COM object is instantiated and called upon by the OnEndRequest handler to translate the IP address to geographic information. Once we have this final piece, we package the information into an object and store it in an array that's also accessible from code running in a regular ASP.NET page.

clickwatch-refresh.aspx


This is the final piece to the backend. It simply parses the aforementioned array, and returns a simple textual representation of every object not yet sent to the calling client. This is an ASP.NET page with a whopping 30 lines of code inside. The visualization client needs to poll this script every few seconds to get a list of new hits to the site.

The Client


The client page is by far the most complex piece, over 1000 lines, but the good news is that it is completely portable. It is an .aspx page but the server-side script is limited to a couple of global variables that define the default size of the map.

The page is entirely self-contained. There are no JavaScript library dependencies or any sort of fancypants Flash/Silverlight/HTML5-Canvas wizardry. It's just plain JavaScript with lots of timers and dynamically created <img> objects. And therein lies the rub - the page may bog down under certain circumstances. Chrome has a very fast JavaScript engine but the display update code seems to be less than optimal, so enlarging the map using the gripper will cause high CPU loads even if you have a small number of dots flying around the screen. IE is at the other end of the spectrum: it will struggle mightily with a large number of objects even on a small map (when Chrome or Safari would still be happily updating at 40+ FPS) but it doesn't seem to be impacted by resizing the map to even something obscenely large.

Installing on an ASP.NET platform


It should be a matter of minutes to be up and running with Clickwatch if your server already runs ASP.NET. Grab the zipfile from the right-hand column, extract it to a temporary location and follow these instructions.

The IP address mapper COM module


Copy the geolookup.x86 or geolookup.x64 folder to its permanent location - there are no special requirements about where you put this. You need to choose the right platform for your server though. If your server is running 32-bit Windows you need the .x86 version, otherwise you'll want the .x64 one - unless you're running a 32-bit ASP.NET runtime on an x64 computer, which is very common, and works fine. So - know your ASP.NET platform and choose the appropriate COM object.

Once you've settled on a location for the geolookup COM module, go to MaxMind.com and download their free binary database. Once extracted you'll have a file called GeoLiteCity.dat. Move this file right next to the IpAddressToGeoInfoGeoIp.dll COM object, and rename it to IpAddressToGeoInfoGeoIp.dat.

Now open a command prompt, navigate to the directory where the COM object was placed, and type "regsvr32 IpAddressToGeoInfoGeoIp.dll". You should get a message indiating success. To test if everything works properly, double-click the testme.js file in the src\IpAddressToGeoInfoGeoIp directory of the zip file that you downloaded. If the COM object and the database have been installed properly, you'll get a message indicating the location of a certain IP address.

This was the hard part.

The web components


Copy everything in the wwwroot directory to your own web root, including the subdirectories. The contents of the bin directory must be in your own bin folder, etc. The only exception is web.config - if you already have a web.config file you should not overwrite your old one, but rather merge the contents of the two files. This should be straightforward.

Testing everything


If you now type http://your-server/clickwatch-refresh.aspx into a browser you should get a nice blank page, or a page with one or more lines of text. If you keep refreshing the page and it remains blank, it means that the HttpModule is not seeing any traffic from the Internet. If Clickwatch cannot resolve an IP address to a geographic location (as is the case with LAN addresses used on intranets) it will silently ignore it. In any case, if you got no error messages then everything seems to be fine - you can now try http://your-server/clickwatch.aspx. If you're not on the same LAN as your server, you should start seeing requests flying across the screen.

Why am I seeing just .aspx files?


Clickwatch will only see what's being handled by the ASP.NET engine. Plain files such as zip archives don't go through our HttpModule so they're invisible to us. You can fix this if you want - just map the extensions you're interested in to ASP.NET. For example, to see PDFs being downloaded in addition to your .aspx pages, right-click your website in IIS manager, select Properties, click the Configuration button near the bottom of the Home Directory tab, and add a mapping for the .pdf extension to the ASP.NET runtime. (Copy the "Executable Path" value from the .aspx mapping for best results.) You need to do this for every file type that you want to see in Clickwatch. And, obviously, you cannot remap other scripts such as legacy .asp or Perl pages to be handled by ASP.NET, so those will remain invisible.

And this, mapping plain files to ASP.NET, is where we may start having some problems. On the Windows 2003 servers that I've tested this with, a single hit on such a newly mapped file will appear exactly 101 times in Clickwatch. This appears to be a bug in the ASP.NET runtime, and the HttpHandler is being called a hundred times more than necessary. (There are a few mentions of this occurring out on the Internets, but there are no resolutions.) If you encounter this problem you will need to run your server in IIS 5.0 isolation mode. (Right-click the "Web Sites" root in IIS Manager, select the "Service" tab and choose "Run WWW service in IIS 5.0 isolation mode".) This fixes things for 32-bit servers, and 64-bit servers with the 64-bit ASP.NET runtime, but in a mixed environment (64-bit server, 32-bit ASP.NET) your website will most likely fail to start up with all kinds of nasty errors in the System log. It appears that IIS 5.0 isolation mode is not compatible with running the 32-bit ASP.NET runtime on 64-bit servers. To remedy this, switch to full 64-bit mode:
C:\>cscript.exe adsutil.vbs set W3SVC/AppPools/Enable32BitAppOnWin64 false
Then:
C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727>aspnet_regiis -i -enable
Start installing ASP.NET (2.0.50727).
......................
Finished installing ASP.NET (2.0.50727).

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727>iisreset

Attempting stop...
Internet services successfully stopped
Attempting start...
Internet services successfully restarted

C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727>
Note that above I've ran aspnet_regiis in the Framework64 directory as opposed to the plain 32-bit Framework one.

Contact


Questions, comments? Drop me a line at .

Please note that I don't have the time to do tech support for Clickwatch - but if you have ideas for improvements or ports to other platforms, I'd be very interested to know.