Sunday, December 30, 2007 12:12:40 PM (GMT Standard Time, UTC+00:00)

Picture the scene: It's the 23rd of December and I'm sat quietly with a bottle of wine, the remains of the chilli that my housemate and I had shared earlier and I was armed with Visual Studio. I was in a good coding mood.

I'd done quite a lot of code and I was interested in seeing how much work that actually was, so I headed over to Robstats to take a look at the past few days of carnage to satisfy my curiousity, because I could. Except I couldn't because for some reason the server was being very slow and the data hadn't been updated for some reason.

I remoted in, expecting that perhaps the doomsday had finally come and Robstats had finally exceeded its capability of data crunching after 6 months of data. I was correct, but I didn't realise to the extent that this was going to kill muy web server. Half a million xml files storing 6 months data, things were bad. I knew when I was writing it that it was going to be bad, I just expected to have its replacement completed a long time ago.

The generation utility was eating memory (200mb on a 256mb server) and it was only halfway through its cycle. The server was groaning under the sheer weight of data and I had to stop it before I lost communication with the server altogether. Too late! I was disconnected and thus began the next few days of woe.

The crash caused a complete memory dump, there wasn't enough resources to create that memory dump and the server completely locked up. I popped off a support request to SynergyWorks and they gave me a hard-reboot. Cue file corruption. Cue explorer corruption. Cue lots of corruption. Over the next few days, things kept crashing and locking up and I had to ask SynergyWorks to reboot several times - just making the problem worse.

I decided to complete Lolstats and Scrobbles, the proper version of all this. Lolstats: A multi-user extensible and very generic stats crunching backend with a fairly flexible xml based data-transformation and presentation layer; and Scrobbles: A public website and frontend for managing your lolstats. 

The only thing preventing me from completing it was that I was focusing on the multi-user and ease of use side of things. It was taking up too much time without enough personal reward or gain. I suddenly had a reason to do it for myself and I got cracking on the bits that would be important to just display the data that Robstats had collected.

I got it working and by the 29th of December the server was completely dead. I got in touch with SynergyWorks and they decided that it would be best to give me a clean server, double the ram on the machine and mount the old data as a disk so I could recover what I could. Not bad support for a Saturday afternoon eh? </Shameless plug>

I went through the checklist and backed up the websites and data, and began an XCOPY of the half million odd xml files that constituted the 'backend' of Robstats. I had a migration utility and I was prepared to use it. Half a million xml files into half a million database tables, ready to be crunched into less than 20,000 records!

Some files were corrupt, some files had missing data because the format of the Robstats backend had changed over the months that the service had been up. I kept tweaking the migration utility until it could finally process all half million files and chuck them into the Scrobbles backend "processing queue". Actually converting the data and adding it to the database only took about 20 minutes, so it wasn't that painful.

As I write this blog post, the Lolstats backend is crunching the data into a more managable form. I've written the windows background tasks and Visual Studio 2005/2008 addin to capture the data and send it to Scrobbles instead of Robstats. The backend is processing about 1000 records every minute and a half, which means with a few improvements the current server should be able to deal with a few hundred users. It's going to take 10 hours to crunch the six months data into those hourly chunks and the pages on the site are now getting generated dynamically per-request (Yes, there is some complex caching going on too) through the Lolstats data presentation system.

I'll be documenting the web services and documenting about the xml format needed to pull the arbitrary data out and display it sometime over the next month. That should appear on the Lolstats domain once I've set that up.

I first need to write a decent website around the backend and provide a front-end for editing the pages being displayed. There is a lot of potential for third-party development and a lot of potential for a lot of data capture systems (games and various). I'd also quite like to use the Lolstats backend to capture data about the Scrobbles website and services. "Because I can".

I also need to purchase some decent graphing software. The freeware I'm currently using to generate graphs is nice and everything, but it's not really suitable for moving forward. If you want to be a part of the beta program, e-mail me on robashton@codeofrob.com and I'll consider you. I'd prefer to only have people who I know won't need help getting things set up. I haven't got some for support just yet.