Friday, October 24, 2008 1:04:51 PM (GMT Standard Time, UTC+00:00)

I picked up JsonFx yesterday as I could really do with a more.. declarative way of creating client-side controls when piddling about with Ajax for a browser-application type thing I am working on. JsonFX seems to provide a nice JBST system, along with data binding and support for compacting scripts/files on the fly, so I'm quite excited about getting into it.

I've had a few difficulties getting it working however, I just kept getting the glorious message "Error loading "~/scripts/sdctls/controls.merge". Either not found or a build error occurred." with no other information supplied. At the time of writing, this error message cannot be found on Google.

The actual JsonFX site is seemingly scant on documentation, and the starter kit was throwing the same wobbly, so I determined that it was either the way I was running the site, or something to do with the environment

Downloading the source for JsonFX and stepping through it, revealed a dependency on the MSScriptControl ActiveX component, presumably for validating scripts before shrinking them and sending them across the wire. Googling for the MSScriptControl dependency came up null though, for some absolutely bizarre reason, perhaps I am just fail at googling today.

Supposedly on not finding the component, an error is supposed to be added to the error handling system in JsonFX and therefore displayed, telling the developer to go download it, but this was not happening.

The ActiveX component can be found here, the Windows Script Control

Problem solved, I've got everything else working by looking at the rather excellent self-documenting starter-kit. If I come across any other gotchas I guess I'll post them, as this library seems rather new and there is very little on the internet about it

Sunday, September 28, 2008 1:22:40 PM (GMT Standard Time, UTC+00:00)

Started the new job about a month ago now, and I'm quite enjoying the actual job itself, the people who work there (and the money they pay).

One thing however, has taken a huge hit. I was often asked by a friend of mine how I managed to get so much stuff done when juggling personal projects with a full time job. The answer has become clear since doing this "real person" thing of commuting and spending standard hours at work. My productivity has absolutely plummeted, I come home too tired to do anything computer related at all. I have ended up either collapsed onto my bed listening to music, or spending time with my newly acquired girlfriend Jo (bad timing, but not complaining). This is not because I do more work at work, but because I'm actually having to travel into work and get up early to prepare, I'm suddenly devoting thirteen hours of my day towards the job instead of the normal seven or eight hours.

I shall resolve to do something about that soon, I can't afford to not be working on my personal projects, Scrobbles or otherwise if I am not putting things out there on the internet, if I am not creating time to learn new technologies then there is little point in being a programmer at all.

Today however, I am working on Scrobbles, my work ethic has not gone anywhere and I have resolved to cross a few things off the list, as pointless at it is in light of the above.

Url re-writing is now done finally, and I'm going to push forwards with getting the rest of the things ticked off as much as possible. And I'll keep you all informed with my coping strategy for developing personal code when faced with the Real World. I think I know what my solution is, but I don't want to talk about it until I know if it is going to work.

Thursday, August 14, 2008 12:31:01 AM (GMT Standard Time, UTC+00:00)

Ploughed through some stuff last night, so here is that to-do list with a few things removed, and the most recent items dealt with. The client is so much more stable now I've tweaked the connection string to turn on some options to deal with the ferocity with which the client can push/pull data from the cache. Also added a few new services so data can be submitted in nice little compressed batches, speeding things up to no end. On a train a while ago I wrote the SDK for .NET and I'll be duplicating that for PHP when I get the chance. The Wiki is actually halfway to being populated but I've had some ideas about that elusive graphical snippet designer so perhaps it won't be needed as much as I thought. We'll see.

• Wiki
• SDK for PHP
• Service Documentation
• Snippet Documentation
• Family/Key/Value Documentation

Wednesday, July 02, 2008 2:15:56 PM (GMT Standard Time, UTC+00:00)

Of course, it goes without saying that the below example has lots of common elements that would be better presented as part of the SDK to save on boiler-plate code.

Click here to see new code for generating that output.

Somewhat better no?

Wednesday, July 02, 2008 12:00:57 AM (GMT Standard Time, UTC+00:00)

Spent most of the evening designing a flag for the upcoming Kendal's Calling and Tan Hill festivals, with my partner in crime Jo, who last worked with me to create the artwork for a present when visiting a band we like called The Witch and the Robot. The flag was a lot harder to come up with an idea for, having a canvas which is so many square feet in diameter. We settled with something we liked in the end though, so we'll have fun sewing and painting that later this week, stand by for photos!

Then the rest of the evening wasted in Diablo II with Owen, but that's all fun and nostalgic so why worry about that eh?

Because of the above, I've barely gotten any Scrobbles done tonight, but I did just sit down and polish off the embedded data from external websites, a very simple fix with a fresh mind and it was done.

So, this is the product of that.

This is a snippet, with nothing in it but an application calling into an external service.

Click here for xml in a new window

Walking through that from top to bottom:

• We have some inputs. These are entered by the user on adding the snippet to their profile. This allows the user to add the snippet multiple times and configure it differently each time.
• We then have the actual application itself, inside a Data section, which simply calls a third party web page and passes it the Character and the Realm originally entered by the user.
• In the Layout section, we simply say "I want this Application's data here". This way, several applications can be utilised within the same snippet, along with all the ordinary stuff you might find in a snippet.

Behind the aspx page being called, the following code can be found:

Click here for code in a new window

All I'm doing here is

• Ensuring the page returns plain old XML
• Creating a context from the sdk which takes care of authentication for us
• Pulling the variables out and pushing them into a query which
• Outputting the results of along with a bunch of text as html. (Well, a limited subset of html, of course).

This results in the following being displayed in the user's page. It's a really trivial example so the amount of overhead required for two lines of text doesn't seem worth it, but I guess that's always the way with trivial examples.

With this, the possibilities for representing data are endless, I can think of hundreds of ways I'd like to use this system so hopefully so will my users. If I get any that is.

It's kinda cool to get this working, as this was one of my original visions when I started last year.

Tomorrow I'll be going all out on Scrobbles again, I think I'm going to work through the documentation and get that completely sorted out, with perhaps a few more API tweaks to allow for direct access to pre-generated queries from external websites.

Tuesday, July 01, 2008 1:01:13 AM (GMT Standard Time, UTC+00:00)

I am really tired, as I mentioned previously I used this weekend's train journeys to start on the code that would allow external websites to embed data in a user's page (ala FB). This of course means creating some sort of auth system so those external websites are allowed access to the user's data (done), adding the application node to the snippet schema (done), creating a schema and a transform for the xml that the external website would pass back to Scrobbles for embedding (done), adding the database support for caching the results of these requests (done), writing the code to actually make those requests when pages are being generated (done).
Still left to do is adding support to the layout transform for actually embedding this data into the final page, but that can wait until tomorrow because having done most of the above just now I am truely knackered.

I got bored an hour ago and decided to see how much code I've got under the Scrobbles namespace after just under a year's development in my spare time. I started at the end of August, just after I stopped playing World of Warcraft. So.. close enough. 11 months development if I'm being kind to myself.

In total, I have:

• 723KB of VB across 258 files
• 164KB of C# across 39 files

This is of course filtering out the Visual Studio generated .designer files and etc, as they're rather large, and I didn't write them.

Of these, the following totals were calculated using SlickEdit.

• 19,000 lines of text
• 11,000 lines of code        (58%)
• 2500 lines of comments    (14%)
• 5500 lines of whitespace  (30%)

It's not actually too bad, of course on top of that I have a couple of thousand lines of javascript written, a thousand lines or so of hand-written XML Schemas and another thousand lines of hand-written XML transform file.

Calculating it, it's very roughly 50 lines a day, every single day with no break. Sadly I spend entire weeks not writing code, but it does mean I'm quite pleased with this average.

Drawing on those seemingly pointless software engineering lectures, (This project counts as being organic right?)

• E = 2.4 * (11)^1.05 = 29.762
• D = 2.5 * (E)^0.38 = 9 months
• P = 3

Obviously this is entirely meaningless, as COCOMO is a rather outdated and worthless calculation used against these modern languages which can practically write themselves, but it's still quite nice to see that I've managed to maintain some semblance of productivity this past year. Come on release!

Thursday, June 26, 2008 4:58:07 PM (GMT Standard Time, UTC+00:00)

This is the list as it stands for things I need to do before release.

• Core Scrobbles
  • Friendly URLs
  • Referal system for 'earning' queries
  • Embedding Data From Third Party Sites
  • Compressed Raw Data Queries
  • Compressed Batch Data Submission
  • Registration System
  • Arbitrary Views (using Snippet System)
  • Automatic Data Submission (Scrobbles App)
  • Security/Validation of All Existing Forms
• Third Party
  • Online submission of WoW Data
  • Heatmaps (location tracking)
• Community
  • Wiki
    • SDK for PHP and .NET (wrappers around system)
    • Service Documentation
    • Snippet Documentation
    • Family/Key/Value Documentation
    • Sync with Scrobbles DB
      
  • Forums
    • Sync with Scrobbles db
      
    • Requests etc

Seems I mostly just have housekeeping tasks left, so I'm going to run through a few of them tonight.

I have a pile of ideas for lots of 'small' projects once I have this completed and I'm anxious to get started on them. I think longterm, I'll make more money from the small projects per hour put in.

Scrobbles may end up being successful and that will be great, but if it's not, it will make a great technical portfolio item, and that was the reason I started it in the first place.

The small projects are to make money, and by spreading my time over many projects with smaller scope and smaller time input required, I hope to make something of them. Can't wait for the rest of this year to have rolled by. I'm feeling really motivated and can't believe my head can actually generate this many ideas!!

Tuesday, June 24, 2008 10:53:41 AM (GMT Standard Time, UTC+00:00)

Last week on a rather long train journey to Morecambe to see British Sea Power, I decided to write the next layer underneath the top level querying system, to cache 'collections' of individual common queries, which the results of could then also be queried in order to speed things up.

This has been planned since the original xml based querying system, but I had decided that writing it on top of that system would be like slapping a band aid on a broken neck. When something is that unworkable, trying to speed things up just by adding additional caching levels is a lost cause.

Happily, also on the train I wrote yet another version of the core querying system, which used a ridiculous level of nested inner joins to whittle down arbitrary data into a workable form. This is entirely counter intuitive, as you're told academically that joins are expensive, and nesting queries is expensive - and that by combining them you must be in for a world of pain. Imagine my surprise on running the query against SQL Server 2005's rather excellent query visualiser, that the resulting path was not only incredibly simple, but also incredibly fast due to the rapidly diminishing size of the data.

Because of the analytics code I already had written, it was very easy for the initial inner statement to cut down the size of the data by a huge amount in one simple query. Because quite a lot of the core queries tend to revolve around single keys, and single values (for example, "get me all the keys/values where KEY=WoWActionName and VALUE=MINING). Caching the results of this query, and even subsequent queries can result in replacing that part of the overall nested query with ("get me all the keys/values where the query = <>").

Running some trials, with the twenty snippets I have written for testing purposes, resulted in just 10 actual unique cached queries. It is important to note that in the original system I was filtering by date/time in the inner-most query - resulting in a unique query for every single page, and in this version I moved the date/time filter to the outermost query. This meant the results of this query could be re-used across all pages generated for that user.

This cut down the time taken to generate a typical 'all-time' page from 50-60 seconds, to 3-5 seconds. A rather massive increase of performance. (The fifteen seconds listed below wasn't when doing it concurrently)

These cached queries could then be refreshed or culled periodically using the background windows service I have written for Scrobbles, and most importantly rather than these resource intensive queries happening on different threads and really killing the server from over-zealous usage of RAM and therefore over-zealous paging and hard disk thashing, I can keep them on a single pipeline until I have a server that can deal with parallelising the whole shebang.

I foresee this not being enough when I have a huge amount of data in there for each user, the amount of data generated for an eight hour session of World of Warcraft is immense, and I only probably have sixty hours of data in there myself. I'll be able to add date and time to those cached queries later, and combine the daily results across a month, and the monthly results across a year and the yearly results across 'all time'. So I'm not worried about that.

I ran some tests on my laptop on the train, and was able to generate 2000 pages (all the pages possible for my single user over a year) in under ten minutes. This was using a threadpool to generate pages concurrently, as that is how it will happen once on the internet. At any one point during this, thirty pages were being generated simultaneously and taking seconds to complete. SQL Server's ram usage didn't even get above 300mb so I'm fairly confident this will work for my first few users. (Going forward!).

I appreciate the above is probably quite hard to follow, so here is a picture of the whole process - showing how each process has been decoupled to allow parallelisation as required in the future.

In other news, life is really busy at the moment and Scrobbles is having a week or so of hiatus. This weekend saw me at the Natural History Museum watching British Sea Power (again, yes I know), culminating in me lying on the stage drunkely trying to play the cornet which I had 'borrowed' from the band, tonight sees me at Victoria Park for Radiohead, as does tomorrow, Friday sees me at Koko seeing a band I saw a few weeks ago called "Die Die Die" (Awful name), but they were quite cool, and Saturday sees me up in Manchester seeing My Bloody Valentine, the legends that they are.

Yes, I have purchased and am using ear plugs...

Monday, June 16, 2008 1:45:17 AM (GMT Standard Time, UTC+00:00)

There we go, all that effort has paid off.

Time taken to generate a single 'one day' page of even the more complicated Scrobbles info is down to under a second, and down to about fifteen seconds for an 'all time' page view. (I cache these with quite a high longevity - think of the single default view that last.fm gives you and how often that updates..). On the old system with the current amount of data, this was going into the 'several minutes' for even a single day page. (Yeah, XQuery not so good...)

Getting the right balance of normalisation was key to this success, and I have written a lot more code than I would have originally liked to, most of it not being used in the final solution. Some of it quite experimental and rather cool though - and perhaps the key to future attempts in optimising Scrobbles. One of my solutions worked out 'groups' of keys which could isolate the data for most requests down over 90%, to generate permanent tables for crunching data into - and this may end up being a good way to go if my current solution doesn't last the distance. With the data for the entire year, this algorithm only generated ten tables - and I was able to index the entire year's data through this process in under five minutes so it was quite scaleable.

That was a bit hard to integrate with the query system as it stood however, so I'm bypassing it and going straight to the core data store for now.

Should be grand though, the background service can happily be constantly ticking over the 'all time' pages on a low priority (Well of course I have a priority queue based system!), and generating the one-day views as they are requested - and perhaps some sort of balance over the monthly views and yearly views based on what month or year it currently is.

At say, three pages per user, and 5 views to be updated constantly, that's fifteen views taking about 120 seconds in processing time altogether. I could still update every single users page more often than last.fm does (for its users' music pages) and support 500 users on the rather underpowered server this system currently sits on. Not that I would of course, because there is little point in updating pages unless people look at them sometimes.

Lots more work to do yet on making things even faster - but with a firm database design to now stand on, I feel a bit more confident about pushing ahead with the real development of the system.

Saturday, June 14, 2008 1:42:23 PM (GMT Standard Time, UTC+00:00)

My checklist has taken a hit this week, with a busy social calendar (organised by me a couple of months ago in the past apparently).

Not only that but I hit some technical issues after I migrated the server-side install of the Scrobbles system across to a new version, along with the database (a process that took over six hours, there being now over a year's data in there).  A migration that was supposed to make the whole system a lot more scaleable and maneagable in the future. De-coupling page requests from page generation and making a whole load of the resource intensive operations parallelisable.

The new system also removed the data-loss incurred by using discrete blocks of time, and moved the system across to an entirely continuous method of time-based data storage. And most importantly was meant to make it a lot easier to form complicated queries by storing a lot of this data in the original XML format, for querying using the XQuery support built into SQL Server 2005. I thought that there was no way I could do anything better than this with my limited knowledge, and that SQL Server's magic black box would just index this data and keep things fast and nifty for me.

In my trials, this seemed valid, and page generation was no faster or slower than the previous iterations of the querying system. On migrating across a larger data set, SQL Server started eating ridiculous amounts of Ram and CPU, before finally giving up in a big heap. Back to the drawing board again, for probably the fourth time.

It should go without saying that I still think the XQuery support in SQL Server 2005 is fantastic, and I think that the solution was simply not compatible with my needs. It was educational to play with however, and I can think of a few projects which would benefit from having the masses of XML on the hard drive transferred into a database which can then do the hard work of actually querying it for information.

I downloaded a full copy of the working database for testing and set about trying to index this massive amount of data myself. Many conversations were had with different people with varying levels of expertise - my colleagues (Karsten and Pat) were helpful over coffee and a pad of paper, and we came up with a new design which should hopefully have been more efficient. I also had quite a few conversations with Paul Evans who let me know of the oh so many potential pitfalls before I even began work on the new prototype system. Sadly, these pitfalls seemed to pop up all too soon and my query graphs were soon looking just as complicated, if not more complicated than the original XML driven attempt.

I think I've finally come to a proper solution now, which involves creating tables on the fly to fit the needs of the system as it evolves. This is a scary solution for me, as it gives away some of the intelligence of the database design to an indirect process rather than directly from me. It also increases the complexity of the code by quite a bit - and I was hoping by just having the one-size fits all database solution to avoid that. 

Sadly in the world where speed and efficiency counts more than anything else, and as I was originally warned at the start of the week, this seems to be quite a standard compromise in the world of database design.

Sunday, June 08, 2008 3:03:06 PM (GMT Standard Time, UTC+00:00)

The last one was starting to get a bit busy, and I was starting to panic because I'm never going to release if I have to work my way through them all.

I've split it up into things I need to do before I release Scrobbles, and those that can wait until its out the door. For one thing, I should stop assuming that my potential users are going to want things, and should really pass the lead onto them. Unless of course nobody wants to use Scrobbles, in which case I'll either start working on one of my many other ideas, or I'll push Scrobbles development in my own direction until I have what users will actually want off me.

I also realised that developing the snippet editor was going to take me a long time, not through technical difficulty - but actually designing something that wasn't harder to use than simply writing some XML was actually quite challenging. It means less people will be able to create snippets if I don't do it, but I'm not looking at wanting a particularly large amount of users right now anyway, just some - so I can start learning off them.

Focusing on the important list, and developing a good pitch so the World of Warcraft users understand what Scrobbles is all about should be my priorities. With the enormous scope that Scrobbles has, I run the risk of overcomplicating the description and confusing people, and if people don't know what software does, they won't use it.

Before Release
-----------------------------------------------

• Core Scrobbles
  • Friendly URLs
  • Referal system for 'earning' queries
  • Embedding Data From Third Party Sites
  • Compressed Raw Data Queries
  • Compressed Batch Data Submission
  • Registration System
  • Sort the database out [fast fast fast]
  • Arbitrary Views (using Snippet System)
  • Ajax-Driven Page Requests
  • Windows Service for Data Generation
    • Page Generation Queue
    • Pending Data Crunching
  • Automatic Data Submission (Scrobbles App)
  • Security/Validation of All Existing Forms
• Third Party
  • Online submission of WoW Data
  • Heatmaps (location tracking)
• Community
  • Wiki
    • SDK for PHP and .NET (wrappers around system)
    • Service Documentation
    • Snippet Documentation
    • Family/Key/Value Documentation
  • Forums
    • Requests etc

Post Release
-----------------------------------------------

• Core Scrobbles
  • Javascript Snippet Editor
  • Create an intallation manager (DOH, re-invent wheel!?) 
  • XML based 'module' installation/uninstallation 
  • Filterable install list 
  • Rollback on fail
  • Create click-once installer for the installation manager
  • Cache At Query Level (in DB)
    • Add cached query pre-generation to Windows Service
  • Crunch data on commonly used keys, primary, secondary, tertiary (speed up queries)
  • WoWWebStats Emulation
• Logistics
  • Finance of new server
  • Add a 'status' to each account for payment info
  • Provide the means with which to easily pay (Paypal/Google Checkout/Direct CC/Phone??) 
  • Limit the number of queries each user is allowed and make this dependent on account status 
  • Third party limitations (payment system too?), to prevent DoS attacks
• Third Party
  • World of Warcraft Blogging 
    • Develop automatic user creation based on Scrobbles data
    • Choose a decent template for advertising revenue
    • Architect a data-driven pipeline for generation of blog posts based on:
      • Location 
      • Activity 
      • Player Character 
      • Party 
      • ???
    • Develop background process to generate blog posts
    • Use the World of Warcraft Armory to populate character profiles 
    • Add capability for characters to automatically post comments on each other's blogs for purposes of hilarity
  • World of warcraft Avatar/Signature Generation
  • Embeddable Widgets into blogs/etc
  • Generate Vista Gadgets from form
     

So, there we have it. I'm not going to copy and paste this entire list again - I'm going to focus on the things I have to do in order to release the product, and then I'll revisit the second list.

Saturday, June 07, 2008 8:49:17 AM (GMT Standard Time, UTC+00:00)

I did quite well this week to get as much done as I did (In my opinion).

My scrobbles page shows quite an improvement in the amount of code being written anyway.

Here is what I have left - I'm going to work on the Javascript based snippet editor this weekend, as that would be a massive win to acchieve. After I've done that, I'll look at replicating those stats found at WoWWebStats and probably come up with some easily implementable features to bring the user experience more in line with what the users expect.

• Javascript Based Snippet Editing/Creation (MAJOR TASK)
• World of Warcraft Scripts + Research  (WWStats)
  • Replicate their stats using the snippet system (Modifying Lua as needed)
• World of Warcraft Data Submittal
  • Create an online page (using the third party data API) to allow online data submission for World of Warcraft (IE: without using the client)
• Write a background service for all data crunching
  • Asynchronous Page Generation
  • Pending Data Crunching
  • Cached Query Generation
• Snippet format work
  • Add capabiity for third party websites to insert data into snippets (using public services
• Generic Server Work
  • Work out how to finance the purchase of a new server
• Security + Validation
  • Go through all pages and check all user-input for limits/etc [Make sure automatic validation is turned off]
  • Validate postbacks for modifiable data and security concernsLogistics
• Add a 'status' to each account for payment info
  • Provide the means with which to easily pay (Paypal/Google Checkout/Direct CC/Phone??)
  • Limit the number of queries each user is allowed and make this dependent on account status
  • Third party limitations (payment system too?), to prevent DoS attacks
• World of Warcraft Automated Blogging
  • Develop automatic user creation based on Scrobbles data
  • Choose a decent template for advertising revenue
  • Architect a data-driven pipeline for generation of blog posts based on
    • Location
    • Activity
    • Player Character
    • Party
    • ???
  • Develop background process to generate blog posts
  • Use the World of Warcraft Armory to populate character profiles
  • Add capability for characters to automatically post comments on each other's blogs for purposes of hilarity
• Online Community
  • Populate Wiki with 'general' information
  • Populate Wiki with stat family documentation
    • Write a script to do this automatically from the Scrobbles database.
  • Populate Wiki for API documentation
  • Create forums for snippet requests/application request
• Make client application for automatic data submission stable (One is already written, it just needs a lot of work!)
  • Automatic World of Warcraft upload (Make this more atomic)
  • Create an intallation manager (DOH, re-invent wheel!?)
    • Elevated installer process
    • XML based 'module' installation/uninstallation
    • Filterable install list
    • Rollback on fail
  • Create click-once installer for the installation manager

Friday, June 06, 2008 6:18:07 PM (GMT Standard Time, UTC+00:00)

Just a small thought that just occurred to me.

I was just creating a quick form, where I pull a list of inputs from the database in to generate a form of those inputs and their current values. In order to do this, I write some code which creates a load of controls and adds them to the ASP.NET form dynamically during form creation. This is something that I do quite frequently, and it has become second nature to seperate the creation of the controls from the setting of the values in those controls, and to ensure that whilst all the controls are created during a postback, that the values are not set. (As they'll be set by the rather clever ASP.NET postback mechanism).

This does however mean that on a postback (When the submit button - or even cancel button is clicked), I am pulling the values from the database and creating a pile of controls so that I can cycle through them and retrieve their new values before updating the database.

Over the past week I have been allocating more and more work to the client via Javascript, and performing a lot of actions asynchronously- where it makes sense to do so, and I had a sudden "woah" moment when I caught myself starting to write some javascript to submit these new values once the user clicked the submit button. I had to stop and remind myself what the purpose of adding asynchronous behaviour actually was. The purpose is to add a fluid and faster user experience when a full postback to the server just simply isn't necessary.

Rather than creating all those controls again, a simple check to whether it's a postback, and then a loop through the inputs would suffice. Once that is done, the page re-directs back to the view screen so those controls never needed re-creating.

Once again - the zealous overdesign of ASP.NET covers up something that in PHP would be stonkingly obvious (And indeed, is the defacto standard method of handling such problems). I almost didn't do it because I'm so used to working within this friendly framework of controls and forms.

Silly Rob... silly Rob.

Friday, June 06, 2008 2:19:35 PM (GMT Standard Time, UTC+00:00)

With work on Scrobbles swiftly moving ahead, I find myself thinking about the inevitable and yet seemingly unreachable release date.

"Users are going to want this, so I should add it now.."

How many times have I now said that to myself? "Users are going to want to customize their pages", "Users are going to want to submit custom data", "Users are going to want to embed content in their pages", "Users will need snippets to be configurable so they don't need to write a new one for each 'key' or 'value'"... Each time I do this, it's for a good reason - I don't want to fail as a service, and therefore I need to be the best service around.

As mentioned previously, my main competitor (in the World of Warcraft arena anyway) is probably WWS (WoW Web Stats) - who have a mature, but nowhere near as flexible system as the one I have written. The keyword there however, is "mature". I took a look earlier and the wealth of information available from it is astounding. I can of course do better, and I do aim to write snippets which emulate the statistics that it throws out.

However, I then need to think about the groups of people who will be involved in these events, and think that perhaps they will want to combine their data and compare each other during raids, I need to possibly write a system that allows snippets to link to further in depth data based on a keyword in that snippet, I need to to write a system that allows users to create 'views' of their data between a user-defined period of time, with inputs coming through the existing snippet data. It needs to be really easy to create these views, possibly from templates so that data about a raid can be retrieved within a set period of time.

What about those casual users who are wanting statistics not about raids, but about their day to day activities? There is still a wealth of data that I am still not collecting, and I'm going to have to create a character of each class and profession in order to find out about them. It is absolutely terrifying how much stuff that I might "miss out" in the initial release of the software.

And there is the clincher, if I get it wrong, there is the chance that a future version of the WoW stuff might make previous data invalid. I can't be having that, so it has to be perfect, or at least - forward compatible to begin with - do I need a system for this??

What about those users for whom stats don't mean too much, I need to write that 'third party' website, WoWScrolls.com, so they can see the potential of throwing all their data at Scrobbles. (Public services are *awesome*). How do I achieve that? My colleagues at work have suggested that I use something like AIML to generate the blog posts and I can see their point, but it still leaves me with the daunting task of actually populating the database with "witty" phrases about each location, each task, each type of character and etc - nevermind creating the actual profiles from the data available at the WoW Armory.

My head is full of ideas, and getting that final feature list is not easy - because the moment I allow people to use Scrobbles they're going to start having even more ideas than I can deal with, and being the sole developer it's going to be very hard to keep up with the demand for features - nevermind technical support, complaints and all the normal day to day problems that come with running a website.

There is also that niggling issue, that releasing a service like this feels a bit like throwing down your cards at the end of a poker round, there is always the risk that your opponents might have a full house - and then what do you do?

Where do I call it quits? I could do with Scrobbles being out before the summer holidays so I could just prioritise my list of ideas and just work on them as much as possible before then, throwing it out in whatever form it has at that time. (Limiting the total users so I have time to assess server load and start thinking about monetizing the operation so I can spend more time on it - World of Warcraft is not the be all and end all of this system after all!).

I start to understand why games and software in general can often take such a long time to get out the door, there is always that one little thing that you just know the software will not be complete without. At some point, the users need to start leading the development strategy, and if their ideas conflict with mine - what on earth do I do then?

Monday, June 02, 2008 10:03:41 PM (GMT Standard Time, UTC+00:00)

Yet another evening of a work closer to the end than I was at the beginning.

I really didn't feel like starting, but I forced myself into it, wrote a small list from the main to-do list and got to tick a few things off it.

Seems setting specific tasks is more useful than I'd have thought. I also figured out some stuff which will help me at work tomorrow.

Monday, June 02, 2008 2:39:15 PM (GMT Standard Time, UTC+00:00)

Over the year of my contract with the university, I have been teaching myself how to do web dev - and coming from a background of professional desktop software development, moving to this world was surprisingly difficult.

I don't mean development as in the ability to put together a few pages about myself, or the ability to design a pretty website.I mean development as in putting together a full featured web application wtih the same features as an equivalent desktop version (if one was to be written).

My chosen area of learning revolved around ASP.NET because I'm already very familiar with the .NET framework. This turned out originally to cause me problems, because when developing ASP.NET applications you're given a lot of things you simply do not need, and you're given a structure to work inside of, that may or may not fit your end goals.

I found going to PHP and doing work in that helped, as I was given direct control over the process of form postbacks, and made to do everything myself. This gave me an understanding of the technology underlying ASP.NET and therefore the ability to work within the framework and create syncronous websites.

Obviously, the future is in asyncronous requests - a world without postbacks, and I've spent the past few months getting to grips with javascript, getting it to talk to the server, and architecting my web applications around a combination of syncronous and asyncronous behaviour.

When developing any code, I try to keep everything as organised as possible, to keep functionality in re-usable libraries, to keep presentation and business logic seperate, and this is where my main headache has been - in developing web software that is as organised as my desktop software. Trying to develop and utilise patterns across the entire web application so that once a few concepts have been described, anybody else could find what they were looking for if modifying/re-using any of this code.

Again, surprisingly difficult to do in a web application environment, as you are occasionally forced into mixing your logic and layout with this eery combination of Javascript, XHTML and VB.NET (Pick a fight on my choice of language and I will hurt you).

ASP.NET advocates the use of re-usable web controls, which can be slotted into web forms, just like in the desktop world, and I often find myself putting those in a seperate class library so I can easily have access to them in my visual studio toolbox. But then these controls end up referring to, or requiring certain markup to be available on a page (such as common dialogs inside of div, or web services exposed via the WebMethod system on pages). They therefore end up needing to be heavily commented, "do not use unless these thigns are present". They should probably be entirely private to the website itself - and even then there is the scope for abuse when you have over 50 seperate pages they could end up being used on. Can you say spaghetti code? No wonder PHP tends to be so all over the place - as you're not even forced into using any framework when writing it.

There are dozens of things like this, that the developer ends up just having to make a decision on. If you want to learn how to use individual components of code, learn how to use the framework, learn how to write the code, how to do little things, then there are books, there are websites to learn from. This has never been a problem.

If you want to learn how to put together, how to architect a solution that's elegant and forward-thinking, there is surprisingly little out there. It's left to the developer to work it out. (I'm talking a lower level than just N-Tier diagrams before anybody asks).

If I was working in a company that did web development, then I would no doubt be picking up on these things from my peers, who would have picked it up off their peers, who would have developed and learned from other people too. I am not however, and have ended up with my own style of doing things which may or may not be in keeping with other peoples.

A reflection on where I am now? I think this kind of learning is all very well and good, but it is harmful to productivity if you're doing it for your job. The amount of times I have now written this web client for the MeAggregator to a certain level of functionality before realising that I can't go any further without doing it all an entirely different way. If I was experienced in this field, I would have had it completed a long time ago.

I am lucky to work in a job where this is acceptable, and hope that the final product of this effort reflects the time I've taken to learn how to do things properly through trial and error.

Writing code is easy, writing huge amounts of code is easy - but writing large volumes of code that is understandable and well designed as an overall concept... it takes effort and knowledge.  The former I'm willing to put in, to extreme levels - but the latter can only come with time, and that's something we all wish we had more of.

Sunday, June 01, 2008 8:43:43 PM (GMT Standard Time, UTC+00:00)

Last week I worked quite heavily through the list, and towards core site completion. This leaves me with the tasks of building the third party websites, developing more client software and the creation of more user-friendly systems.

These are the tasks I have left to do, with a few more added. I aim to complete the ones in italic by next Sunday. I am however out all day on Wednesday, and at Microsoft on Thursday/Friday (I think), so time will be tight.

• Generic Server Work
  • Work out how to finance the purchase of a new server
• Snippet format work
  • Add capabiity for third party websites to insert data into snippets (using public services)
• Editing of pages (Ajax stylee, I already had a syncronous version done)
  • Hide/Show editing controls - based on user authentication
  • Renaming of Pages
  • Adding snippets to pages
  • Editing the inputs to those snippets
• Security + Validation
• Go through all pages and check all user-input for limits/etc [Make sure automatic validation is turned off]
• Validate postbacks for modifiable data and security concerns
• Write a background service for all data crunching
• Asynchronous Page Generation
• Pending Data Crunching
• Cached Query Generation
• Logistics
  • Add a 'status' to each account for payment info
  • Provide the means with which to easily pay (Paypal/Google Checkout/Direct CC/Phone??)
  • Limit the number of queries each user is allowed and make this dependent on account status
  • Third party limitations (payment system too?), to prevent DoS attacks.
• World of Warcraft Scripts + Research
  • Check out WowStats and see what stats they present to their users
  • Replicate those stats using the snippet system (Modifying Lua as needed)
• Javascript Based Snippet Editing/Creation (MAJOR TASK)
• World of Warcraft Automated Blogging
  • Choose a technology to build on (Going to build it myself in asp.net, it will actually be easier that way)
  • Develop automatic user creation based on Scrobbles data
  • Choose a decent template for advertising revenue
  • Architect a data-driven pipeline for generation of blog posts based on
    • Location
    • Activity
    • Player Character
    • Party
    • ???
  • Develop background process to generate blog posts
  • Use the World of Warcraft Armory to populate character profiles
  • Add capability for characters to automatically post comments on each other's blogs for purposes of hilarity
• Online Community
  • Populate Wiki with 'general' information
  • Populate Wiki with stat family documentation
    • Write a script to do this automatically from the Scrobbles database.
  • Populate Wiki for API documentation
  • Create forums for snippet requests/application requests
• World of Warcraft Data Submittal
  • Create an online page (using the third party data API) to allow online data submission for World of Warcraft (IE: without using the client)
• Make client application for automatic data submission stable (One is already written, it just needs a lot of work!)
  • Automatic World of Warcraft upload (Make this more atomic)
  • Create an intallation manager (DOH, re-invent wheel!?)
    • Elevated installer process
    • XML based 'module' installation/uninstallation
    • Filterable install list
    • Rollback on fail
  • Create click-once installer for the installation manager

Sunday, May 25, 2008 8:37:06 PM (GMT Standard Time, UTC+00:00)

In the interest of self organisation at work, I'm going to write a blog entry each week over at Redgloo with a list of the things I wish to acchieve at work. My boss can then comment and add things and I can cross them off as I get them done.

This is quite a cool idea, and I'm going to start doing it for Scrobbles as well, although maybe not as regularly. I'm dangerously close to finishing this project (at least to the point where people can start using it), and this week and weekend has seen a forced increase in productivity - the likes of which not seen for quite a while from me. I'm not entirely sure how I managed it, as at the beginning of the week I really didn't want to work on it. I just forced myself to by setting some tasks and got carried away from there. Let's try to continue this eh?

I tend to keep a text file on my desktop containing the current 'to-do' list, and as I have this, I can safely post the list as it stood at the beginning of the week, and cross out the things I got done this week also. I have backdated this entry to a week ago, to reflect this.

The way I'll do this, is each time I post, I'll post a revised version of the list, containing only the things I still need to do, and as I get things completed, I'll cross them out of the most recently written blog entry :) If I think of new things that need adding, I'll either edit the blog entry, or if enough has been achieved to warrant it, I'll make a new entry.

I'll highlight the 'next' planned work actions with italic lettering, and these will be things I aim to get done within a week's time.

• Write the public services for third party websites to retrieve/submit data
• Authentication
• FB Application like system, where users choose who is allowed to pull their data from the site
• Totalling queries
• List Queries
• Interval Queries
• User Info queries
• Generic Server Work
• Update database on server to catch up with development version
• Update server site to take on all this new code I've been writing
• Get the automated backup scripts working again
• Work out how to finance the purchase of a new server
• Snippet format work
• Add capability for user inputs to be given to snippets
• Add capabiity for third party websites to insert data into snippets (using public services)
• Editing of pages (Ajax stylee, I already had a syncronous version done)
• Hide/Show editing controls - based on user authentication
• Adding pages
• Removing pages
• Renaming of Pages
• Adding snippets to pages
• Moving snippets around pages
• Editing the inputs to those snippets
• Javascript Based Snippet Editing/Creation
• World of Warcraft Automated Blogging
• Buy domain (wowscrolls.com)
• Choose a technology to build on
• Develop automatic user creation based on Scrobbles data
• Choose a decent template for advertising revenue
• Architect a data-driven pipeline for generation of blog posts based on
• Location
• Activity
• Player Character
• Party
• ???
• Develop background process to generate blog posts
• Use the World of Warcraft Armory to populate character profiles
• Add capability for characters to automatically post comments on each other's blogs for purposes of hilarity
• Online Community
• Populate Wiki with 'general' information
• Populate Wiki with stat family documentation
• Populate Wiki for API documentation
• Create forums for snippet requests/application requests
• World of Warcraft Data Submittal
• Create an online page (using the third party data API) to allow online data submission for World of Warcraft (IE: without using the client)
• Make client application for automatic data submission stable (One is already written, it just needs a lot of work!)
• Automatic World of Warcraft upload (Make this more atomic)
• Create an intallation manager (DOH, re-invent wheel!?)
• Elevated installer process
• XML based 'module' installation/uninstallation
• Filterable install list
• Rollback on fail
• Create click-once installer for the installation manager
• Logistics
• Add a 'status' to each account for payment info
• Provide the means with which to easily pay (Paypal/Google Checkout/Direct CC/Phone??)
• Limit the number of queries each user is allowed and make this dependent on account status
• Third party limitations (payment system too?), to prevent DoS attacks.

Sunday, December 30, 2007 12:12:40 PM (GMT Standard Time, UTC+00:00)

Picture the scene: It's the 23rd of December and I'm sat quietly with a bottle of wine, the remains of the chilli that my housemate and I had shared earlier and I was armed with Visual Studio. I was in a good coding mood.

I'd done quite a lot of code and I was interested in seeing how much work that actually was, so I headed over to Robstats to take a look at the past few days of carnage to satisfy my curiousity, because I could. Except I couldn't because for some reason the server was being very slow and the data hadn't been updated for some reason.

I remoted in, expecting that perhaps the doomsday had finally come and Robstats had finally exceeded its capability of data crunching after 6 months of data. I was correct, but I didn't realise to the extent that this was going to kill muy web server. Half a million xml files storing 6 months data, things were bad. I knew when I was writing it that it was going to be bad, I just expected to have its replacement completed a long time ago.

The generation utility was eating memory (200mb on a 256mb server) and it was only halfway through its cycle. The server was groaning under the sheer weight of data and I had to stop it before I lost communication with the server altogether. Too late! I was disconnected and thus began the next few days of woe.

The crash caused a complete memory dump, there wasn't enough resources to create that memory dump and the server completely locked up. I popped off a support request to SynergyWorks and they gave me a hard-reboot. Cue file corruption. Cue explorer corruption. Cue lots of corruption. Over the next few days, things kept crashing and locking up and I had to ask SynergyWorks to reboot several times - just making the problem worse.

I decided to complete Lolstats and Scrobbles, the proper version of all this. Lolstats: A multi-user extensible and very generic stats crunching backend with a fairly flexible xml based data-transformation and presentation layer; and Scrobbles: A public website and frontend for managing your lolstats. 

The only thing preventing me from completing it was that I was focusing on the multi-user and ease of use side of things. It was taking up too much time without enough personal reward or gain. I suddenly had a reason to do it for myself and I got cracking on the bits that would be important to just display the data that Robstats had collected.

I got it working and by the 29th of December the server was completely dead. I got in touch with SynergyWorks and they decided that it would be best to give me a clean server, double the ram on the machine and mount the old data as a disk so I could recover what I could. Not bad support for a Saturday afternoon eh? </Shameless plug>

I went through the checklist and backed up the websites and data, and began an XCOPY of the half million odd xml files that constituted the 'backend' of Robstats. I had a migration utility and I was prepared to use it. Half a million xml files into half a million database tables, ready to be crunched into less than 20,000 records!

Some files were corrupt, some files had missing data because the format of the Robstats backend had changed over the months that the service had been up. I kept tweaking the migration utility until it could finally process all half million files and chuck them into the Scrobbles backend "processing queue". Actually converting the data and adding it to the database only took about 20 minutes, so it wasn't that painful.

As I write this blog post, the Lolstats backend is crunching the data into a more managable form. I've written the windows background tasks and Visual Studio 2005/2008 addin to capture the data and send it to Scrobbles instead of Robstats. The backend is processing about 1000 records every minute and a half, which means with a few improvements the current server should be able to deal with a few hundred users. It's going to take 10 hours to crunch the six months data into those hourly chunks and the pages on the site are now getting generated dynamically per-request (Yes, there is some complex caching going on too) through the Lolstats data presentation system.

I'll be documenting the web services and documenting about the xml format needed to pull the arbitrary data out and display it sometime over the next month. That should appear on the Lolstats domain once I've set that up.

I first need to write a decent website around the backend and provide a front-end for editing the pages being displayed. There is a lot of potential for third-party development and a lot of potential for a lot of data capture systems (games and various). I'd also quite like to use the Lolstats backend to capture data about the Scrobbles website and services. "Because I can".

I also need to purchase some decent graphing software. The freeware I'm currently using to generate graphs is nice and everything, but it's not really suitable for moving forward. If you want to be a part of the beta program, e-mail me on robashton@codeofrob.com and I'll consider you. I'd prefer to only have people who I know won't need help getting things set up. I haven't got some for support just yet.

Friday, January 19, 2007 11:16:55 AM (GMT Standard Time, UTC+00:00)

While I may have misforgivings about how dasBlog does certain things, I will hand it to the dasBlog team that the blogging engine is not only very powerful but appears to have a very nice API.

It is with this in mind that I am now writing a presentation layer as part of the main application that drives the rest of this site so I can maintain my standards compliancy(it's become a bit of an obsession for some reason) and keep my constistent theme.

Once I have done this, I may consider refactoring it for public consumption for those wishing a more themeable dasBlog. Because I'm making no modifications to dasBlog itself it can be considered as a plug-in on steroids :)

Monday, January 15, 2007 12:01:09 AM (GMT Standard Time, UTC+00:00)

Ahem yeah, what an overused word, but in this modern world of *gasp* browser competition it's starting to become more relevant.

I've pretty much finished putting content in, although I'll be re-writing vast tracts of it as I get time as much of the text is just placeholding to look pretty.

Now I'd like to complain about IE6, IE7 and Firefox.

Firefox: XSL transforms - why oh why do you read the <xsl:output> tag to work out what you're going to do with the transformed document? In what world does that make sense? If I'm outputting valid XHTML and my output type is XML, I expect you to render the XHTML, not get confused and fall over because the output document doesn't match your expectations.

IE7: max-width? Who decided to leave this out of the product? I mean really - one of the most useful CSS tags for making sure that my content doesn't escape the safety of the enclosure I've given it and it's not in IE7. My fix is included below the IE6 complaint.

IE6: margin: auto; - Why do you not do this? After resizing my images, I wanted them centered and the only way to get you to do this was to make an extra div with text-align: center set and place the image container inside that. What is up with that I ask you? I thought that when I turned CSS compliancy mode on with the XHTML DOCTYPE you might possibly, just possibly become CSS compliant but I guess I was just asking too much.

Resizing and Centering Images without losing CSS or XHTML compliancy.

First off, my CSS for centering the images and text underneath them:

A Big Screenie from Black and White 2