Grid Computing at the Santa Barbara .NET User Group

Hi All,

Went down to the Santa Barbara .NET Users Group last night for pointers on how to get the SLO .NET User Group up and running and to introduce myself. Everyone down there was very nice and seemed definitely into the idea. Now hopefully I can start getting members here, I have some announcements going out in the local newsletters for the Chamber Of Commerce and the Society Of Technical Communications. We'll see how it goes.

Saw a pretty intriguing presentation by Kim Greenlee ( http://krgreenlee.blogspot.com ) of Digipede ( http://www.digipede.net/ ). Digipede has built this pretty interesting Grid based computing system that essentially allows a system to offload its work to a large network of nodes that can even include the receptionist's computer at the front desk. Processing "agents" are dropped onto each machine that you want to participate in your grid. Each machine also can participate in a "pool" of functionality so that it only asks for work that it can handle. In this way it's slightly different than a cluster because the machines themselves are responsible for controlling how much and what type of work they take on during their down time, so you don't have to know exactly how many or what types of machines you are distributing work to.

They've also come up with a fairly straightforward SDK that allows you to create "Grid Objects" in your code to hand your processes off, assembly and all, to the processing grid. Your code strategies will have to shift slightly to account for such things as configuration variables (you'll have to set them as properties internally in your grid object because there is no guarantee the processing agent has access to the same config) and your assembly dependencies (if the grid object can't figure out all the DLLs to send over, you have to tell it where to find them), but other than that it seems a pretty decent way to distribute work and the performance gains were most impressive. In some instances Kim was able to achieve a 90% reduction in processing time on a distribution network of 9 nodes.

I have only two real concerns about this approach which I did not have time to broach with Kim in detail; one is that the distribution of work is Web Services based which immediately removes the ability to have transaction control. One of the places this would come in handy would be on large scale database inserts or bulk imports of data but you would have to be careful not to step out of the transaction scope when assigning jobs and tasks to the distribution grid. The other is that it is possible for the grid to get bogged down on one large high priority task and leave the others waiting. This means that if you implement this in a Web Server environment, where web servers are waiting on your distribution network, you have to be really sure your web servers are getting the highest priority attention so that your users aren't held waiting, or that you've set up a specific "pool" devoted to web server task processing. Otherwise, the possible logjam would defeat the purpose of offloading the work in the first place.

Kim is also presenting at Code Camp in October and I recommend checking out the presentation!

Rob