Legionnaire Technologies

With new product offerings from Amazon, Google, Rackspace and others, grid computing has been thrust onto the stage to feed the data driven society in which we live.  Grid computing is a scalable fault tolerant solution that will change the way websites will be hosted.  By leveraging multiple computers to process requests, the dependency on dedicated server hardware become s non-existent.

Grid computing began to power largest scientific research projects leveraging multiple computers and processing large amounts of data.  The computers or nodes are managed by controllers that send the requests to the grid.  The controllers hand out jobs to available nodes that can come on and off the grid.  If a job is not processed by one of the nodes, the controller passes the job to another node.  This architecture has created a fully functioning distributed-computing environment.  These distributed grids are used for scientific research such as the SETI project as well as render nodes used to make the animated movies shown on the silver screen.  A large project has its tasks processed at various locations to leverage the available computing power.

In the hosting realm, grid computing has taken two forms.  Companies like Amazon, GoGrid, Joyent, and others have created a platform that will allow a user to create an infinite number of virtual computers to process request.  While others like Google and Mosso allow the applications to be created and will scale the horsepower of the grid to meet demand on the application.

I was recently asked to do some research for a project that put me in contact with a solutions team at Amazon.  Amazon's EC2 cloud allows users to create an army of computers to act as the clients own virtual grid.  They can then scale their grid to meet their demands.  This solutions can operate like the traditional grid having a large task and scaling the processing horsepower to complete the job.  The New Your Times recently leveraged Amazon's EC2 to index their entire article archive since 1891 in a matter of hours rather than several months.

Amazon's EC2 is not limited to traditional projects.  Infrastructure managers are given the flexibility to design their virtual environment to meet their own needs.  This is something that services from Google and Mosso don't offer.  The system architect can leverage individual servers to complete dedicated tasks, much like a traditional design would operate but in a virtual environment.  This would allow for utilization of Amazon's storage space that offers S3.

The solutions from companies like Google and Mosso are designed to be more application centric, however each company takes an opposite approach.  Google's newly released App Engine provides a way for developers to leverage Google's infrastructure but currently limits developers to coding their applications in Python.  App Engine leverages Google's proprietary file structure and Big Table database across the grid to scale applications to meet demand.

The solution by Rackspace's Mosso allows developers to put traditionally *nix files and Windows files in the same directory but when the file is accessed it will be served through the native hosting environment.  Like App Engine as the demand on the application increases so will the infrastructure that supports it.  This unique approach will be the easiest transition for users who are familiar with a control-panel based hosting environment.  

As grid computing becomes more mainstream hosting providers not able to provide the scalability and reliability of grid computing will no longer be acceptable.  With new applications and more data becoming assessable online every day, demand increases.  Leveraging the power of a grid system increases reliability and performance and possible reduce overall overhead.  A great place to start researching grid/cloud computing is the Structure08 conference.

Last update: 25-06-2008 13:36