, ,

The Killer App for Cloud…Big Data

The Killer App for Cloud…Big Data

Written by David Linthicum for Nelson Hilliard

There has been a lot of talk around the use of the cloud as the perfect host for organizations that manage terabytes or petabytes of business data.  In the past, and much is the case today, those with very large databases who looked to leverage that information in strategic ways had to buy million dollar database engines, and pray that they could keep up.  The cloud could change all of that.

There are two emerging concepts here.  First is the ability to manage large datasets more efficiently than traditional relational technology using emerging technology such as MapReduce.  Second is the on-demand availability of commodity servers to support the MapReduce architecture, which is all about the distribution of the workload among available servers. 

For those of you unfamiliar with the concept, MapReduce is a software framework brought to us by guys like Google, Yahoo, and Facebook to support large distributed data sets on clusters of computers. What’s unique about MapReduce is that it can process both structured and unstructured data through the use of a distributed “share nothing”-type query-processing system.

The “Map” portion of MapReduce is the master node that accepts the request and divides it between any number of worker nodes. The “Reduce” portion means that the master node considers the results from the worker nodes and combines them to determine the answer to the request. When thinking about MapReduce think about making things simpler, and much more distributed.

There are open source software instances that leverage MapReduce, such as Hadoop, which continues to gain popularity as a very efficient approach to managing large data sets.  Right now many cloud computing providers are adopting Hadoop for their users, and many are finding it to have a huge advantage over traditional approaches to database processing, also now living in the cloud. 

The killer app here is around the fact that data growth is exploding.  While we can certainly solve that problem with million dollar super servers, for now, we really need to get much more innovative in how we manage “big data.”  Much as we began to move to relational databases back in 1985, perhaps it’s time we move to another model in 2018 now that the game has significantly changed. 

However, the adoption of this technology has moved rather slowly.  There are vast differences in the way you manage data from Hadoop, and the way you use Oracle and other big commercial databases.  However, as more executives wait to get the information they require from their data, the need for this kind of technology will turn from a request to an outright demand.  Companies that leverage this technology now will find a huge strategic advantage in their markets.  They will better understand their data, but do so without the price tag of conventional approaches to data management.

Remember to Subscribe to our Youtube Channel for the Latest Cloud Computing Tech Jobs, News, and Cloud Shows.


David S. Linthicum is a managing director and chief cloud strategy officer. David is internationally recognized as the worlds No.1 cloud computing industry expert, pundit and thought-leader.

(Disclosure: David Linthicum’s views in the blogs, video shows and podcasts are his OWN and are NOT financially sponsored by Nelson Hilliard)

Connect with David on LinkedIn and Twitter

At Nelson Hilliard we specialise in cloud technologies, sourcing the top 20% of cloud professionals inspired to work for you through our specialised marketing and profiling. If you are interested in having a quick talk to me regarding your employment needs please feel free to reach out.

You can also check my availability and book your 15 minute discovery call here.

Brad Nelson