1 (866) 866-2320 Resources Events Blog

Hey, Big Data is Actually 11 Years Old!

Blog

Hey, Big Data is Actually 11 Years Old!


 

Thought Big Data is a brand new concept?

Not really. The foundations for Big Data were already laid down over 10 years ago by Gartner analyst Doug Laney. In an analyst report, he wrote about the advent of Big Data, anticipating the growth and impact that Big Data would have on business and technology. 

Back then, he was motivated to look at Big Data in an attempt to help clients that were becoming increasingly encumbered by their data assets and to help them get a handle on how to recognize, and more importantly, deal with this. He established the foundations for how Big Data is being described today by putting the nascent concept into perspective, describing Big Data as a 3-dimensional data challenge of increasing data volume, velocity and variety. (See his research note 3-D Data Management: Controlling Data Volume, Velocity and Variety). 

Now taking a look at Big Data today, and we can see that Doug's original outlook of 3-D Management has become ubiquitous. Recently, our CEO, Sasha Gilenson, took a look at the relevancy of Big Data for IT Operations in his post Big Data For IT? Get A Handle On Small Data First! He outlined the 3 areas as established originally by Doug Laney, for evaluating if data in IT measures up to be called Big Data. Sasha raised the issue that "Big Data is composed of three dimensions: Variety, Velocity and Volume. For any of these dimensions, how does IT perform in making the best of its' collected information?" 

As Big Data catches momentum across the business and technology landscape, let's look back at how Doug Laney established the conceptual foundations for Big Data.

Big Data Surges Forward

Back in 2001, Doug Laney described the emerging Big Data revolution as the outcome of the effect of the e-commerce surge. This was the result of a rise in merger & acquisition activity, increased collaboration, and the drive for harnessing information as a competitive catalyst "for driving enterprises to higher levels of consciousness about how data is managed at its most basic level."

In the report, he described how E-commerce, in particular, had exploded data management challenges along three dimensions: volumes, velocity and variety. Today, we see everywhere how these components serve as the foundation for describing Big Data behavior. 

Volume, Velocity, and Variety

Volume

Doug Laney said that enterprises had come to see information as a tangible asset. The volume was boosted by the lower cost of e-channels enabling enterprises to offer more goods or services and then collect (up to) 10x the quantity of data about an individual transaction—thereby increasing the overall volume of data to be managed.  Velocity

Doug Laney described the growth of Big Data out of "E-commerce has also increased point-of-interaction (POI) speed, and consequently the pace data used to support interactions and generated by interactions." As Doug Laney described, Big Data origins grew out of the sphere of the Web. Organizations generated terabytes of new data on a daily basis.  Variety

Doug Laney anticipated that data variety would underpin Big Data growth, saying "Through 2003/04, no greater barrier to effective data management will exist than the variety of incompatible data formats, non-aligned data structures, and inconsistent data semantics." 

As Big Data becomes the main talk of IT and business, it is worthwhile to give recognition to Doug Laney's foresight. In describing how to take on the Big Data challenge, he said over 10 years ago that "IT organizations must look beyond traditional direct brute force physical approaches to data management." 

Today, we see that Big Data analytics has become the industry buzzword, comprising the data V's (variety, volume and velocity) that Doug Laney originally described, and now inundates most of today's organizations. With the big data trend, organizations have silos of information that exist in a variety of locations -- inside a firewall, on the Web, the cloud, even data that is owned by other enterprises or by customers and suppliers.

Your Turn
What is your Big Data challenge, and how are you staying on top of the variety of sources of data in your organization?

About the Author
Martin Perlin