Big Data for IT? Get a Handle on Small Data First!
Big Data Analytics for IT: Eat Your Own Dog Food, or Nothing But a Buzzword?
Most 2012 predictions from analysts and other industry thought leaders talk about how Big Data is becoming a major priority on the CIO's agenda. So, immediately, the question comes up – should IT look at their own data in the same way as Big Data, in order to manage operations more efficiently. After all, IT is responsible for the systems that provide the organization with the very analysis tools themselves for Big Data.
"'IT is always saying they want to find ways to get closer to the business -- [big data] is a phenomenal opportunity to do exactly that,' says Eric Williams, CIO at Catalina Marketing. Rather than waiting for the pieces to fall into place, savvy IT leaders should start prepping themselves and their organizations to get ahead of the transformation, say analysts such as Gartner's Mark Beyer." ('Big Data' Prep: 5 Things IT Should Do Now) Vendors selling various IT management solutions (particularly in the CEP and Analytics space) saw that they can ride this hot industry trend, and started to claim that Big Data is a new silver bullet that will finally address long-time IT operations problems. So, in the end, does IT need to start 'eating their own dog food' before selling this concept to the business? Out of this excitement a lot of hype has been generated around Big Data. Blogger, Johan Louwers, observes similar sentiments in his blog "My view on the IT world": "Big data is currently a buzz word and as we all know buzz words are not always good. It has happened in the past that a buzzword made that a perfectly good solution or product was killed because it was simply so buzzed it could never live up to the expectations. Big data is currently seen as a solution to everything as also cloud computing is seen." (How did data become big-data)
Does IT Really Face Big Data Challenges? Does IT Use All the Collected Data?
Examining the applicability of Big Data for IT, we need to look at its' potential value and also at its' potential for making a real change in the way IT operates. Overall, IT produces four major types of data:
- IT portfolio and projects information including plans, budgets, HR etc.
- IT service data including helpdesk calls, planned changes, system stats etc.
- Infrastructure and application events information
- IT inventory and environment configuration
The Big Four (System Management vendors: IBM, HP, BMC and CA) provide solutions that allow (in theory) to manage any aspect of IT to collect and analyze this data . But if you check with their reference customers, do they say that – have they succeeded in leveraging IT data across the board? The answer is no. Today not one of these customers can claim that the data is efficiently used 360 degrees.
Variety, Velocity and Volume - Three Dimension of Big Data and IT
Big Data is composed of three dimensions: Variety, Velocity and Volume. For any of these dimensions, how does IT perform in making the best of its' collected information?
Almost the entire set of IT information can be mapped to very specific structures. Also relationships between the information elements are clear and explicit. The only source of unstructured information, that is valuable for analysis, are application and infrastructure events.
Big Data origins are in the sphere of the Web where organizations are generating terabytes of new data on a daily basis. For IT, event log information coming from network, server infrastructure and applications is the only likely source to generate new records with such frequency, leaving IT to not really reach the magnitude the is defined by Big Data.
Even event information that up to now could fit Big Data criteria does not amount to terabytes and petabytes that need to be rapidly analyzed in order to provide the insights necessary for optimizing IT management and operations.
The bottom line is that IT generates significant amounts of data with high frequency, particularly in various event logs. However this is still far from being called a Big Data challenge.
What are the Real Reasons For IT Failing To Leverage Information?
Why does IT still not succeed in taking advantage of the information that could scale up operations and drastically increase levels of service? The answer is not in taking a Big Data approach, but that IT has the wrong focus.
Most IT executives still believe that better processes or automation will solve all their IT operations issues.
- Very few IT organizations have succeeded in fully implementing complete sets of IT management and operations processes (i.e defined by ITIL). As a result wherever they are lacking a process, they also lack data
- The data generated by the process is also quite subjective. Taking the change management process as an example, the fact that you registered a change, assessed its' potential impact and approved it, does not provide any information about the actual change implementation, its' content, correctness and success
- Even when processes are in place and accurate data is generated, it is not integrated (even when it is there). This is done really rarely. So, you would not need a Big Data approach in order to correlate performance metrics and infrastructure changes information.
What IT Really Needs Is a Better Way to Handle Data (Big or Small)
Instead of becoming enamored with terabytes of information and chasing an elusive silver bullet, IT needs to simply better address how information is handled. No matter whether the data is big or small, everything that IT focuses on should be to achieve one thing: making data smart and actionable.
This means switching to an analytics based approach and relying on actual information captured as it happens within the IT environment, making IT more efficient.
Yes, I believe that Big Data will become relevant for IT as it expands to the cloud, adopts agile development and operations practices, incorporates mobile technologies and implements other exciting technologies and practices. But before the Big Data challenge becomes relevant, IT should change its' current management and operations approach. IT should start looking hard at analytics-driven IT solutions for supporting modern IT environments - Big Data or not