Large enterprises have wrestled with massive amounts of data for a long time. As technology migrates out of the IT department and into the boardroom for many businesses, it’s creating complications and confusion in many areas. There is a rush to process all this data to get answers. The challenge: what are the right questions?
Big Data, Or Whatever You Want To Call It
One area in particular is Big Data. There have been many discussions about what exactly is Big Data, which I’m not going to get into here. Some big thinkers have weighed in on Big Data (worth it for Merv Adrian’s quote at the end) and I had an interesting discussion recently with a colleague about the definition of Big Data.
To simplify the discussion, I will say that businesses today are collecting and have access to more data points in their ecosystem then ever. I see a theme emerging lately on the interwebs in regard to Big Data:
What insights will Big Data give us?
Small Data
If there’s Big Data there must be Small Data. Businesses are used to Small Data. They built systems of record to manage the data they created in the course of business — sales, financial, operations — and stored them in systems with nice acronyms like CRM and ERP. Eventually, they found ways to mine information out of these systems of record, applying Business Intelligence (BI) technology to slice and dice the data, and create “dashboards” to keep an eye on things.
BI is not a new topic. It’s where businesses typically go for answers, based on the Small Data they’ve generated in fairly controlled internal ecosystems. Small Data has helped guide businesses in strategic planning by giving them answers to questions that had been plaguing them for decades. It gave answers to questions the pesky MBA’s and management consultants had been bugging them about for a long time. It allowed them to take the pulse of their businesses with powerful data crunching — in real time, no less!
Life, The Universe and Everything
Sometimes life imitates art. In The Hitchhiker’s Guide to the Galaxy, one of my favorite — and oft-quoted — fictional series, there is a great search for the Answer to the Ultimate Question of Life, the Universe, and Everything. The answer, we find, is 42. A subsequent search for the “Ultimate Question” reveals that it’s “What do you get if you multiply six by nine?” While it’s a fictional account, it’s ironically not far from the truth of what we’re facing with Big Data: finding answers before we have questions.
There seems to be a belief that Big Data will give us bigger answers. I see one troubling trend here. In our rush to get value out of Big Data we are crunching the numbers, often with traditional methodologies, without really understanding the answers. Or, worse yet, misinterpreting the answers.
Bigger Data, Bigger Answers
In a recent thread in my social graph, there’s a lot of discussion about Klout and it’s value (or lack thereof). It occurs to me that efforts like Klout — trying to get value out of newly found data points — are a symptom of Big Data zeal. Klout gives you an answer to an as yet unasked question.
As a society we are used to fairly objective measures. A credit score is a good example. It takes into account a finite, slowly changing data set with a specific scope. Over time we’ve been trained to accept scores in general as accurate and meaningful.
Klout applies fancy algorithms to come up with a number that represents your “influence.” We are trained to give scores credibility, so the knee-jerk reaction in our society is to give this score credibility. The problem is not that it’s not accurate. It’s a question of whether it’s meaningful.
Klout most likely measures something accurately, but what? Klout gives you an answer: a number. But what is the question?
Ask The Right Questions
The biggest challenge we have with big data is wrapping our brains around the implications. We are capturing data points never before possible and creating databases that never existed. Those management consultants and MBA’s, who’ve long advised businesses, don’t even know what questions to ask…yet. It’s only just now finding it’s way into the curriculum of business schools. The need to process these unprecedented data sets requires heretofore unrecognized skill sets. Some say we need data scientists (What Is a Data Scientist (and What Isn’t)) and mathematicians.
As we struggle to come to terms with our new Big Data paradigm it’s important to keep an open mind and a healthy dose of perspective. As was the case with Small Data, there is a wealth of information and value to mined and evaluated. One thing is certain: before we can find answers we need to start asking the right questions.
This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines of a smarter planet.
Related articles
- Klout, Big Data and the Meaning of “Opt Out” (fredmcclimans.com)
- Big Data Is Hot Air, Says Wharton MBA Prof (bigthink.com)
