Big Data Has Arrived

It’s a coming out party for data. We are recording data at historically unprecedented rates.  Have you noticed you keep getting bigger storage devices and somehow continue to fill them up? You’re not alone.

Not Your Grandparents Data

For a long time data was the exclusive domain of dedicated departments of large enterprises powered by  complex IT infrastructure. The primary purpose was to capture and maintain transaction data for reporting purposes. Simple stuff like accounting general ledger, sales performance, P&L. The focus was on analysis of the past with terms like data warehouse and business intelligence being thrown into the mix.

Finding A Needle In A Haystack

Big Data is as much about quality as it is about quantity, and it’s very much about analysis. We are accumulating data (or it’s being thrown at us) that is unstructured and disconnected. The challenge is to make sense of it all — to extract and/or synthesize the proverbial needle in the haystack. Yes, there’s a lot of data, but we have the computing power to make sense of it: on demand, in the cloud. There is tremendous potential to impact just about every facet of our lives.

The Final Encylopedia

When we have all the world’s knowledge at our fingertips, what do we do with it? We have seen some of the applications already, in sales, marketing, communications, customer service — the list goes on.

There is tremendous opportunity, but there is also risk. Pervasive use of digital media is making available an unprecedented amount data, much of which we don’t even realize we’re giving away. Ever uploaded a picture to Facebook? Twitter? Do you know What Your Digital Photos Reveal About You?

Here are some issues we as individuals and organizations will have to consider today and going forward:

Issue 1: Filter: There’s information overload, which creates a strong need to separate the useful data from the noise. Curation has become popular theme. How will we identify and retain the data that’s relevant to us? 

Issue 2: Retention: With my 25+ years of IT management and support experience my knee-jerk reaction is to consider how we can store, manage, protect, archive and retire it all. Information is a living entity and like all living things has a useful lifespan. At some point it must be retired and/or destroyed.  How will we continue to manage and maintain the data that is important to us?

Issue 3: Analysis: A key component of the value proposition of Big Data is analysis. Chris Heuer of Deloitte recently made this statement at at Social Media Masters in NY: “We are moving from systems of record to systems of engagement.” What can analysis of Big Data do to change your life ? Your business? 

Issue 4: Impact: Big Data represents opportunities and risks — social, business, security, privacy — there are many. These are issues we need to address as a society. How do we learn to live with the opportunities and risks inherent in Big Data?

Based on the 9/19/11 #usguyschat we felt it was worthwhile to expand on the topic of Big Data. As Ken Rosen put it, “we scratched part of the surface and have an opportunity to go deeper.” So I have attempted to place it in a broader perspective.