Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
May 14, 2012
prev slideNext slide

7 Key Drivers for the Big Data Market

I attended the Goldman Sachs Cloud Conference and participated on a panel focused on “Data: The New Competitive Advantage”. The panel covered a wide range of questions, but kicked off covering two basic questions:

“What is Big Data?” and “What are the drivers behind the Big Data market?”

While most definitions of Big Data focus on the new forms of unstructured data flowing through businesses with new levels of “volume, velocity, variety, and complexity”, I tend to answer the question using a simple equation:

Big Data = Transactions + Interactions + Observations

The following graphic illustrates what I mean:

Big Data Diagram















ERP, SCM, CRM, and transactional Web applications are classic examples of systems processing Transactions. Highly structured data in these systems is typically stored in SQL databases.

Interactions are about how people and things interact with each other or with your business. Web Logs, User Click Streams, Social Interactions & Feeds, and User-Generated Content are classic places to find Interaction data.

Observational data tends to come from the “Internet of Things”. Sensors for heat, motion, pressure and RFID and GPS chips within such things as mobile devices, ATM machines, and even aircraft engines provide just some examples of “things” that output Observation data.

With that basic definition of Big Data as background, let’s answer the question:

What are the 7 Key Drivers Behind the Big Data Market?


  1. Opportunity to enable innovative new business models
  2. Potential for new insights that drive competitive advantage


  1. Data collected and stored continues to grow exponentially
  2. Data is increasingly everywhere and in many formats
  3. Traditional solutions are failing under new requirements


  1. Cost of data systems, as a percentage of IT spend, continues to grow
  2. Cost advantages of commodity hardware & open source software

There’s a new generation of data management technologies, such as Apache Hadoop, that are providing an innovative and cost effective foundation for the emerging landscape of Big Data processing and analytics solutions. Needless to say, I’m excited to see how this market will mature and grow over the coming years.

Key Takeaway

Being able to dovetail the classic world of Transactions with the new(er) worlds of Interactions and Observations in ways that drives more business, enhances productivity, or discovers new and lucrative business opportunities is why Big Data is important.

One promise of Big Data is that companies who get good at collecting, aggregating, refining, analyzing, and maximizing the value derived from Transactions, Interactions, and Observations will put themselves in a position to answer such questions as:

What are the behaviors that lead to the transaction?

And even more interestingly:

How can I better encourage those behaviors and grow my business?

So ask yourself, what’s your Big Data strategy?

~ Shaun Connolly



Doug Laney says:

Great piece Shaun. I like the equation which equates to what I’ve been calling “subtransactional data.” Also cool to see the industry finally adopting the “3V”s of big data over a decade after Gartner first defined them. For future reference, and a copy of the original article I wrote in 2001, see: –Doug Laney, VP Research, Gartner, @doug_laney

Shaun Connolly says:

Thanks for the comment Doug.

I checked out the link you provided and in the comments of that post you mention “I’ve recently expanded to 12V’s that cover the full spectrum of big data challenges and opportunities”. Did you publish anything on those 12 V’s yet? I’m always looking for additional layers of detail that can help people understand the topic.

Steve Friedberg says:

Good post; the key element that’s increasingly become more important to businesses is the ability to rapidly (repeat, rapidly) do the deep dive analytics on the vast amounts of data. Volume and Velocity are absolute keys.

Whether it’s adding an analytical accelerator to an existing EDW, or designing a system from scratch that will seamlessly pull in data from multiple, disparate sources, time-to-results has become the imperative. It’s what companies of all sizes are demanding, because they know it’s become increasingly available at a price point that’s no longer prohibitive.

Alicia Brayboy says:


Great blog post. Many articles are focusing on the issue of Big Data, but few illustrate the tangible benefits that this revolution provides.

We are focusing on the benefits for businesses by helping them use Big Data for sales insight. (Please see our chapter, “Finding Big Growth in Big Data” in McKinsey & Company’s new book:

We’ve seen the tremendous difference between organizations that capitalize on big data and those who are left behind. Our customers are in the first category and are blowing their competitors away, surfacing to the top of their industries. Hopefully, many more will continue to follow this trend.

The opportunity for growth is extraordinary.

– Alicia Brayboy, Lattice Engines, @Lattice_Engines

Partha Mohanty says:

I agree and appreciate everyone’s comment posted here. I feel we need to highlight the importance of distingushing “Data” from “Information”. Data is RAW and Information is PROCESSED DATA which carries some meaning. We should be particular about the sanctity, accuracy, reliability, compliance, accountability, security etc of Big Data. Mere Big Data does not give insight. Secondly, Big Data outcome should be in sync with the varieties of sources and processes/ methodologies from where data is churned out to facilitate meaningful decisions. Business world keeps changing along with Business rules. Therefore, we should guide the industry with some charter and guidelines to educate and channelize them in the right direction to build frameworks of Big Data Analytics. Patterns with successful results could be shared across domains to adopt Best Practices. Investments on Big Data should yield meaningful return….Partha Mohanty, e-Governance Research, MGRM

Raakesh Kaul says:


Beautifully summarized to help understand the value of Big data given the ever changing consumer behavior, albeit very few CIO’s have either adopted or leveraged the benefits regardless of nature of companies business. .

Shalin says:

What is the software you used to create this graphic? you can contribute these to creately diagram community.

Vikas Kumar says:

Big data and Analytic are growing very fast to fulfill the business demands. Big data analytic will help in many businesses verticals now, making the decision very fast and getting actionable insight also.

Reggie Ezeh says:

Great Article Shaun.
While Pratha Mohanti felt the need to highlight the important distinguishing “Data” from “Information”, I do also feel we need to highlight the importance of distinguishing between just data and good data.
Veritas Technologies LLC reported that every year, organizations spend $7.2 million on storing their data, they spend $4.3 million on storing their unrecognized files types and $3.0 million on storing their videos. Of these data 41% is stale (hasn’t been used or modified in over 3 years), 12% is ancient (hasn’t been used or modified in over 7 years) and 5% is orphaned (doesn’t have an owner). Interestingly, a portion of these are personal files stored on company resources.
It has always been the belief of many that more data equals better results and more accurate predictions and ultimately better business decisions. I am of the opinion that more “good” data (as against “more data” ) is the driver of better analytics. Poor data behaviors exposes organizations to risk and drives up cost astronomically. Most data governance strategies are focused on data security, volume and access, not data “value”. For companies to harness the true value of Big Data, these issues need to be addresses as part of data strategy

Dean says:

Big Data does have technical, business and financial aspects to it and this is evident in the Big Data solutions provided by Digital Group ( and other leading providers.

Prakhar says:
Your comment is awaiting moderation.

hello from the future.

Leave a Reply

Your email address will not be published. Required fields are marked *