Last year Hadoop celebrated it’s tenth birthday, young in the land of data technologies. But the growth in popularity of Apache Hadoop is not slowing down anytime soon. In fact, results from the 2016 Big Data Maturity Survey indicates 97% of respondents plan to do more big data initiatives in the next 3 months.
The Rise Of Big Data Analytics or Business Intelligence
More interesting in the survey though is the rise in business intelligence (BI), overtaking ETL as the primary workload. Over the years we have grown accustomed with the data science and data preparation workloads running on Big Data technologies, but now 75% of respondents will be running BI on their Big Data deployments.
So, why do I view this as the Big Data Tipping Point? In my view it means Big Data is growing up, becoming mission critical, is becoming trusted by the business. A few key factors are shaping this.
For Big Data to really become come of age and be the mission critical data platform for BI, it first needs to support interactive SQL style queries to support an iterative OLAP style analysis. These fast Big Data SQL technologies are now readily available through Hive LLAP, Spark SQL, AtScale and others.
Run Hadoop Anywhere
The switch for enterprises to run their applications in the cloud has dominated IT strategy for the past number of years, but BI in the cloud has not exactly followed the same path. That is until now. In the Big Data Maturity Survey 72% of respondents this year indicate they will run Big Data workloads in the cloud compared to only 53% that currently run in the cloud. Offerings like Microsoft Azure HDInsight and more recently Hortonworks Data Cloud for AWS mean there is a growing list of cloud native big data services to tap into for all kinds of common data use cases.
Importance of Governance
Along with the rise in BI (more users all over the enterprise) and running this in the cloud, the need for better security and governance is paramount. In fact the survey bears this out with Governance as the fastest growing concern, up 21% over last year. Over the past 12 months many of the Apache Foundation efforts to bring Apache Ambari, Apache Atlas and Apache Ranger to the next level are clear indications that the voice of the enterprise customer is being heard and understood in the community.
One last statistic that caught my eye in this year’s survey was who drives the big data initiative. And it won’t come as a surprise that IT (at 57%) is still the main driver, but interestingly, when business executives drive they are 19% more likely to achieve tangible value. (BTW Did you know Hortonworks can help you drive big data success with tutorials, training, guidance, industry experience and support?
Want to get more insights from the report? Get your copy of the 2016 Big Data Maturity Survey.