For most businesses data is the foundation upon which they wish to build better customer experiences, deliver new innovative products or improve operational efficiencies. However, for many showing the return on their data investments remains illusive. A recent McKinsey report indicates only some 30% of proposed benefits were achieved over the past 5-6 years.
In February Hortonworks commissioned Forrester Research to explore challenges associated with big data adoption and understand the current trends shaping architectural choices. According to the study 3 out of 4 decision makers surveyed are expanding their use of big data in areas of automated decisioning (eg fraud detection), real-time analytics and also new innovative products.
One of they key challenges highlighted in the research is that the current approaches to bringing the data from the disparate systems and apps are heavily labor-intensive and time-consuming, leading to frustration with the business community waiting for fast results. Over the past decade, the application strategies adopted by companies have gone from standardizing on monolithic business suites, running in your own data center, towards best of breed applications running in the cloud. Just to answer the same type of business questions that we did over the past decade now demand a bigger effort to integrate these apps to create an integrated view of your business.
In addition to the expansion of our app landscape, there are now so much other interesting data we could also get our hands on – customer product reviews on Amazon.com or clickstream data from your web or a stream of data from a personal health monitoring device an individual might be carrying. While the application data integration is challenging because of the distributed nature of the apps, the new data adds the complexities of massive volumes and the fact that the data itself is not formally structured like we used to. In short, our old approaches are not able to help much with these new problems. Read how Prescient, a travel safety company is pushing this to the limit by ingesting data from thousands of data streams.
The third element of our data landscape is about real-time streams of data and distributing decisions to the edges – managing data in motion. One of the biggest advances over the past few years is around artificial intelligence (AI) and machine learning (ML) and how it relates to the Internet of Things (IoT). Consider the example of streaming data from a personal health monitoring device – simply ingesting and storing data about a person’s heart rate is less valuable than for the device to call the doctor autonomously if the person’s heart rate has been dangerously elevated for more than one minute.
Fortunately, over the past 10 years big data technologies, NoSQL and other technologies have come of age to handle these new data problems, but as our McKinsey report illustrates, many challenges remain. New technologies usually means a steep learning curve for the organization to adopt the new technology and become fluent with this new technology. The larger challenge often though is changing our processes and culture. Many of these new data initiatives are stuck in science fair project mode because we never designed it to scale to real life. A good example is around security and governance – in the initial pilot project this is not a key criteria, but once you need to take this into mission critical broad use it becomes a critical capability. If you neglected this in your original technology selection you have just gone down the river without a paddle.
A key objective for organizations is to create the scale and competence to manage all the data they have – data at rest and data in motion. Having a connected data architecture that provides a standard enterprise-wide way to manage, govern and secure your data is becoming an imperative for success. In our research more than 95% of respondents believe they need to implement such architectures, while acknowledging that only some 5% actually think they already have something workable in place.
If you are embarking your big data journey and are interested in learning about the key considerations, please join us for our webinar on May 3, 2017 that will feature guest speaker Noel Yuhanna, Principal Analyst with Forrester Research to discuss the results of the study in context of their work on Big Data Fabric and how companies can embrace this strategy to accelerate growth and innovation. I will join Noel to share how Hortonworks’ Connected Platforms are used by customers to unlock their own growth and drive innovative new products.
Register now for your seat in our “Accelerate business outcomes with a connected data strategy” webinar.