This morning at Hadoop Summit, I gave a talk about how data will transform everything and the need for Connected Data Platforms. Let me explain what I mean.
Until now, enterprise data was largely structured and not particularly diverse. This reality spawned what we know now as traditional IT best practices: buy standardized packaged solutions, or use them to replace custom solutions and implement an EDW based on user requirements.
We have now moved from that old world to billions of connected devices creating and generating more data than ever before – both inside and outside the corporate firewall. All of us as consumers are creating this data every moment of every day with infinite variety. This will only accelerate as the global proliferation of connected devices continues, and as consumer and customer expectations for relevant content and social interaction increases.
The new data is very diverse. Whether it’s from a device, a sensor or the web, a car, a phone, its not structured transactional data under an ERP system any more. Analyzable data lives everywhere. In the cloud, at the edge, in the data center, on the device. We can’t put this genie back in the bottle.
It’s beyond big data, CIOs now need to think about all data. The implication is that the technologies and best practices of the past are not sufficient. This is creating upheaval in the IT architectures of businesses and many CIOs are re-platforming their data layers.
We see Connected Data Platforms as the key innovation that is going to drive success in this new world of data.
One big thing driving this re-platforming is cloud versus data center. Now everything is cloud-centric, and internal data silos are being replaced by data clouds that all need to connect.
Another is the need to analyze data where it lives. Data has gravity. Although its getting cheaper all the time, data movement is still relatively expensive, so we need to run analytics as close to where the data lives as possible. It’s paramount to figure out ways to get data out of the old data-center silos and connect it to the data cloud and edge of our networks in order for breakthrough kinds of value creation via analytics to happen.
The data and apps of the past were all about centralization; pull all the data in, normalize it, put it into silos, then figure out what to do with it. In the new world order that is simply not even imaginably workable, let alone realistic, and it certainly prevents value creation.
Both of these reasons are why see the need for a new kind of Connected technology platform that is all about data versus app-centricity, cloud and data center, and that can connect all data wherever it naturally lives.
Today we need distributed analytics, distributed data and distributed applications.
This is a business not just a technology mandate. Amidst all of the new data, it is no longer acceptable to just analyze historical information to report on and understand what happened. You’ve actually got to take decisive action while things are happening.
The technical implication of this is we need to connect streaming data, historical data and analytics. The only way to have this work is to have all of our data platforms be connected, whether in cloud or in the data center, so they can communicate with each other, connect all the data that is relevant and the analytics that are relevant at a service level that is relevant to the business.
The notion of being able to connect data-at-rest and data-in-motion and different data platforms, across multiple cloud providers and in data centers with true application portability, is the central differentiator between the future of data versus the past.
The implications of being able to connect data and software in this way are huge value creation and transformation for businesses.
Once you remove the silos, become more data and less app-centric, you can start to do analytics in real time as data is streaming. So rather than running a report to see what happened and then having 17 meetings internally to determine what action to take after the fact, and it’s too late, you can move the time of decision right to when your people are interacting with the customer. In other words, to the point when the decision needs to be made. In some cases even being pre-emptive before an event happens.
There is an important shift: from Converged To Connected – the opposite of traditional centralized systems. There is a similar process shift.
The shift is from: find the data sources, then pull it all together with ETL, centralize and normalize the data and then run analytics. To capture the data wherever it is, analyze it in real time, understand the signals, process intelligent action and execute in real time—before while the transaction is happening in front of the customer.
In the new world data is everywhere — at the edge, in multiple clouds and in the data center. Connected Data Platforms allow analytics to be portable in order to move to the data in real time across this decentralized footprint, right to the very edge
This goes way beyond real-time streaming or real time brokerage or online ad placement, which are based on simple rules. As you become able to combine what’s happened in the past and what’s happening now it means you can apply adaptive and pre-emptive machine-learning algorithms to the decision making process. In other words, machine learning models which can both adjudicate decisions, develop over time and can even start to predict behavior.
This is very intuitive for us all as consumers. We all instinctively know that when we get an offer to buy something from a company, the offer itself changes our perception and behavior. We weren’t planning to purchase, but now we are. We naturally expect that the rules which applied to us from then on also have changed.
With this kind of ‘natural’ autonomous decision-making capability in your business, you start having a much much more intimate relationship with your customer. Way beyond the concept of ‘white-glove’ treatment or ’platinum’ status.
What happens is your understanding of the interactions with your customers, or your suppliers and partners, inevitably develops and this can be nothing less than transformational for your business.
The impact of these changes is profound across every industry. In retail, for example, the ability to analyze customer buying behavior down to the level of the in-store beacons or smart hangers and being able to map this in real time to social signals and online visual search is providing a 360 degree view of the customer that fundamentally changes the concept of a loyalty program.
Tie this to the inventory system, store operations, and real time supply chain optimization from POS systems, and retail businesses can start to transform how they do business. Look at the results some companies are already achieving: doubling promotion revenue lifts, cutting fraudulent transactions and storage spend in half. Game-changing results.
The future of what data will transform is awesome to imagine. We just outlined a few of them this morning. Also read Scott’s blog today. I’m sure the community and the ecosystem will continue to grow and thrive and find new value from data. After all, I believe that data is everyone’s product and their most important corporate asset.