As those of us with long careers in the technology field know, when a revolution in technology occurs, point solutions and applications tend to arrive in the marketplace first. In the beginning, they make it easy to streamline and automate processes, but before long, scores of these solutions have proliferated.
Applications that were initially beneficial become problematic by creating multiple stovepiped copies of data and drive unnecessary expenses for storing and processing the same data multiple times. The “big data revolution” is presenting businesses with these challenges now. I’ll call this challenge global data management. It’s a complex challenge, but by being proactive, companies can overcome it.
Business leaders face acute challenges with global data management today. Over time, they have created data marts, data warehouses, and data lakes on prem and in the cloud while also remaining in their original locations. Originally, the intent of all this was to simplify business processes and reduce costs, but it has ultimately created “data entropy.” Copies of data are falling out of sync—returning different answers from different systems. That creates a lack of trust and credibility that businesses must urgently address. It’s also worth noting that this entropy generates more costs in the end because the prior solutions never went away—they must still be managed and maintained.
Managing all of these complicated data structures can prevent businesses from operating with the agility they need to successfully compete now and in the future. As business leaders try to capitalize on their data’s potential so they can enhance the business or even identify new lines of business—pursuing a path of digital transformation—they’re confronting the proliferation of increasingly complex systems and the resulting entropy of spend. Businesses must rationalize all of their data assets and make sure that, when they move into the cloud, they don’t end up grappling with these costs. Using the strategy of convergence that’s been in place to date, that’s far easier said than done, and it makes the mission of the chief data officer (CDO) particularly complicated.
At this time, CDOs have an especially tough mandate because they must simultaneously balance several priorities: high-level business needs, competitive pressures, and the rapid pace of change introduced into the business by applications that are based on a new paradigm. This role has to effectively partner with the chief security officer, chief risk officer, and other C-level colleagues to ensure that governance and security are properly handled across the entire company. This is no easy task; there are frequently diverging requirements within the business that must first be reconciled for such an endeavor to be possible.
While adroitly balancing these multiple business priorities, the CDO must also determine how to keep data fragmentation and multiple copies of data sets from manifesting within the environment as a result of the inevitable trend toward entropy. This business leader must execute rapid application development as well, so the company doesn’t have to painfully rip and replace its existing systems after entropy and data sprawl have set in.
For the last 35 years, IT infrastructure was designed and implemented using convergence as the guiding paradigm. Businesses would bring all of their data into the data center, put it in one place, and then take it out as needed. Before long, it will be physically impossible to manage their data in this way. Soon, data will live all over the place; some data will live its entire life cycle in the cloud. Companies may have multiple cloud footprints and multiple data centers housing all of their collections. At this critical juncture, businesses need to fundamentally change their thinking about global data management and adopt a paradigm of connection rather than convergence.
By connecting their data, businesses will be able to make integrated, timely decisions and reap greater value from their investment. They will also be able to take advantage of enterprise-class multitenancy operational management, security, and governance across all of those assets and confirm that they’re compliant. With the EU’s General Data Protection Regulation set to take effect in 2018, this will be a top priority for businesses operating in Europe. Being able to accurately identify where someone’s data has been, where it is, how it’s been used, and that it’s compliant (including being able to guarantee that all instances of it have been deleted, if necessary) will be of the utmost importance.
Companies need to start looking for services that provide all of these key capabilities, like the Hortonworks DataPlane Service. The right service will provide a future-proof framework for your data architecture by offering a common integration point for security and governance. Look for a provider that allows you to move to, from, and across multiple cloud footprints, all while guaranteeing compliance and greater agility in the transition to and from the cloud.
Businesses face considerable challenges with respect to their global data management, and the rate of entropy they are experiencing now will only accelerate as we advance into our connected future. By leveraging the benefits of a data platform that intelligently connects, secures, and governs their data assets, companies can ideally position themselves to capitalize on the tremendous opportunities that lie within their data.
If you’re looking for a way to manage your data from end to end, read more about the Hortonworks DataPlane Service.