Organizations today are capturing, storing, and processing massive volumes of data, but failing to unlock the value. According to Wikibon’s 2017 Big Data Worldwide Forecast, failure rates with big data projects remain stubbornly high. This poses a major challenge for business leaders, who struggle to find ways to derive value from their growing stockpiles of digital information. Why aren’t these big data projects succeeding? And how can organizations fix what’s broken?
“Big data” is a nebulous term that has been around for so long that it’s passed through the Gartner Hype Cycle for Emerging Technologies, meaning it’s already undergone the hype, backlash, and ultimate acceptance that accompany most tech innovations. Definitions vary, of course, but essentially “big data” refers to the enormous data sets that capture, store, and process everything from emails to audio recordings, video clips, photographs, and even social media posts. This motley stockpile of bits is often called “unstructured data,” because it isn’t organized in a structured way, unlike the content in a traditional column-and-row database that’s built for fast search and retrieval.
Too often, organizations try to use legacy tools to extract value from big data—think of the creaky IBM or Oracle systems of yesterday—or they simply aren’t sure what to do with all the information they’re amassing. It’s not that the tech isn’t there, it’s that people haven’t gone about it in the right way: their strategy isn’t fully formed, or their commitment is lacking, or the application of the technologies is lagging.
A December 2016 study from the McKinsey Global Institute (MGI) explores the “transformational potential of big data,” a topic it first studied back in 2011. Its conclusion is that the promise of big data analytics has not been overhyped, but its progress has been “uneven,” with a great deal of its value “still on the table.” MGI notes that big data’s report card is mixed thus far: Its greatest success has occurred in location-based services in U.S. retail, including areas where competitors are “digital natives”—most notably Amazon. On the flip side, manufacturing, healthcare, and the EU public sector have captured less than 30 percent of the potential value of big data that MGI spotlighted five years ago.
One of the elements of a digital transformation is to revolutionize the customer experience. It’s about establishing a single view of a specific subject—say, a retail customer, healthcare patient, or company employee—and using it to drive your strategy forward. This single-view approach can correct organizational shortcomings that often make it hard to see the big picture. With a legacy solution, for instance, a healthcare company’s accounting team might use its own database to manage a payment system, while the customer care team saves call logs in a customer relationship management platform, and the clinic stores patient data in an electronic medical record system.
These corporate fiefdoms don’t share data easily, if at all. Utilizing big data requires establishing the three pillars of digital transformation: operational efficiency, agile analytics, and the ability to extract maximum value from data by asking the right questions. By gaining a single view of big data, organizations extract value from information that might otherwise have been overlooked because it was scattered across multiple channels, groups, and platforms.
Big data streams, such as those from social media networks, can make or break your business. For instance, hundreds of thousands of product and service reviews are posted daily on Yelp, Amazon, Facebook, Twitter, and other social outlets—and these will determine the success of your product.
Digital transformation becomes even more critical in the nascent Internet of Anything (IoA) era, where tens of billions of wearables, sensors, mobile devices, and other connected “things” are generating a staggering amount of data, which is doubling every two years. Analyzing past events is no longer enough: organizations must nudge themselves toward the next big thing. This requires platforms that connect data-at-rest—say, information archived on a corporate server—and data-in-motion, such as real-time retail promotions or sensor data from manufacturing lines.
To truly unlock actionable insights, an effective digital transformation must build the intersection between active and inactive data. According to Gartner, 32 percent of businesses that embarked on a digital transformation in 2015 said their operations are now digital businesses—meaning they’ve become data-driven organizations in which data is arguably their most valuable asset.
In the end, organizations must learn how to tell stories with their data. In this series, we’ll explore how businesses can utilize big data effectively and head down the path toward digital transformation. Says Piet Loubser, vice president of product and solutions marketing at Hortonworks, this journey “is about the people, it’s about the culture, and it’s also about your strategy—building the right strategy.”
For more information on working with large data sets, check out the Hortonworks Data Platform (HDP) data discovery tools and solutions.