Hadoop Summit in San Jose wrapped up a few weeks ago. This was the ninth year and, wow, have we come a long way. It’s been a decade for Apache Hadoop and five years for Hortonworks.
Hadoop Summit is the leading conference for Hadoop and data management, and this year saw well over 4,000 attendees coming together. Some even flying 10,000 miles around the world to get to San Jose. You could feel the buzz and the excitement everywhere from the exhibition hall to the keynotes, breakouts and BOFs, and to the ‘Ten Years of Hadoop’ party. We also witnessed that many first timers attended this Summit, with a dramatic increase in business users. This year was definitely a great cross section of “hoodies and suits.”
One highlight was the celebration of the community on stage on day one. We have all each other to thank for the tremendous success of the technology. There were over 170 sessions that were voted on by the Hadoop community and presented by the experts across the industry.
The conversation has shifted fundamentally, from technology to business value driven by digital and data transformations. It’s also about managing all the data-in-motion and data-at-rest and finding ways to solve problems and power the many enterprises who want to gain competitive advantages by harnessing the power of all data. Many companies are using Connected Data Platforms to achieve this advantage. From what we see, the market has evolved and there is a realization that everyone is focusing on all of their data.This year, we had more sponsors and partners than ever from across the industry, and we heard from lots of companies doing amazing things. Top level innovation sponsor Microsoft talked about how Algorithms, Cloud, IoT, and Data (ACID) is driving the revolution of ambient intelligence. Top sponsors EMC, BMC, Google Cloud Platform and HPE, talked Cloud, Data, and how they are helping customers achieve transformations.
Let me highlight a few other things that make Hadoop Summit unique in the industry.
A number of the Hortonworks keynotes focused on how modern enterprises are transforming as a result of data and the future of data including:
Rob Bearden talked about how being able to bring all of the data under management from point of origination to point of rest transforms virtually every
industry and allows them to evolve into new business models.
In retail, for example, to be able to analyze buying behavior down to the smart hanger and map this in real time to social signals and online visual search to provide a 360 degree customer view. In addition, Hortonworks named Microsoft Azure HDInsight as our Premier Connected Data Platforms Cloud Solution.
Arun Murthy drilled down into the current and future state of Hadoop and exciting community initiatives aimed at enabling the next wave of modern data apps that are well governed, secure and easy to deploy in data centers and the cloud. We heard the latest about Connected Data Platforms, ease of use for Spark and agile analytic development, and the ability to create assemblies of engines and services via a user friendly user experience and then run and operate them as single data application that can instantly scale up and down. Plus, how we have worked closely with partners and the community to integrate security and governance.
Sanjay Radia discussed the technical underpinnings and next wave of investments aimed at enabling Hadoop in the cloud to truly shine. Sanjay touched on key technical considerations related to shared data, shared metadata, and SLA-driven processing. And since speed to value is a key driver for embracing cloud, a look at enabling rapid deployment for the most common use cases will also be covered
Across the conference, we also heard a lot from customers reinforcing the same themes. From every industry we heard about transformations driven by data for leading edge companies. For the first time, Hadoop Summit had a business track where 20 customer speakers including Arizona State University (ASU), Comcast, Capital One, Express, Ford, GE Digital, Macy’s, Progressive, RBC, Salesforce.com, Schlumberger and Webtrends presented their use of the platform to transform their businesses.
Eight customers including Progressive, ASU, Comcast, Macy’s and Capital One also participated in keynotes. During the customer a keynote panel, Seetha Chakrapany from Macy’s described how the 150-year-old brand is transforming its business in the digital economy. He said, “[Retail is] a highly competitive industry and we have to continuously innovate and push the envelope. Big data solutions are making us much smarter about how we communicate to customers. Now can collect as much history as possible and scale. When you are looking at a customer you need to understand the life stage and we need to have all their history available at any point in time so we can be contextually relevant.”
Beata Puncevic of Blue Cross Blue Shield of Michigan commented, “The potential in the healthcare industry is tremendous … we have an opportunity now to allow us to say what’s the right treatment and what’s the right way to apply that treatment. Who is the right provider? What’s the right hospital for you? These are all things that improve the quality of life, which is very much our mission, but also to reduce the cost of healthcare for the entire ecosystem.”
Kelly Cook of Conoco Phillips added, “We’re fairly new at this… but the technology is pregnant with opportunity for us. We brought (Hadoop) in as a less expensive alternative and a way to defer upgrades or expansion. One of the use cases is advanced analytics and machine learning. Really taking the data and figuring out how we can apply the resources of a Hadoop cluster to the next generation of learning. We are looking at literally anything and everything across the organization that we can optimize, from supply chain assets to the maintenance schedules to the operation of wells.”
Separately, Andrew Leamon from Comcast discussed his company’s approach to respond to real-time data and how this is improving Customer Experience. Comcast is making better business decisions with a shorter time to insight as open-source technologies becomes a force multiplier for the organization.
When used properly, big data leads companies down a path to more personalized interactions with their customers and influences business strategy. Brian Durkin and Pawan Divakarla from Progressive Insurance talked about how one of the country’s top insurers continues to unlock answers hidden in more than 15 billion miles of driving data.
Ultimately, Hadoop Summit is all about a vibrant and ever growing community coming together to leverage open technologies to transform business, and potentially to change the world.
In this vein, Jay A. Etchings and Dr. Ken Buetow, of ASU, talked about how they leverage computational tools to understand complex traits such as cancer, liver disease, and obesity. ASU has joined Baylor College of Medicine, Booz Allen Hamilton, Mayo Clinic, OneOme and Yale New Haven Health in a consortium of experts and organizations who will help to define the next generation of genomics research.
Hortonworks has set this consortium up following our support of the White House’s Precision Medicine Initiative and we believe it will be a huge step forward in accelerating genomics-based precision medicine. Only together like this can we address diseases like cancer with data. This was a ‘wow moment’ for many attendees.
Finally, we announced an evolution of the name of Summit next year and an expansion of its role, from Hadoop Summit to Dataworks Summit. Of course, as the defacto platform for all of this transformation, Hadoop will still be at the core but we’ve seen such a rapid interest, expansion and connection to new open projects like Spark and business themes that we needed to broaden the scope.
We can see the continued growth trajectory at Hortonworks in our business and we are already planning to stay ahead of it by making adjustments to our business to reflect the appetite in the market, with an even greater investment in our customer facing functions globally.
If you made it to San Jose, I hope you enjoyed it as much as we did! If you didn’t you can find our keynote videos here.
See you at the next Hadoop Summits in Melbourne and Tokyo.