This is a unique moment in time. Fueled by open source, Apache Hadoop has become an essential part of the modern enterprise data architecture and the Hadoop market is accelerating at an amazing rate.
The impressive thing about successful open source projects is the pace of the “release early, release often” development cycle, also known as upstream innovation. The process moves through major and minor releases at a regular clip and the downstream users get to pick the releases and versions they want to consume for their specific needs.
Assembling a complete platform like HDP requires choosing the right stable version of Apache Hadoop as the foundation and then integrating, and packaging the optimal versions of all the other ASF components into a well-tested, certified data platform.
“Open Source Communities: The Developer Kingdom” adds a little more color around this process.
Since we are committed to delivering HDP completely in the open, we introduce enterprise feature requirements into the public domain and we code to address those requirements. We code and we contribute everything back to the wide array of ASF projects in HDP. Why? Because we want the Hadoop market to work at scale, and in order to make this happen, we know code is king, so we practice what we preach. We contribute everything.
The enterprise-focused initiatives are an important element of our approach. The Stinger Initiative has successfully rallied contributions from hundreds of developers across dozens of companies in order to address SQL in Hadoop needs for the enterprise. The Data Governance Initiative was recently formed with Aetna, Merck, Target and SAS to address the data stewardship, lineage, lifecycle management, and privacy issues that are increasingly important.
The Open Data Platform initiative (ODP), announced this week, aims to rally enterprise end users and vendors alike around a well-defined common core platform (the ODP Core) against which big data solutions can be qualified.
It’s simple. All upstream production happens within the ASF projects according to the ASF governance model. Individuals working for ODP member companies are encouraged to participate and contribute to ASF projects as they see fit and in accordance to ASF processes. Since Hortonworks engineers do all of their coding in these ASF projects, we’re more than happy to help newcomers learn the Apache way and contribute.
The ODP, on the other hand, is focused on enabling downstream consumption of a common set of Hadoop-related components, and more importantly, specific versions of those components. Harmonizing the broader market around Apache Hadoop version 2.6 and Apache Ambari 2.0, for example, will help simplify the onboarding of manageable, YARN-based solutions that can ride atop the common core platform.
Increasing the compatibility among Hadoop-based platforms and solutions will free up the broader big data ecosystem to focus on more important things such as data-driven applications that deliver proactive insights for the business. Innovation will advance even faster in the market with all the ODP members building upon the same downstream Apache Hadoop kernel- Apache Hadoop, Yarn and Apache Ambari.
At Hortonworks, our founding belief is that innovation and adoption of platform technologies like Hadoop is best accomplished through collaborative open source development under the governance model of an entity like the Apache Software Foundation (ASF).
In order to enable a data platform like Hadoop to be easy to use and enterprise-grade, you don’t go it alone. You do it by working with your customers and the broader ecosystem to enable:
Our approach to the market is about enabling our customers to embrace Hadoop in a way that makes sense for their business. It’s about enabling our partners in a way that drives joint value from the alliance in a way that’s respectful to each other in the process. And it’s about rallying a community in a way that drives innovation around shared goals.
Done right, open source promotes an equitable balance of power; done together, it offers a fair exchange of value between vendor and consumer.