This guest post is from Gavin Sherry, Vice President of Engineering, Data, at Pivotal. A long time contributor to database technology, Gavin was one of the early contributors to the PostgreSQL project. This led him to join the Greenplum Database R&D team. More recently, Gavin launched Pivotal HAWQ, Pivotal’s SQL on Hadoop engine.
In the ten years since Hadoop was first conceived at Yahoo!, the big data market has taken off. Yet, with this incredible growth has come fragmentation. There are multiple distributions to choose from, many comprised largely of vendor specific extensions. Enterprise customers looking to harden their Hadoop deployments for production use cases find themselves stuck: they must either customize their workloads to a particular distribution, limit their use of Hadoop, or deploy a complex multi-vendor Hadoop distribution infrastructure.
Today, Pivotal is joining forces with Hortonworks to lead the charge to help enterprises worldwide achieve more and faster business value with Hadoop, along with industry leaders like IBM, Verizon, Infosys, General Electric, and SAS. Our goal is to create a standardized core platform of compatible versions of Apache Software Foundation projects that provide a stable base against which Big Data solutions providers can qualify solutions. This will facilitate compatibility and ease interoperability across the Big Data ecosystem while enabling a new level of choice for enterprises. Pivotal and Hortonworks are working together to ensure our respective platforms are the first to meet these expectations.
Pivotal and Hortonworks are founding members of an initiative called Open Data Platform (ODP) whose mission will be to enable collaboration between vendors and end users of Big Data technology. It will do this by jointly developing and promoting a common core of Apache Hadoop related technologies and enhancing compatibility amongst Hadoop related projects by driving interest amongst vendors and enterprise users in contributing to Apache Hadoop related projects according to ASF guidelines.
Our hope is that ODP will simplify the process development, testing and certification for Big Data users, application developers and Big Data software vendors.
Open Data Platform members intend to base their commercial Hadoop distributions on this core offering called the Open Data Platform (ODP) Core. Joined by a number of other Hadoop ecosystem vendors, solution providers and enterprise users of Hadoop, each member of the initiative will invest in the Open Data Platform to drive enterprise-grade capabilities.
In a blog post announcing the formation of the Open Data Platform, Scott Yara illustrates how this will accelerate more choice and lower business risk for enterprises relying on Hadoop.
Ahead of the Open Data Platform’s formation, Pivotal and Hortonworks have been aligning efforts around Hadoop, with a shared goal of enabling a large ecosystem of analytics frameworks and tools to interoperate with our respective platforms out of the box. Much of this work will serve as a starting place for the Open Data Platform Core.
In doing this, enterprises will have the choice of using either Pivotal HD or the Hortonworks Data Platform (HDP) interchangeably with components of the new Pivotal Big Data Suite 2.0, also announced today.
In addition, we’re excited to announce that Hortonworks will provide advanced engineering support for both distributions of Hadoop, allowing enterprise customers to receive consistent world-class Hadoop support services.
The SQL and In-memory components of the Pivotal Big Data Suite running on Hortonworks Data Platform serves as a powerful example of how the Open Data Platform initiative will change the big data ecosystem forever. Until now, customers running the Hortonworks Data Platform were unable to leverage the secret sauce of the Pivotal Big Data Suite: Pivotal’s advanced SQL on Hadoop engine, HAWQ; Pivotal’s in-memory technologies, Pivotal GemFire; nor, Pivotal’s massively parallel processing data warehousing system, the Greenplum Database. With the formation of ODP and joint collaboration between Pivotal and Hortonworks, enterprises will be able to build the infrastructure they choose, with industry leaders supporting them.
Last year, while working within the Ambari open source project to create an industry standard way for administering Hadoop clusters, Pivotal had its first opportunity to team up with Hortonworks on a key Hadoop related project. In so doing, we found a great partner in Hortonworks, and came to realize that both our companies share similar philosophies about how to advance innovation through collaboration in open source.
At the same time, industry-wide open collaboration on another open source project, Cloud Foundry, was reaping its first benefits and exciting customers and partners alike. Hortonworks recognized the value that collaboration and partnership brings to the market as a whole, and agreed to spearhead the maturation of the Hadoop ecosystem with Pivotal.
At Pivotal, we are incredibly excited by this alliance and the Open Data Platform, and look forward to seeing what a difference we can make for our customers and the industry.