The pace of innovation within the Apache Hadoop community is truly remarkable, enabling us to announce the availability of Hortonworks Data Platform 2.1, incorporating the very latest innovations from the Hadoop community in an integrated, tested, and completely open enterprise data platform.
The advancements in HDP 2.1 span every aspect of Enterprise Hadoop: from data management, data access, integration & governance, security and operations. All of this is delivered via Apache Software Foundation projects. While there are many enhancements to all projects, below are just a few key highlights of HDP 2.1.
HDP delivers on the commitments made last year with the final phase of the Stinger Initiative; a concerted effort to improve the performance of Apache Hive and SQL query in Hadoop. Apache Hive is already the most-widely used data access engine for Hadoop. And for good reason as it also has the widest commitment of community development.
On top of the innovations in YARN and Apache Tez, 145 developers across 45 unique companies (Microsoft, SAP, Facebook, Hortonworks to name just a few) have contributed over 390,000 lines of code to Apache Hive.
HDP 2.1 includes Apache Falcon, an open source project that delivers a reliable, repeatable and simple framework for managing the flow of data in and out of Hadoop. This control structure, along with a set of tooling to ease and automate the application of schema or metadata on sources, is critical for the successful integration of Hadoop into your modern data architecture.
Security is addressed in Hadoop across multiple layers and in this release we added numerous new security advances; most notable include ACLs for HDFS and Grant/Revoke functions for Apache Hive. However, the largest security advancement is the addition of Apache Knox.
Apache Knox provides perimeter security through a single point of authentication/access for your cluster and integrates with your existing Active Directory or LDAP implementations. Again, Knox is an example of a key technology being contributed to open source by a major ecosystem vendor and is indicative of the community force behind the delivery of the Enterprise Hadoop platform completely in open source.
Stream processing has emerged as a key use case for Hadoop and as a result we have been supporting Apache Storm for stream processing within dozens of our customers. As announced when we initiated the work several months ago, we are now including Storm as a native component of the Hortonworks Data Platform.
Another key use case for Hadoop is Search, and we are extremely pleased to announce that we are adding support for Apache Solr in HDP 2.1 to enable native search functionality. Apache Solr extends Hadoop with a powerful user interface for advanced search applications that unlocks a range of use cases focused on user search across very large data sets stored in Hadoop.
HDP 2.1 includes the very latest version of Apache Ambari which supports new platform services including Storm, Falcon, and Tez, provides extensibility and rolling restarts, as well as other significant operational improvements.
This is clearly a major milestone for the Hadoop community and a significant release of the Hortonworks Data Platform and a full list of capabilities can be found in the release notes.
We have made available a single VM download of HDP 2.1 so that users can get started today, while a complete version of the product for both Linux and Windows will be available later in April.