Announcing the General Availability of Hortonworks Data Platform (HDP) 2.6.5, Apache Ambari 2.6.2 and SmartSense 1.4.5

We are excited to make several product announcements including the general availability of :

• HDP 2.6.5
  • Apache Kafka 1.0
  • Apache Spark 2.3
• Apache Ambari 2.6.2
• SmartSense 1.4.5

HDP 2.6.5 is an important release for Hortonworks given it is the first release that enables Apache Kafka 1.0 and Apache Spark 2.3

Hortonworks Data Platform 2.6.5

With this release, we achieved GA for Apache Kafka 1.0 and Apache Spark 2.3, as well as fixed bugs andadded key enhancements to the individual components.

Some of the new new features and enhancements include:

Apache Spark:

• ORC/Parquet Feature Parity
  • Spark extends its vectorized read capability to ORC data sources.
  • Structured streaming officially supports ORC data source with API and documentation
• Python Pandas UDF, with good performance and easy to use for Pandas users. This feature supports financial analysis use cases.
• Structured streaming now supports stream-stream joins.
• Structured streaming that goes to millisecond latency (Alpha). New continuous processing mode provides the best performance by minimizing the latency without waiting in idle status.

Apache Kafka:

• Apache Kafka 1.0 is now GA and provides important new features including more stringent message processing semantics and support for message headers and transactions, performance improvements, and advanced security options.
• Significant improvements to the Kafka Controller that speed up controlled shutdown. ZooKeeper session expiration edge cases have also been fixed as part of this effort.
• Better diagnostics for simple authentication and security layer (SASL) authentication failures, better handling of disk failures, and the Streams builder API has been cleaned up to be easier to use.

Apache Atlas

• Performance improvements in Hive-Atlas hook
• Optimizations of Kafka notifications processing in Atlas to provide better throughput and lower latency during metadata capture

Apache Ranger

• Ranger Audit infrastructure performance optimization via  better Solr memory utilization
• One way SSL support for MySQL

Apache Knox

• Secure data during movement between clusters and cloud by passing SSO Cookies through to Proxied APIs

Ambari 2.6.2 – Release Notes

The latest maintenance release of Ambari contains a number of fixes. For more details on the specific fixes, please see the Release Notes.

SmartSense 1.4.5 – Release Notes

• Added Diagnostic Data Capture for Druid and Superset
• Increased key size for stronger encryption of SmartSense bundles

Notable Details for HDP 2.6.5

Product Documentation:

• Release Notes
• New Features
• Support Matrix
• HDP 2.6.5 Software Repositories
• Ambari 2.6.2  Installation Guide
• SmartSense 1.4.5 Installation Guide
• The full documentation set at docs.hortonworks.com