We are excited to make several product announcements including the general availability of :
- HDP 2.6.5
- Apache Kafka 1.0
- Apache Spark 2.3
- Apache Ambari 2.6.2
- SmartSense 1.4.5
HDP 2.6.5 is an important release for Hortonworks given it is the first release that enables Apache Kafka 1.0 and Apache Spark 2.3
Hortonworks Data Platform 2.6.5
With this release, we achieved GA for Apache Kafka 1.0 and Apache Spark 2.3, as well as fixed bugs andadded key enhancements to the individual components.
Some of the new new features and enhancements include:
- ORC/Parquet Feature Parity
- Spark extends its vectorized read capability to ORC data sources.
- Structured streaming officially supports ORC data source with API and documentation
- Python Pandas UDF, with good performance and easy to use for Pandas users. This feature supports financial analysis use cases.
- Structured streaming now supports stream-stream joins.
- Structured streaming that goes to millisecond latency (Alpha). New continuous processing mode provides the best performance by minimizing the latency without waiting in idle status.
- Apache Kafka 1.0 is now GA and provides important new features including more stringent message processing semantics and support for message headers and transactions, performance improvements, and advanced security options.
- Significant improvements to the Kafka Controller that speed up controlled shutdown. ZooKeeper session expiration edge cases have also been fixed as part of this effort.
- Better diagnostics for simple authentication and security layer (SASL) authentication failures, better handling of disk failures, and the Streams builder API has been cleaned up to be easier to use.
- Performance improvements in Hive-Atlas hook
- Optimizations of Kafka notifications processing in Atlas to provide better throughput and lower latency during metadata capture
- Ranger Audit infrastructure performance optimization via better Solr memory utilization
- One way SSL support for MySQL
- Secure data during movement between clusters and cloud by passing SSO Cookies through to Proxied APIs
The latest maintenance release of Ambari contains a number of fixes. For more details on the specific fixes, please see the Release Notes.
- Added Diagnostic Data Capture for Druid and Superset
- Increased key size for stronger encryption of SmartSense bundles
Notable Details for HDP 2.6.5