The components in a modern data architecture vary from one enterprise to the next and the mix changes over time. Many of our Hortonworks subscribers need support ensuring that their Hortonworks Data Platform (HDP) clusters are optimally configured. This means that they need proactive, intelligent cluster analysis.
As businesses onboard new workloads to the platform, it taxes the resources of Hadoop operators. And so our customers have asked Hortonworks for guidance and best practices to reduce their operational risk and efficiently resource their staff for Hadoop operations.
Many of the best practices that Hortonworks has developed over the years of working with Apache Hadoop take into account a number of cluster diagnostic variables that take time to collect and analyze.
Apache Ambari helps analyze these variables and allows our customers to understand the health of their cluster through Ambari’s single pane of glass that manages its configuration and services. As Open Enterprise Hadoop becomes evermore critical to enterprise data management, companies need a more proactive approach to achieving the optimal configuration.
Late last year, Hortonworks began to outline objectives for a new proactive support service that would add value for customers with:
So Hortonworks decided to focus our efforts in areas with the biggest potential impact. This included providing tools to quickly capture cluster diagnostics and display them in one central location, both for support case resolution, and as input to an analytical service that can produce configuration-related recommendations to improve cluster performance, security, and operations.
This was the genesis of Hortonworks SmartSense, which is a collection of tools and services that help Hortonworks Data Platform’s operators quickly resolve issues, and also act on proactive recommendations that help avoid future issues.
The first step in the process is to quickly capture cluster diagnostic information. To accomplish this, we’ve created a tool called the Hortonworks Support Tool, or HST for short. HST plugs into Ambari and allows Hadoop Operators to quickly combine and display cluster diagnostic information in a single bundle that can be attached to a support case for troubleshooting, or analyzed by Hortonworks SmartSense.
Hortonworks SmartSense then analyzes that diagnostic information and produces recommended configurations affecting performance, security, and operations. For the upcoming release of Hortonworks SmartSense, we plan to deliver recommendations for the following components:
Hortonworks research shows that recommendations related to the components mentioned above will provide the most value to our support customers, because these components are the most heavily used and they cause the largest number of configuration-related support cases. In fact, our analysis has shown that across all components in HDP, 25%-30% of support cases are created when the configuration of a component has not kept pace with that component’s actual use. That is when Hortonworks SmartSense recommends changes for optimization..
As customers mature with their use of Open Enterprise Hadoop, they use HDP’s components in more complex ways, they add new users with different data types and workloads, and they need to update their cluster configuration to maintain optimal performance, security, and operations. Hortonworks SmartSense facilitates these updates with dynamic recommendations for updates as configuration calculations, best practices, field experiences, and real-world operating conditions evolve.
Once HST collects the data and analyzes the bundle, new recommendations are displayed in the Hortonworks Support Portal. Each recommendation includes:
All of this information enables Hadoop operators to quickly evaluate the proposed recommendation, and then either apply it (or defer making the change if the timing is not right).
In summary, Hortonworks SmartSense enables customers to take advantage of a new proactive service in any Hortonworks subscription that provides faster support case resolution by easily capturing log files and metrics for insight into the root causes of issues. Hortonworks SmartSense also provides proactive cluster configuration via an intelligent stream of cluster analytics and data-driven recommendations.
Our goal at Hortonworks is to continue providing the world’s best support experience for Hadoop., Hortonworks SmartSense is the next step in that journey. We have further enhanced the value of our support by making Hortonworks SmartSense part of every HDP subscription.