Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
June 11, 2015
prev slideNext slide

Introducing Hortonworks SmartSense

The components in a modern data architecture vary from one enterprise to the next and the mix changes over time. Many of our Hortonworks subscribers need support ensuring that their Hortonworks Data Platform (HDP) clusters are optimally configured. This means that they need proactive, intelligent cluster analysis.

As businesses onboard new workloads to the platform, it taxes the resources of Hadoop operators. And so our customers have asked Hortonworks for guidance and best practices to reduce their operational risk and efficiently resource their staff for Hadoop operations.

Proactive Support

Many of the best practices that Hortonworks has developed over the years of working with Apache Hadoop take into account a number of cluster diagnostic variables that take time to collect and analyze.

Apache Ambari helps analyze these variables and allows our customers to understand the health of their cluster through Ambari’s single pane of glass that manages its configuration and services. As Open Enterprise Hadoop becomes evermore critical to enterprise data management, companies need a more proactive approach to achieving the optimal configuration.

Late last year, Hortonworks began to outline objectives for a new proactive support service that would add value for customers with:

  • Rapid collection of cluster diagnostic information
  • Concise and actionable recommendations for resource-constrained Hadoop operations staff
  • Proactive views of configuration problems before they result in cluster degradation or downtime
  • Dashboards on cluster configuration that help the ecosystem keep pace with changing cluster topologies and workloads

So Hortonworks decided to focus our efforts in areas with the biggest potential impact. This included providing tools to quickly capture cluster diagnostics and display them in one central location, both for support case resolution, and as input to an analytical service that can produce configuration-related recommendations to improve cluster performance, security, and operations.

This was the genesis of Hortonworks SmartSense, which is a collection of tools and services that help Hortonworks Data Platform’s operators quickly resolve issues, and also act on proactive recommendations that help avoid future issues.

ss_1

Hortonworks SmartSense – An Insider’s View

The first step in the process is to quickly capture cluster diagnostic information. To accomplish this, we’ve created a tool called the Hortonworks Support Tool, or HST for short. HST plugs into Ambari and allows Hadoop Operators to quickly combine and display cluster diagnostic information in a single bundle that can be attached to a support case for troubleshooting, or analyzed by Hortonworks SmartSense.

Hortonworks SmartSense then analyzes that diagnostic information and produces recommended configurations affecting performance, security, and operations. For the upcoming release of Hortonworks SmartSense, we plan to deliver recommendations for the following components:

  • The Operating System
  • HDFS
  • YARN
  • MapReduce2
  • Apache Hive and Apache Tez

 

ss_2

Hortonworks research shows that recommendations related to the components mentioned above will provide the most value to our support customers, because these components are the most heavily used and they cause the largest number of configuration-related support cases. In fact, our analysis has shown that across all components in HDP, 25%-30% of support cases are created when the configuration of a component has not kept pace with that component’s actual use. That is when Hortonworks SmartSense recommends changes for optimization..

As customers mature with their use of Open Enterprise Hadoop, they use HDP’s components in more complex ways, they add new users with different data types and workloads, and they need to update their cluster configuration to maintain optimal performance, security, and operations. Hortonworks SmartSense facilitates these updates with dynamic recommendations for updates as configuration calculations, best practices, field experiences, and real-world operating conditions evolve.

Once HST collects the data and analyzes the bundle, new recommendations are displayed in the Hortonworks Support Portal. Each recommendation includes:

  • A description of the recommended change, with a justification of that recommendation
  • Specific steps to follow in order to apply the recommendation. Commonly, this is done using Apache Ambari
  • A list of the components and services that are affected
  • A description of associated risks or potential side effects of implementing the recommendation
  • An indication of the hosts that will be affected by the recommendation

All of this information enables Hadoop operators to quickly evaluate the proposed recommendation, and then either apply it (or defer making the change if the timing is not right).

Conclusion

In summary, Hortonworks SmartSense enables customers to take advantage of a new proactive service in any Hortonworks subscription that provides faster support case resolution by easily capturing log files and metrics for insight into the root causes of issues. Hortonworks SmartSense also provides proactive cluster configuration via an intelligent stream of cluster analytics and data-driven recommendations.

Our goal at Hortonworks is to continue providing the world’s best support experience for Hadoop., Hortonworks SmartSense is the next step in that journey. We have further enhanced the value of our support by making Hortonworks SmartSense part of every HDP subscription.

Learn More

Tags:

Comments

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    If you have specific technical questions, please post them in the Forums

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>