newsletter

Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

AVAILABLE NEWSLETTERS:

Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
September 05, 2018
prev slideNext slide

Solving the Needs of DevOps / Developers with Streams Messaging Manager

Two weeks ago, we announced the GA of Streams Messaging Manager (SMM) 1.0 to cure enterprises of their Kafka Blindness. We discussed how SMM addresses the needs of three groups in the enterprise: Platform Operations, DevOps/Developers and Security/Governance teams. In this second installment of the SMM blog series, we focus on the DevOps/Developers persona and explain how SMM addresses their unique needs and use cases.

DevOps/Developers Use Cases/Questions

The DevOps / Developers persona is most interested in the entities (producers, topics, consumers) specific to their applications. So let’s assume we are on the DevOps team responsible for monitoring a streaming analytics application that is deployed in production based on this architecture.

The below table represents a set of needs/requirements/questions that a DevOps user might have when monitoring this application.

Answering DevOps/Developers Questions with SMM

Lets walkthrough how SMM can answer these questions for DevOps/Developers.

  1. Select the Topic filter and and select all the IoT gateway topics by searching for all topics that start with gateway. 
  2. When the filter is applied, SMM displays intelligent filtering by showing only the producers that are sending data to the 4 gateway topics and the consumer groups only consuming data from those topics. So, when the user selects the 4 gateway topics, SMM displays 34 of the 83 producers sending data to those topics and 3 of the 26 consumer groups consuming data from it. Key metrics for the selected topics are shown including data-in/out, number of messages, number of consumer groups, etc. This addresses Use Cases 1 and 2. 
  3. Click on DATA-IN to sort on data throughput-in across all topics. We see that gateway-europe-raw-sensor has significantly more data coming in than any other topic: 18 MB totaling 88K in the last 30 minutes. This addresses Use Case 3. 
  4. Expand the topic panel for gateway-europe-raw-sensor to get more details of the topic like partition layout. Click on the topic to see who are all producers sending data to each partition of that topic. We see that all the producers are trucks from the EU fleet. Also note that all five partitions for that topic has 0 B going out and we see no data flowing from the topic to any consumer group. This could be worth investigating of why a topic with the most amount of producers has no consumers. This addresses Use Case 4 and 5. 
  5. Click on another topic called gateway-west-raw-sensors that has consumers. This topic has three producers sending data to it and a NiFi consumer consuming from it. We also see that two of the four partitions for this topic has no data in it which indicates there is partition skew issue. This addresses Use Case 6 and 7. 
  6. Click on the explorer/magnifying glass icon to search for events in the selected Kafka topic. This addresses Use Case 8. 

Finding Slow Consumers Is Just the Beginning – The Power of SMM-Apache Atlas Integration

In addition to the eight use cases defined above, one of the most critical and common use case for DevOps is to identify slow consumers. Slow consumers are defined as applications that cannot keep up with the rates at which producers generate data. This can result in apps not being able to identify actionable insights on time, etc. SMM makes it very easy to find slow consumers for a given topic through its intelligent filtering and sorting capabilities.

However, finding slow consumers is meaningless if the user is not able to quickly identify and understand what the consumer is doing to  cause the lag so that it can be addressed. This bring us to one of the most powerful and differentiating features of SMM. With SMM’s Atlas Integration, the user is able to see metadata about consumer client application allowing the user to quickly understand why the consumer is slow to process. Key metadata about Kafka consumer application include visibility into what the consumer application is doing, where the consumer application is running, and what downstream systems the app is talking to.

The short video below showcases these powerful capabilities in SMM including:

  • Identify a slow consumer
  • Explore details/metadata about the slow consumer app via the SMM/Apache Atlas Integration.
  • Navigate to where the slow consumer is running.
  • Make changes to the slow consumer app to increase the speed of processing.
  • Verify that the change made the consumer faster in processing data.

 

SMM has clearly changed the game for enterprises that are struggling with Kafka operational and visibility challenges. Specifically, SMM enables the DevOps / Developer personas to achieve complete control and gain deep visibility into their Kafka flows and streams. DevOps can seamlessly work across multiple Kafka environments from a single SMM instance. In a future post, I will cover another awesome capability of SMM – its REST server. The entire SMM UI is driven by a powerful set of REST endpoints that are made available to the developer. This enables developers to integrate any enterprise applications like ticketing/application monitoring apps with SMM.

What’s Next?

Register for our upcoming webinar “Curing Kafka Blindness with Hortonworks Streams Messaging Manager” on September 6th 2018. Dinesh and I will be showcasing the product in great detail with some exciting demos of the product as well. Join us!

Leave a Reply

Your email address will not be published. Required fields are marked *