Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Sign up for the Developers Newsletter

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.


Get Started


Ready to Get Started?

Download sandbox

How can we help you?

* I understand I can unsubscribe at any time. I also acknowledge the additional information found in Hortonworks Privacy Policy.
closeClose button
December 18, 2018
prev slideNext slide

Monitoring Kafka Streams Microservices with Hortonworks Streams Messaging Manager (SMM)

In last week’s blog Secure and Governed Microservices with HDF/HDP Kafka Streams Support, we walked through how to build microservices with the new Kafka Streams support in HDF 3.3 and HDP 3.1 that is fully integrated with Ranger, Schema Registry and other platform services. This blog is all about monitoring these microservices with Hortonworks Streams Messaging Manager (SMM).

Monitoring MicroServices with SMM

In a microservices architecture, you will see a proliferation of stand-alone decoupled services. Hence, monitoring and managing these services becomes extremely critical. SMM provides users a powerful tool to monitor and visualize their microservices and understand how data flows across these services.

Working off the trucking fleet use case example from the previous blog, you can view each of the three microservices as a consumer and producer as depicted in the below diagram.

Lets first focus on monitoring the stream between MicroService 1 and MicroService 2 where MicroService 1 is a Kafka producer into the driver-violation-events topic and MicroServce 2 is a consumer from that topic. The below video showcases how to monitor the interactions between these two Kafka Streams microservices.


As the above video showcased, SMM cured the Kafka blindness for the streams app between the two microservices shedding light on some important information including the rate at which MicroService 1 was producing data, the lag of MicroService 2 and understanding the details of the internal Kafka changelog topics created by the Kafka streams join operator.

Another common use case for monitoring Kafka streams application using SMM is the following:

  • Detect a Kafka Streams microservice that is lagging considerably behind in processing messages.
  • Scale out the Kafka Streams microservice by spinning up new instances of the app (adding new instances to the consumer group).
  • Validate that adding new instances has decreased the lag of the consumer group.

The below video showcases how SMM address this use case.


The addition of Kafka Streams to HDF 3.3 and HDP 3.1 integrated with platform services like Ranger, Schema Registry and other platform services provides app developers a comprehensive platform to build secure and governed microservices. With SMM, devops and platform operations teams have enterprise  tools to debug, monitor, and troubleshoot these microservices built using Kafka Streams.

What’s Next

In the next installment of the Kafka Analytics blog series, we walk through the new Hive and Kafka integration for the SQL access pattern.

Leave a Reply

Your email address will not be published. Required fields are marked *