Last week, we discussed how the omnipresence of Kafka in application architectures has led to the Kafka Blindness afflicting DevOps/AppDev, Platform Operations and Security/Goverenance teams within the enterprise. These groups are struggling to effectively battle this blindness with the current tooling available.
Over the last 12 months, the Hortonworks team has been working with these customers to come up a more effective solution to fight Kafka Blindness.
Today, we are excited to announce the culmination of this effort as Hortonworks Streams Messaging Manager (SMM). SMM is a brand new open source component that aims to cure Kafka Blindness by providing a single monitoring platform.
The product and engineering teams at Hortonworks conducted 20+ interviews with subject matter experts at our biggest Kafka customers to understand how to cure this illness. Based on the corpus of data points collected from these interviews, four key design principles clearly emerged that shaped how SMM was built.
When teams work with Kafka, they often visualize data streaming across 4 key entities: producer clients, topics, brokers, and consumer clients. Kafka producer clients publish data to Kafka topics. These Kafka topics are broken up into partitions that reside on different Kafka brokers. Then Kafka consumer clients consume data from different topics.
This meta-model of how data streams throughout the different Kafka components plays fundamental role of how users use SMM to monitor and troubleshoot their Kafka clusters.
For an AppDev/DevOps user, he/she is only interested in the entities associated with the apps that they are monitoring. SMM provides two powerful, distinctive features that will help users quickly hone in on entities of interest and see how data flows across them.
The first of these features is what we call intelligent filtering. SMM provides intelligent filtering so that when a user selects a producer, broker, topic or consumer, only related entities are shown. For example, to filter all the IOT gateway Kafka topics, the user would use the topic filter to search for “gateway” which results in 4 of these gateway topics. When the filter is executed, SMM intelligently applies other filters such that only producers sending data to those 4 topics and only consumer groups consuming from them are displayed. This intelligent filtering gets applied regardless of the entry point. If you filter by producers or consumers, this same intelligent filter gets applied. This enables users to quickly hone in on entities of interest when troubleshooting and debugging Kafka issues.
The second powerful feature is about visualizing how data streams/flows across your Kafka cluster. You can select any entity and visualize how data flows with respect to the entity selected.
The below short video showcases these two powerful features and others.
SMM will be a game changer helping various teams in the enterprise to get over their Kafka blindness. However, most of these organizations will also want to integrate SMM with their other enterprise tools (e.g: APM, case/ticketing, etc.) Thus, clean REST endpoints are exposed for each of the capabilities that SMM offers. The below shows a swagger view of a subset of different endpoints available in SMM.
For Kafka to become the de facto enterprise streaming event standard in the enterprise, it must adhere to the security, governance and monitoring standards of the organization. To meet these needs, SMM has been fully integrated with platform services like Atlas for governance and lineage needs, Ranger for security and access control management, Ambari for infrastructure level monitoring and lifecycle actions for the cluster and Grafana to be able to graph Kafka metrics over time. See below for an example of SMM and Atlas Integration.
SMM is the first component in the HDF portfolio to be delivered as a DataPlane application. This design decision provides three powerful benefits.
The below shows the SMM app being enabled for a Kafka cluster in the DataPlane platform.
This blog was intended to get you introduced to SMM and is the first installment in the SMM Blog series. In the next blog of the series, we will walk through how Platform Operations, DevOps and Security/Governance teams use SMM to address their needs/questions. In the meanwhile, check out www.hortonworks.com/smm.