cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button

Apache Kafka

A fast, scalable, fault-tolerant messaging system

Apache™ Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system. Kafka is often used in place of traditional message brokers like JMS and AMQP because of its higher throughput, reliability and replication.

Kafka works in combination with Apache Storm, Apache HBase and Apache Spark for real-time analysis and rendering of streaming data. Kafka can message geospatial data from a fleet of long-haul trucks or sensor data from heating and cooling equipment in office buildings. Whatever the industry or use case, Kafka brokers massive message streams for low-latency analysis in Enterprise Apache Hadoop.

What Kafka Does

Apache Kafka supports a wide range of use cases as a general-purpose messaging system for scenarios where high throughput, reliable delivery, and horizontal scalability are important. Apache Storm and Apache HBase both work very well in combination with Kafka. Common use cases include:

  • Stream Processing
  • Website Activity Tracking
  • Metrics Collection and Monitoring
  • Log Aggregation

Some of the important characteristics that make Kafka such an attractive option for these use cases include the following:

Feature Description
Scalability
    Distributed system scales easily with no downtime
Durability
    Persists messages on disk, and provides intra-cluster replication
Reliability
    Replicates data, supports multiple subscribers, and automatically balances consumers in case of failure
Performance
    High throughput for both publishing and subscribing, with disk structures that provide constant performance even with many terabytes of stored messages

How Kafka Works

Kafka’s system design can be thought of as that of a distributed commit log, where incoming data is written sequentially to disk. There are four main components involved in moving data in and out of Kafka:

  • Topics
  • Producers
  • Consumers
  • Brokers

Kafka Cluster Diagram

In Kafka, a Topic is a user-defined category to which messages are published. Kafka Producers publish messages to one or more topics and Consumers subscribe to topics and process the published messages. Finally, a Kafka cluster consists of one or more servers, called Brokers that manage the persistence and replication of message data (i.e. the commit log).

Kafka Partition Diagram

One of the keys to Kafka’s high performance is the simplicity of the brokers’ responsibilities. In Kafka, topics consist of one or more Partitions that are ordered, immutable sequences of messages. Since writes to a partition are sequential, this design greatly reduces the number of hard disk seeks (with their resulting latency).

Another factor contributing to Kafka’s performance and scalability is the fact that Kafka brokers are not responsible for keeping track of what messages have been consumed – that responsibility falls on the consumer. In traditional messaging systems such as JMS, the broker bore this responsibility, severely limiting the system’s ability to scale as the number of consumers increased.

Kafka Broker Diagram

For Kafka consumers, keeping track of which messages have been consumed (processed) is simply a matter of keeping track of an Offset, which is a sequential id number that uniquely identifies a message within a partition. Because Kafka retains all messages on disk (for a configurable amount of time), consumers can rewind or skip to any point in a partition simply by supplying an offset value. Finally, this design eliminates the potential for back-pressure when consumers process messages at different rates.

Hortonworks Focus for Kafka

The Apache Kafka community is working to improve security and to enable it to work with YARN as the architectural center of Apache Hadoop.

YARN enablement

  • Utilize all cluster resources included in the blueprint for Enterprise Hadoop
  • Minimize hardware footprint
  • Reduce total cost of ownership
Focus Planned Enhancements
Security
  • Integrate Kafka with Apache Ranger
  • Authentication with a Kerberized cluster
  • Authorization to publish to or subscribe to messaging queues
  • Encryption of data in motion
Operations
  • Management view — using the Ambari Views Framework to provide visibility to critical management and monitoring metrics.

Recent Kafka Releases

Kafka is fully supported and included in HDP today.

0.8.0

  • Intra-cluster replication support
  • Support multiple data directories
  • Asynchronous request processing
Apache Kafka Version Prior Enhancements
0.7.0
  • Support for block compression
  • Space-dependent log retention

Forums

Kafka in our Blog

Kafka in the Press

Webinars & Presentations