Monitoring Apache Kafka using the ELK stack

One can effectively monitor Apache Kafka infrastructure using the ELK stack (Elasticsearch, Logstash, Kibana). Here’s a breakdown of how to achieve this:

1. Data Collection:

You have a few primary ways to get Kafka-related data into your ELK stack:

• Filebeat with the Kafka Module: This is the most common and recommended approach for collecting Kafka logs.
  • Functionality: The Filebeat Kafka module is specifically designed to collect and parse Kafka server logs, controller logs, and potentially other Kafka component logs. It understands the common log formats and structures them into fields that Elasticsearch can easily index and Kibana can visualize.
  • Configuration: You’ll need to enable the Kafka module in your filebeat.yml file and configure the paths to your Kafka log files (e.g., server.log, controller.log, state-change.log).
  • Setup: After configuring the module, run filebeat modules enable kafka and filebeat setup -e to load the necessary index templates and Kibana dashboards.
  • Pros: Easy to set up, handles parsing of Kafka logs, provides pre-built Kibana dashboards for quick visualization.
  • Cons: Primarily focuses on log data, might not provide detailed real-time metrics out-of-the-box.
• Metricbeat with JMX: To monitor Kafka metrics (broker health, topic details, consumer lag, etc.), you can use Metricbeat’s Java Management Extensions (JMX) metricset.
  • Functionality: JMX exposes various internal metrics of the Kafka brokers and other Java-based components. Metricbeat can poll these metrics and ship them to Elasticsearch.
  • Configuration: You’ll need to configure the JMX metricset in metricbeat.yml to connect to your Kafka broker’s JMX port (usually 9999). You’ll specify which JMX MBeans and attributes you want to collect.
  • Pros: Provides detailed performance and operational metrics.
  • Cons: Requires more configuration to specify the desired metrics.
• Logstash (Less Common for Direct Kafka Logs): While Filebeat is generally preferred for log shipping, you could use Logstash to ingest Kafka logs. However, Filebeat’s Kafka module simplifies this process significantly. Logstash is often used if you need more complex log processing or enrichment before sending data to Elasticsearch.

2. Data Processing (Logstash – Optional but Powerful):

• Functionality: Logstash can sit between Filebeat/Metricbeat and Elasticsearch to further process and enrich your Kafka data. This might involve:
  • More complex parsing: If Filebeat’s module doesn’t fully meet your needs.
  • Data enrichment: Adding geographical information based on IP addresses in logs, for example.
  • Filtering: Dropping irrelevant log events or metrics.
  • Transformation: Restructuring data for better analysis in Kibana.
• Configuration: You’ll define pipelines in Logstash that specify inputs (e.g., Beats from Filebeat/Metricbeat), filters (for processing), and outputs (Elasticsearch).

3. Data Storage (Elasticsearch):

• Functionality: Elasticsearch is the distributed search and analytics engine where your Kafka logs and metrics will be indexed and stored.
• Index Management: You’ll likely configure index patterns (e.g., filebeat-*, metricbeat-*) to organize your Kafka data based on the source and time.

4. Data Visualization and Analysis (Kibana):

• Functionality: Kibana is the powerful visualization layer of the ELK stack. You can create dashboards, charts, graphs, and tables to monitor various aspects of your Kafka environment.
• Pre-built Dashboards: The Filebeat Kafka module often provides pre-built Kibana dashboards that offer immediate insights into your Kafka logs (e.g., log levels, error rates, exceptions). Metricbeat also has some basic JMX dashboards, but you’ll likely need to create custom ones tailored to the specific Kafka metrics you’re collecting.
• Custom Visualizations: You can create your own visualizations in Kibana to monitor specific metrics like:
  • Broker CPU and Memory Usage: Track resource utilization on your Kafka nodes.
  • Under-Replicated Partitions: Identify potential data loss risks.
  • Consumer Lag: Monitor how far behind your consumers are in processing messages.
  • Producer Throughput: Track the rate at which data is being produced.
  • Request Latency: Monitor the time it takes for Kafka to handle requests.

Example Data Flow:

1. Kafka Brokers: Generate logs and expose JMX metrics.
2. Filebeat (on Kafka brokers or dedicated nodes): Collects Kafka logs.
3. Metricbeat (on Kafka brokers or dedicated nodes): Collects Kafka JMX metrics.
4. (Optional) Logstash: Processes and enriches the data from Filebeat and Metricbeat.
5. Elasticsearch: Indexes and stores the log and metric data.
6. Kibana: Provides dashboards and visualization tools to analyze the Kafka data.

Key Kafka Metrics to Visualize in Kibana:

• Log Analysis (from Filebeat):
  • Log level distribution (INFO, WARN, ERROR).
  • Error and exception counts over time.
  • Broker activity patterns.
  • Controller election events.
• Broker Metrics (from Metricbeat JMX):
  • kafka.server.BrokerState
  • kafka.server.ReplicaManager.UnderReplicatedPartitions
  • kafka.server.RequestChannel.RequestQueueSize
  • kafka.jvm.memory.heap.used
  • system.cpu.total.pct
  • system.memory.actual.used.pct
  • diskio.read.bytes and diskio.write.bytes
• Topic Metrics (from Metricbeat JMX):
  • kafka.server.BrokerTopicMetrics.BytesInPerSec
  • kafka.server.BrokerTopicMetrics.BytesOutPerSec
  • kafka.controller.ControllerStats.LeaderElectionRateAndTimeMs
• Consumer Group Metrics (from Metricbeat JMX):
  • kafka.consumer.FetcherLagMetrics.ConsumerLag (This often requires specific configuration to expose)
  • kafka.consumer.ConsumerCoordinatorMetrics.CommitOffsetLatencyAvg

In summary, monitoring Kafka with the ELK stack typically involves using Filebeat to collect and parse logs and Metricbeat to gather JMX-based metrics. Logstash can be used for more advanced processing, and Kibana provides the visualization and analysis capabilities to gain insights into your Kafka cluster’s health and performance.