Monitoring Apache Kafka infrastructure using New Relic

One can effectively monitor Apache Kafka infrastructure using New Relic through several methods:

1. Kafka On-Host Integration (Recommended for most self-managed Kafka deployments):

• How it works: This integration runs on the same host as your Kafka brokers and directly collects metrics and inventory data specific to Kafka. It leverages JMX (Java Management Extensions) to gather detailed metrics about brokers, topics, producers, and consumers. It also pulls inventory data from Zookeeper.¹
• Key Features:
  • Monitors key metrics for clusters, brokers, producers, consumers, and topics.
  • Automatic discovery of brokers using Bootstrap or Zookeeper.
  • Provides pre-built dashboards for Kafka overview.
  • Allows you to create custom dashboards and set up alerts.²
• Installation: Requires installing the New Relic Infrastructure agent on your Kafka hosts first.³ Then, you install and configure the Kafka integration. Configuration involves specifying your cluster name, Kafka version, broker discovery strategy (Zookeeper or Bootstrap), and JMX connection details.⁴
• Metrics Collected (Examples):
  • broker.bytesWrittenToTopicPerSecond
  • broker.IOInPerSecond / broker.IOOutPerSecond
  • broker.messagesInPerSecond
  • replication.isrExpandsPerSecond / replication.isrShrinksPerSecond
  • replication.unreplicatedPartitions
  • consumer.bytesInPerSecond
  • consumer.maxLag
• Documentation: https://docs.newrelic.com/install/kafka/

2. Java Agent (for monitoring Java-based Producers and Consumers):

• How it works: If your Kafka producers and consumers are written in Java, you can use the New Relic Java agent to automatically instrument them. This provides insights into the performance of your applications interacting with Kafka.
• Key Features:
  • Collects metrics about messaging rates, latency, and lag for Java-based clients.
  • Automatic instrumentation of Kafka’s Java client library.
  • Option to collect event data for more flexible querying with NRQL.
  • Supports distributed tracing to track messages across your application and Kafka.
• Installation: Requires installing the New Relic Java agent in your Java application. You might need to enable specific Kafka-related configurations in the newrelic.yml file.
• Metrics Collected (Examples):
  • MessageBroker/Kafka/Internal/consumer-metrics/records-consumed-rate
  • MessageBroker/Kafka/Internal/producer-metrics/record-send-total
  • Kafka/Streams/stream-thread-metrics/poll-latency-avg (for Kafka Streams)
• Documentation: https://docs.newrelic.com/docs/apm/agents/java-agent/instrumentation/java-agent-instrument-kafka-message-queues/

3. OpenTelemetry (for a vendor-agnostic approach):

• How it works: OpenTelemetry is an open-source observability framework that provides a standardized way to collect telemetry data (metrics, traces, and logs).⁵ You can use OpenTelemetry to instrument your Kafka brokers and client applications and then export the data to New Relic.
• Key Features:
  • Vendor-agnostic instrumentation.
  • Flexibility in collecting various telemetry data.
  • Integration with the New Relic OTLP (OpenTelemetry Protocol) endpoint.
  • Allows for correlating infrastructure and application performance data.
• Installation: Involves setting up OpenTelemetry SDKs and Collectors for your Kafka components and configuring the OTLP exporter to send data to your New Relic account.
• Documentation: https://docs.newrelic.com/docs/opentelemetry/ and https://github.com/newrelic/newrelic-opentelemetry-examples have examples for various technologies.

4. Kafka Connect New Relic Connector (for sending data from Kafka Connect to New Relic):

• How it works: If you are using Kafka Connect, you can use the New Relic Kafka Connect Sink Connector to stream data from Kafka topics directly into New Relic as events.⁶
• Key Features:
  • Directly ingest data from Kafka topics into New Relic.
  • Allows you to analyze Kafka data within the New Relic platform.
• Installation: Requires configuring and deploying the New Relic Kafka Connect Sink Connector in your Kafka Connect environment.
• Documentation: Search for “New Relic Kafka Connect Sink Connector” in the New Relic documentation or on platforms like Confluent Hub.

Choosing the Right Method:

• For comprehensive monitoring of your Kafka brokers and overall cluster health, the Kafka On-Host Integration is generally the recommended approach.
• If you need detailed performance insights into your Java-based producers and consumers, use the New Relic Java Agent.
• If you prefer a vendor-agnostic approach or have a complex environment with various technologies, OpenTelemetry offers flexibility.
• If you want to directly ingest data from Kafka Connect pipelines into New Relic, use the Kafka Connect New Relic Connector.

You can even use a combination of these methods for a holistic view of your Kafka ecosystem within New Relic. Remember to consult the official New Relic documentation for the most up-to-date instructions and configuration details.