To customize the Kafka Datadog integration settings from defaults, the Aiven command-line client is required. All the same operations can also be done over the API with user_config options.
First of all you need to confirm that you have a Datadog endpoint configured and enabled on the Kafka instance, instructions on how to do that can be found from the Getting started with Datadog article.
After installing the Aiven command-line client and having the integration enabled, you can list the enabled integrations with avn service integration-list <myservice>. You should have one integration with the integration type datadog, you need the service integration ID of that line.
Sending custom Kafka metrics
For example to enable sending the kafka.log.log_size metric to Datadog, you can use avn service integration-update -c kafka_custom_metrics=kafka.log.log_size <integration-id>. You can also send a combination of custom metrics by separating them with a comma:
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset <integration-id>The following custom metrics are currently supported:
- kafka.log.log_size
- kafka.log.log_start_offset
- kafka.log.log_end_offset
Finally, after the update goes through and Kafka has started collecting some logs you should be able to see the data on your Datadog explorer:
Each of these metrics are also tagged with topic and partition to be able to monitor each Kafka topic and partition independent of each other.
Filtering metrics sent by topics and/or consumer groups
You can choose which topics and/or consumer groups metrics to send to DataDog using the following configuration options: include_topics, exclude_topics, include_consumer_groups, exclude_consumer_groups.
For example, to get metrics from only 2 topics, topic1 and topic2, we can do:
avn service integration-update -c include_topics=topic1,topic2 <integration-id>Got here by accident? Learn how Aiven simplifies working with Apache Kafka: