First of all you need to confirm that you have a Datadog endpoint configured and enabled on the Kafka instance, instructions on how to do that can be found from the Getting started with Datadog article.
After installing the Aiven command-line client and having the integration enabled, you can list the enabled integrations with
avn service integration-list <myservice>. You should have one integration with the integration type
datadog, you need the service integration ID of that line.
Sending custom Kafka metrics
For example to enable sending the
kafka.log.log_size metric to Datadog, you can use
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size <integration-id>. You can also send a combination of custom metrics by separating them with a comma:
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset <integration-id>
The following custom metrics are currently supported:
Finally, after the update goes through and Kafka has started collecting some logs you should be able to see the data on your Datadog explorer:
Each of these metrics are also tagged with
partition to be able to monitor each Kafka topic and partition independent of each other.
Filtering metrics sent by topics and/or consumer groups
You can choose which topics and/or consumer groups metrics to send to DataDog using the following configuration options:
For example, to get metrics from only 2 topics,
topic2, we can do:
avn service integration-update -c include_topics=topic1,topic2 <integration-id>
Got here by accident? Learn how Aiven simplifies working with Apache Kafka: