To customize the Kafka Datadog integration settings from defaults, the Aiven command-line client is required. All the same operations can also be done over the API with  user_config options.

First of all you need to confirm that you have a Datadog endpoint configured and enabled on the Kafka instance, instructions on how to do that can be found from the Getting started with Datadog article.

After installing the Aiven command-line client and having the integration enabled, you can list the enabled integrations with avn service integration-list <myservice>. You should have one integration with the integration type datadog, you need the service integration ID of that line. 

For example to enable sending the kafka.log.log_size  metric to Datadog, you can use  avn service integration-update -c kafka_custom_metrics=kafka.log.log_size <integration-id>. You can also send a combination of custom metrics by separating them with a comma:

avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset <integration-id>

The following custom metrics are currently supported:

  • kafka.log.log_size
  • kafka.log.log_start_offset
  • kafka.log.log_end_offset 

Finally, after the update goes through and Kafka has started collecting some logs you should be able to see the data on your Datadog explorer:

Each of these metrics are also tagged with topic and partition to be able to monitor each Kafka topic and partition independent of each other.

Did this answer your question?