Configuring Datadog metrics with Aiven for Apache Kafka

The instructions in this article show you how to customize the Kafka Datadog integration settings using the Aiven command-line client. You can also do all the same operations over the API with the user_config options.

The following metrics are currently supported:

kafka.log.log_size
kafka.log.log_start_offset
kafka.log.log_end_offset

Each of these metrics are also tagged with topic and partition so that you can monitor each Kafka topic and partition independently.

To send customized metrics to Datadog:

Make sure that you have a Datadog endpoint configured and enabled in your Aiven for Apache Kafka service.
For details on how to set up Datadog integration, see this article.
Run the following command in the Aiven command-line client to list the enabled integrations:
```
avn service integration-list SERVICE_NAME
```
The output should show you that you have one integration with datadog as the type. Make note of the service integration ID for that line.
Run the service integration-update command to customize the metrics that are sent to Datadog.
For example, to send the kafka.log.log_size metric:
```
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size INTEGRATION_ID
```
To send a combination of custom metrics, separate them with a comma. For example:
```
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset INTEGRATION_ID
```
When the update is successful and Kafka has started collecting logs, you can see the data in your Datadog explorer:

You can choose the topic and consumer group metrics that you want to send to Datadog using the following configuration options:

include_topics
exclude_topics
include_consumer_groups
exclude_consumer_groups

For example, to send metrics from only topic1 and topic2, run the following command:

avn service integration-update -c include_topics=topic1,topic2 INTEGRATION_ID

Learn how Aiven simplifies working with Apache Kafka:

Managed Kafka as a Service