The instructions in this article show you how to customize the Kafka Datadog integration settings using the Aiven command-line client. You can also do all the same operations over the API with theĀ user_config
options.
The following metrics are currently supported:
kafka.log.log_size
kafka.log.log_start_offset
kafka.log.log_end_offset
Each of these metrics are also tagged with topic
and partition
so that you can monitor each Kafka topic and partition independently.
To send customized metrics to Datadog:
Make sure that you have a Datadog endpoint configured and enabled in your Aiven for Apache Kafka service.
For details on how to set up Datadog integration, see this article.Run the following command in the Aiven command-line client to list the enabled integrations:
avn service integration-list SERVICE_NAME
The output should show you that you have one integration with
datadog
as the type. Make note of the service integration ID for that line.Run the
service integration-update
command to customize the metrics that are sent to Datadog.
For example, to send thekafka.log.log_size
metric:avn service integration-update -c kafka_custom_metrics=kafka.log.log_size INTEGRATION_ID
To send a combination of custom metrics, separate them with a comma. For example:
avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset INTEGRATION_ID
When the update is successful and Kafka has started collecting logs, you can see the data in your Datadog explorer:
You can choose the topic and consumer group metrics that you want to send to Datadog using the following configuration options:
include_topics
exclude_topics
include_consumer_groups
exclude_consumer_groups
For example, to send metrics from only topic1
and topic2
, run the following command:
avn service integration-update -c include_topics=topic1,topic2 INTEGRATION_ID
Learn how Aiven simplifies working with Apache Kafka: