The instructions in this article show you how to customize the Kafka Datadog integration settings using the Aiven command-line client. You can also do all the same operations over the API with theĀ user_config options.

The following metrics are currently supported:

  • kafka.log.log_size

  • kafka.log.log_start_offset

  • kafka.log.log_end_offset

Each of these metrics are also tagged with topic and partition so that you can monitor each Kafka topic and partition independently.

To send customized metrics to Datadog:

  1. Make sure that you have a Datadog endpoint configured and enabled in your Aiven for Apache Kafka service.
    For details on how to set up Datadog integration, see this article.

  2. Run the following command in the Aiven command-line client to list the enabled integrations:

    avn service integration-list SERVICE_NAME

    The output should show you that you have one integration with datadog as the type. Make note of the service integration ID for that line.

  3. Run the service integration-update command to customize the metrics that are sent to Datadog.
    For example, to send the kafka.log.log_size metric:

    avn service integration-update -c kafka_custom_metrics=kafka.log.log_size INTEGRATION_ID

    To send a combination of custom metrics, separate them with a comma. For example:

    avn service integration-update -c kafka_custom_metrics=kafka.log.log_size,kafka.log.log_end_offset INTEGRATION_ID

    When the update is successful and Kafka has started collecting logs, you can see the data in your Datadog explorer:

You can choose the topic and consumer group metrics that you want to send to Datadog using the following configuration options:

  • include_topics

  • exclude_topics

  • include_consumer_groups

  • exclude_consumer_groups

For example, to send metrics from only topic1 and topic2, run the following command:

avn service integration-update -c include_topics=topic1,topic2 INTEGRATION_ID

Learn how Aiven simplifies working with Apache Kafka:

Did this answer your question?