You can set Kafka to automatically create a topic when a message is produced to a topic that does not exist. By default, this feature is turned off, but you can turn it on using the Aiven command-line client.

Note: Even though Kafka creates topics implicitly, the user account that produces a message to a non-existing topic must have admin permissions. Kafka validates the access control list (ACL) before it creates the topic. To change permissions, go to the Users page for your service in the Aiven web console.

When a topic is created automatically, it uses the default values defined in the service configuration for partition count, replication factor, retention time, and other properties. Unlike when you create a topic in the Aiven web console or with the Kafka Admin API CreateTopics request, these values are not explicitly provided in automatically created topics.

Once you have installed the client:

  1. Log in with your service credentials:

    avn user login  # unless you have already logged in

  2. Run the following command to create new topics automatically when necessary:

    avn service update SERVICE_NAME -c kafka.auto_create_topics_enable=true

  3. To change the default settings for topics, run the following commands:

    avn service update SERVICE_NAME -c kafka.default_replication_factor=3
    avn service update SERVICE_NAME -c kafka.num_partitions=4
    avn service update SERVICE_NAME -c kafka.log_retention_bytes=1000000
    avn service update SERVICE_NAME -c kafka.log_retention_hours=24

    Tip: To see the current default values, run the following command:

    avn service get SERVICE_NAME -v

    Note: Some of these settings may affect topics created with the Admin API CreateTopics request if the setting is not included in the topic configuration field of the request. For example, if you create a topic with an empty configuration, the topic uses the default kafka.log_retention_hours and kafka.log_retention_bytes values for the service in the same way as automatically created topics. This does not affect topics that created in the web console or Aiven command-line client, because they always specify these values when you create a topic.

You can also change the settings given in these steps in the Aiven web console by selecting the corresponding settings in the Advanced configuration section of the Overview page for your service.

Learn how Aiven simplifies working with Apache Kafka:

Did this answer your question?