You can set Kafka to automatically create a topic when a message is produced to a topic that does not exist. By default, this feature is turned off, but you can turn it on using the Aiven command-line client.
Note: Even though Kafka creates topics implicitly, the user account that produces a message to a non-existing topic must have
admin permissions. Kafka validates the access control list (ACL) before it creates the topic. To change permissions, go to the Users page for your service in the Aiven web console.
When a topic is created automatically, it uses the default values defined in the service configuration for partition count, replication factor, retention time, and other properties. Unlike when you create a topic in the Aiven web console or with the Kafka Admin API
CreateTopics request, these values are not explicitly provided in automatically created topics.
Once you have installed the client:
Log in with your service credentials:
avn user login # unless you have already logged in
Run the following command to create new topics automatically when necessary:
avn service update SERVICE_NAME -c kafka.auto_create_topics_enable=true
To change the default settings for topics, run the following commands:
avn service update SERVICE_NAME -c kafka.default_replication_factor=3
avn service update SERVICE_NAME -c kafka.num_partitions=4
avn service update SERVICE_NAME -c kafka.log_retention_bytes=1000000
avn service update SERVICE_NAME -c kafka.log_retention_hours=24
Tip: To see the current default values, run the following command:
avn service get SERVICE_NAME -v
Note: Some of these settings may affect topics created with the Admin API
CreateTopicsrequest if the setting is not included in the topic configuration field of the request. For example, if you create a topic with an empty configuration, the topic uses the default
kafka.log_retention_bytesvalues for the service in the same way as automatically created topics. This does not affect topics that created in the web console or Aiven command-line client, because they always specify these values when you create a topic.
You can also change the settings given in these steps in the Aiven web console by selecting the corresponding settings in the Advanced configuration section of the Overview page for your service.
Learn how Aiven simplifies working with Apache Kafka: