Topics can be automatically created in Kafka when a message is produced to topic that doesn't exist. By default the feature is disabled, but can be enabled using the Aiven command line client. Once the client is installed, log in:
avn user login # unless you have already logged in
Running the following command enables creating topics for a service named mykafka:
avn service update mykafka -c kafka.auto_create_topics_enable=true
Even though Kafka creates topics implicitly, the user that produces a message to a non-existing topic must have admin permissions. Kafka validates the ACL before it creates the topic on behalf of the user. To change permissions, go to the Users view of a Kafka service's view in the web console.
Changing default settings for topics
When Kafka auto-creates a topic, it uses default values defined in the service's configuration for partition count, replication factor, retention time etc. since the values are not explicitly provided like when the topic is created from the Aiven console or with the Kafka Admin API's CreateTopics request.
These values can be changed with the Aiven client:
avn service update mykafka -c kafka.default_replication_factor=3
avn service update mykafka -c kafka.num_partitions=4
avn service update mykafka -c kafka.log_retention_bytes=1000000
avn service update mykafka -c kafka.log_retention_hours=24
To see the current values, run:
avn service get mykafka -v
Notice that some of these settings may affect topics created with the Admin API CreateTopics request if the setting is not included in the request's topic config field. For example creating a topic with an empty config will cause the topic to use kafka.log_retention_hours and kafka.log_retention_bytes set by service update the same way as auto-created topics do. Topics created via the console or Aiven client are not affected by this behavior because they always specify these values when creating a topic.
Got here by accident? Learn how Aiven simplifies working with Apache Kafka: