Kafka is a distributed streaming platform that is technically implemented as a series of append-only logs that make up different topics as well as partitions within topics. When new data is written to Kafka, it is appended to the end of one of the logs.

Data retention period

To avoid running out of disk space, Kafka drops the oldest messages from the beginning of each log after their retention period expires by default. Aiven for Apache Kafka allows you to configure the retention period for each topic.

Aiven does not limit the maximum retention period in any way, and setting retention to a value of -1 disables time-based content expiration altogether (unlimited retention).

To change this for a single topic:

  1. Log in to the Aiven web console and select your service.

  2. Click the Topics tab and then click the topic that you want to modify.

  3. In the Topic info view, click Modify.

  4. If there is no field for Retention ms, select it from the drop-down menu.

  5. Change the Retention ms value to -1.

You can also change the retention at a broker level on the Overview page under Advanced configuration, by adding kafka.log_retention_hours and setting that to -1.

Note: Using high retention periods without monitoring the available storage space can cause your service to run out of disk space. These situations are not covered by our SLA.

As the data is not being backed up, the primary mechanism for keeping your Kafka data durable is the number of Kafka brokers that your topic is set to replicate to. Setting an unlimited retention period for a topic only makes sure that Kafka itself does not truncate older data.

Log compaction 

Instead of dropping all of the data that is in the beginning of a topic, you can also configure the topic to use Log compaction, which allows you to retain the latest value for each record key even if the retention period of the message has expired. This means that messages must always have an explicit key when using log compaction. You can enable log compaction for new topics in the Kafka topic management section of the Aiven web console.

Learn how Aiven simplifies working with Apache Kafka:

Did this answer your question?