To prevent Kafka from malfunctioning, the Aiven platform detects low available disk space in Kafka services.
If any of the service's nodes is above the critical threshold of disk usage, the Access Control List (ACL) used to authorize API requests by Kafka clients is updated on all nodes to prevent operations that would further increase disk usage. These include
- The Write and IdempotentWrite operations used to produce new messages by clients
- The CreateTopics operation that creates one or more topics, each carrying some overhead on disk
Clients encountering this blocking of writes by ACL should encounter an error like
TopicAuthorizationFailedError: [Error 29] TopicAuthorizationFailedError: mytopic (Example from using the python-kafka client).
Actions to make more space available
Upgrade to a larger plan
This can be done from the Aiven console or with Aiven CLI client. New nodes with more disk space are launched. As data from the old nodes are moved to new ones, disk usage will return to below the critical level and the ACL is updated to allow writes again.
Delete one or more topics
Another way to get more available disk space is to get rid of old data. The simplest way to achieve this is to delete an entire topic. This can be done from the Aiven console or with the Aiven CLI client. It's also possible to use the DeleteTopics API from any native Kafka client. Notice that in this case the client must be connecting as an admin level user to have the necessary permissions. It may take up to a few minutes before Kafka's Log Cleaner starts removing the data files on disk, after which the ACL is updated to allow writes again.
Decrease retention time/size
A third way to make more space available without deleting an entire topic is to reduce the retention time or size for one or more topics. If you are aware how old the oldest messages are in a topic, more space can be made available by lowering the retention time for the topic. This can be done from the Topics tab in the Aiven Console by changing the "Retention (hours)" setting for a particular topic. Notice that just changing the value smaller will not make more space available, unless there are messages in the topic that are older than the given value.
To reduce the size retention configuration for the topic, start by checking how much space the topic currently uses. This can be seen by clicking Info in the Aiven Console's Topics tab for a given topic. Use a Kafka client with admin permissions to alter the "retention.bytes" topic config to a value smaller than the current size.
Kafka's log cleaner should start removing data on disk within a few minutes, and the ACL allows writes again if enough space was made available.