Zookeeper is an important part of Apache Kafka. It tracks the status of Kafka nodes and provides a view of the current state. To make sure that we are able to manage your Kafka cluster in the best way possible, we do not allow direct access to Zookeeper. Instead, we monitor and manage it using the Aiven platform to make sure that we can react as quickly as possible if any issues occur.
Instead of directly accessing Zookeeper, you can set advanced configuration flags in the Aiven web console and use the API. Almost all of the flags the you would need to set can be found here. However, you may need a different option or to change something related to a more rare scenario.
Without Zookeeper, some of the command line tools that come with Kafka do not work. As Kafka has matured, the reliance on Zookeeper for some of these commands has been removed and you do have options available for handling configuration changes.
As an example scenario:
Requirement: To create a topic on a Kafka cluster that has a retention rate of 30 minutes.
Limitation: The Aiven web console and API support hours as the smallest unit for the topic retention rate.
Solution: Connect to your cluster using Kafka tools and set the retention rate.
The following steps show you how to resolve this use case for a new topic.
Note: It is currently not possible to set the configuration for an existing topic as that requires a connection to Zookeeper.
Follow the steps in this article to create a Java keystore and truststore for your Aiven for Apache Kafka service.
kafka.propertiesfile with the following contents:
ssl.truststore.password=<PASSWORD SET WHEN CREATING TRUSTSTORE>
ssl.keystore.password=<PASSWORD SET WHEN CREATING KEYSTORE>
ssl.key.password=<SAME AS KEYSTORE.PASSWORD>
Connect to your cluster.
Download Kafka from https://kafka.apache.org/downloads.
Unpack the downloaded archive.
The unpacked content includes a
binfolder that contains a large selection of
.shfiles. These files act as CLI wrappers for the more commonly used Kafka administrator commands.
Run the following command to check that you can connect to your cluster:
$ ./bin/kafka-topics.sh --bootstrap-server <YOUR SERVICE URI> --command-config kafka.properties --list
Run the following command to create a new topic with a retention rate of 30 minutes.
kafka-topicsscript for this and set the retention value in milliseconds (
(100 * 60) * 30 = 180000).
$ ./bin/kafka-topics.sh --bootstrap-server https://chrisg-cust-demo-kafka-business-demo.aivencloud.com:18789 --command-config ssl.properties --topic new-topic --create --config retention.ms=180000
kafka-topics.shcommand with the
--describeoption to check the details of your topic and the retention rate.