Zookeeper is an important part of Apache Kafka. It tracks the status of Kafka nodes and provides a view of the current state. To make sure that we are able to manage your Kafka cluster in the best way possible, we do not allow direct access to Zookeeper. Instead, we monitor and manage it using the Aiven platform to make sure that we can react as quickly as possible if any issues occur.

Instead of directly accessing Zookeeper, you can set advanced configuration flags in the Aiven web console and use the API. Almost all of the flags the you would need to set can be found here. However, you may need a different option or to change something related to a more rare scenario.

Without Zookeeper, some of the command line tools that come with Kafka do not work. As Kafka has matured, the reliance on Zookeeper for some of these commands has been removed and you do have options available for handling configuration changes.

As an example scenario:

Requirement: To create a topic on a Kafka cluster that has a retention rate of 30 minutes.

Limitation: The Aiven web console and API support hours as the smallest unit for the topic retention rate.

Solution: Connect to your cluster using Kafka tools and set the retention rate.

The following steps show you how to resolve this use case for a new topic.
Note: It is currently not possible to set the configuration for an existing topic as that requires a connection to Zookeeper.

  1. Follow the steps in this article to create a Java keystore and truststore for your Aiven for Apache Kafka service.

  2. Create a kafka.properties file with the following contents:

    security.protocol=SSL
    ssl.truststore.location=/fullPath/client.truststore.jks
    ssl.truststore.password=<PASSWORD SET WHEN CREATING TRUSTSTORE>
    ssl.keystore.location=/fullPath/client.keystore.jks
    ssl.keystore.password=<PASSWORD SET WHEN CREATING KEYSTORE>
    ssl.key.password=<SAME AS KEYSTORE.PASSWORD>

  3. Connect to your cluster.

    1. Download Kafka from https://kafka.apache.org/downloads.

    2. Unpack the downloaded archive.
      The unpacked content includes a bin folder that contains a large selection of .sh files. These files act as CLI wrappers for the more commonly used Kafka administrator commands.

    3. Run the following command to check that you can connect to your cluster:

      $ ./bin/kafka-topics.sh --bootstrap-server <YOUR SERVICE URI> --command-config kafka.properties --list

  4. Run the following command to create a new topic with a retention rate of 30 minutes.
    Use the kafka-topics script for this and set the retention value in milliseconds ((100 * 60) * 30 = 180000).

    $ ./bin/kafka-topics.sh  --bootstrap-server https://chrisg-cust-demo-kafka-business-demo.aivencloud.com:18789  --command-config ssl.properties --topic new-topic --create --config retention.ms=180000

  5. Run the kafka-topics.sh command with the --describe option to check the details of your topic and the retention rate.


Did this answer your question?