Zookeeper is an important part of Apache Kafka, running to track the status of Kafka nodes and provide a view of the current state. In order to ensure we are able to manage your Kafka cluster in the best way possible, we do not allow direct access to Zookeeper and, instead, we monitor and manage it using the Aiven platform to make sure we can react as quickly as possible if any issues were to occur.

Instead of giving access to Zookeeper, we allow you to set Advanced Configuration flags in the Aiven Web Console and using the API. Almost all of the flags you would need to set can be found here. However, you may find that you need a different option or to change something that is a bit more of an edge case scenario.

Without Zookeeper some of the command line tools that come with Kafka will not run. As Kafka has matured, the reliance on Zookeeper for some of these commands has been removed and you do have a reasonable set of options available for you to configure. Let's take a look at a recent real-life scenario that a customer required assistance with:

 =============== 

Requirement:

You want to create a topic on your Kafka cluster that has a retention rate of 30 minutes.

Limitiation

The Aiven Web Console and API supports hours as the smallest unit for topic retention rate.

Solution:

Connect to your cluster using Kafka tools and set the retention rate yourself.

=================

In this help article, we are going to go through the use case of setting the retention rate for a new topic to 30 minutes (note: it is currently not possible to set the configuration for an existing topic as it does require a connection to Zookeeper).

1. Once you have logged into your Aiven Console, navigate to the Kafka service you want to configure. From the Overview tab, you will see Connection Information and you will need to download the following files:

    1. Access Key

    2. Access Certificate

    3. CA Certificate

    

2. Some command line knowledge is required here, to create a keystore and a truststore for Kafka to use when it connects. Open up a terminal and navigate to where you saved your certificates.

3. First, we need to sign your Access Certificate and create a keystore. Make a note of the passwords you use here!

$ openssl pkcs12 -export -in <YOUR ACCESS CERT> -inkey <YOUR ACCESS KEY> > host.p12

$ keytool -importkeystore -srckeystore host.p12 -destkeystore client.keystore.jks

4. Now, import your CA certificate into a truststore:

$ keytool -keystore client.truststore.jks -alias CARoot -import -file CA.pem
#When prompted, enter 'yes' to trust the certificate

  

5. Create a kafka.properties file with the following contents:

security.protocol=SSL
ssl.truststore.location=/fullPath/client.truststore.jks
ssl.truststore.password=<PASSWORD SET WHEN CREATING TRUSTSTORE>
ssl.keystore.location=/fullPath/client.keystore.jks
ssl.keystore.password=<PASSWORD SET WHEN CREATING KEYSTORE>
ssl.key.password=<SAME AS KEYSTORE.PASSWORD>

6. Connecting to your cluster. You can download Kafka from https://kafka.apache.org/downloads and untar it anywhere. Inside is a bin folder with a large selection of sh files that act as a CLI wrappers for more common Kafka admin commands. First we want to know if we can connect to our cluster ok:

$ ./bin/kafka-topics.sh --bootstrap-server <YOUR SERVICE URI> --command-config kafka.properties --list

7. Setting retention topic to 30 minutes. 30 minutes in milliseconds is (100 * 60) * 30 = 180000. NOTE: We use the kafka-topics script here and it is NOT possible to change the retention for an existing topic, only for setting it when creating new ones:

$ ./bin/kafka-topics.sh  --bootstrap-server https://chrisg-cust-demo-kafka-business-demo.aivencloud.com:18789  --command-config ssl.properties --topic new-topic --create --config retention.ms=180000

8. Done. Now you can run the --describe  command to check the details of your topic and the retention rate.


Did this answer your question?