You asked (multiple times), we listened! Our Apache Kafka experts and Open Source Team have submitted a Pull Request to Apache Kafka in order to allow Mirrormaker 2 to replicate topic data between clusters without adding a prefix. If you have used Mirrormaker 2 then you are surely familiar with this scenario:

You want to replicate topic orders from source cluster eu-kafka-prod to target cluster eu-kafka-dev using Mirrormaker 2. You configure your Replication Flow and you soon see that Mirrormaker 2 has created a few topics to track the replication and the topic you want to see appears on your target cluster, except something is different, you see:

eu-kafka-prod.orders

For most use cases, this is fine. However, you might be using a backup Kafka cluster for Disaster Recovery. In this case, you want your consumers and producers to be able to switch with minimal downtime; so you do not want to modify the topic names in their configuration.

This functionality will be available in an upcoming Kafka release but we have gone ahead and implemented it on Aiven for Apache Kafka because of the high demand.

To remove the topic prefix, you will need to change something called the replication_policy_class and, while support in the Aiven Console is incoming, it is currently available in the Aiven API, let’s go through how it can be done:

Using the API

curl --request POST \
--url https://api.aiven.io/v1/project/%7Bproject%7D/service/%7Bservice_name%7D/mirrormaker/replication-flows \
--header 'Authorization: Bearer REPLACE_BEARER_TOKEN' \
--header 'content-type: application/json' \
--data '{"emit_heartbeats_enabled":true,"enabled":true,"replication_policy_class":"org.apache.kafka.connect.mirror.IdentityReplicationPolicy","source_cluster":"eu-kafka-prod","sync_group_offsets_enabled":true,"sync_group_offsets_interval_seconds":1,"target_cluster":"eu-kafka-dev","topics":["orders"]}'

Using the Aiven CLI

avn mirrormaker replication-flow create -s eu-kafka-prod -t eu-kafka-dev eu-mm2 "{\"replication_policy_class\": \"org.apache.kafka.connect.mirror.IdentityReplicationPolicy\"}"

Support for this in the Aiven Console is in development but, if you simply cannot wait for replicating without that pesky prefix then this is the way to go.

Want to revert or modify the policy? The default (with prefix) is conveniently known as: org.apache.kafka.connect.mirror.DefaultReplicationPolicy. Using the same method as above you can modify your existing Replication Flows.

NOTE: This replication policy will NOT support active-active replication as the topics will keep the same name and offsets can not be accurately tracked. Creating the same replication flows between a source and a destination WILL create an infinite loop. For active-active, please use the org.apache.kafka.connect.mirror.DefaultReplicationPolicy.

Issues? Questions? You can always get in touch with us: support@aiven.io

Did this answer your question?