What is Kafka MirrorMaker?

Kafka MirrorMaker is a tool that makes it possible to copy data across two Apache Kafka clusters. You can set up your Kafka mirror for example in a different cloud provider to ensure that your data is always available and distributed. It’s possible to mirror either selected topics or to replicate your entire Kafka instance. This article will cover both the Aiven internal MirrorMaker replication case and the external where the assumption is that you want to use MirrorMaker to migrate to Aiven.    

Aiven Kafka MirrorMaker service integration

Now it’s possible to mirror Kafkas with the Aiven MirrorMaker service integration. Aiven’s unique cloud offering makes it easy for you to replicate data across cloud providers and regions.

The minimum requirement for mirroring is a Kafka Business-8 plan for both the source and the destination service. You can find the detailed pricing information for these services from aiven.io/kafka.

1. Navigate to your Kafka service and modify your service integrations

Service integrations can be managed by clicking the manage button in your service overview page. For more information about available Aiven service integrations, see our help article.

2. Select your Aiven Kafka service to integrate with

Then select Kafka MirrorMaker as the integration you want to enable.

At this point, you can either select an existing Kafka service for integration or create a new one.

3. Configure your MirrorMaker whitelist options

With MirrorMaker whitelisting you can select which topics you wish to replicate to your mirror. By default, every topic will be replicated and this is presented with the configuration value .*. The whitelist option is a Regular expression that you can configure manually but we additionally offer a simpler UI to select the topics you wish to mirror.

After clicking enable, your Kafka MirrorMaker will start to operate and you will start to see data being replicated to the destination Kafka service.

Kafka MirrorMaker use from a cluster outside Aiven 

If you're looking to transfer data to or from Aiven it is also possible for you to set up Kafka MirrorMaker by yourself. 

The MirrorMaker consumer config would look something like: 

auto.offset.reset = earliest
bootstrap.servers = my_kafka_url:12345
enable.auto.commit = false
group.id=source_mirrormaker_from_myownkafka
exclude.internal.topics=true
client.id=source_mirrormaker_from_myownkafka
security.protocol=SSL
ssl.protocol=TLS
ssl.key.password=changeit
ssl.keystore.location=/path/to/myownkafka_keystore.p12
ssl.keystore.password=changeit
ssl.keystore.type=PKCS12
ssl.truststore.location=/path/to/myownkafka_truststore.jks
ssl.truststore.password=changeit
ssl.truststore.type=JKS

And the MirrorMaker producer configuration should look something like:

acks=all
client.id=dest_mirrormaker_to_aiven
bootstrap.servers=aiven_kafka_url:12345
group.id=dest_mirrormaker_to_aiven
max.in.flight.requests.per.connection=1
retries=1000000
security.protocol=SSL
ssl.protocol=TLS
ssl.key.password=changeit
ssl.keystore.location=/path/to/aiven.service.keystore.p12
ssl.keystore.password=changeit
ssl.keystore.type=PKCS12
ssl.truststore.location=/path/to/aiven.service.truststore.jks
ssl.truststore.password=changeit
ssl.truststore.type=JKS

 

Did this answer your question?