What is Kafka MirrorMaker?
Kafka MirrorMaker is a tool that makes it possible to copy data across two Apache Kafka clusters. You can set up your Kafka mirror for example in a different cloud provider to ensure that your data is always available and distributed. It’s possible to mirror either selected topics or to replicate your entire Kafka instance. This article will cover both the Aiven internal MirrorMaker replication case and the external where the assumption is that you want to use MirrorMaker to migrate to Aiven.
Aiven Kafka MirrorMaker service integration
Now it’s possible to mirror Kafkas with the Aiven MirrorMaker service integration. Aiven’s unique cloud offering makes it easy for you to replicate data across cloud providers and regions.
The minimum requirement for mirroring is a Kafka Business-8 plan for both the source and the destination service. You can find the detailed pricing information for these services from aiven.io/kafka.
1. Navigate to your Kafka service and modify your service integrations
Service integrations can be managed by clicking the manage button in your service overview page. For more information about available Aiven service integrations, see our help article.
2. Select your Aiven Kafka service to integrate with
Then select Kafka MirrorMaker as the integration you want to enable.
At this point, you can either select an existing Kafka service for integration or create a new one.
3. Configure your MirrorMaker whitelist options
With MirrorMaker whitelisting you can select which topics you wish to replicate to your mirror. By default, every topic will be replicated and this is presented with the configuration value
.*. The whitelist option is a Regular expression that you can configure manually but we additionally offer a simpler UI to select the topics you wish to mirror.
After clicking enable, your Kafka MirrorMaker will start to operate and you will start to see data being replicated to the destination Kafka service.
Kafka MirrorMaker use from a cluster outside Aiven
If you're looking to transfer data to or from Aiven it is also possible for you to set up Kafka MirrorMaker by yourself.
The MirrorMaker consumer config would look something like:
auto.offset.reset = earliest
bootstrap.servers = my_kafka_url:12345
enable.auto.commit = false
And the MirrorMaker producer configuration should look something like: