Kafka logs integration enables to produce service logs into a kafka topic. The integration can be used for your aiven Kafka services but also for external Kafka clusters registered in the project's service integrations page.
As usual, the feature is better to be explained by example. For this example, you need aiven PostgreSQL service (source) and aiven Kafka service (destination) with Kafka REST enabled. You also want to create a topic for your service logs.
Prerequisites for this example:
- Aiven PostgreSQL service (source)
- Aiven Kafka service (destination)
- Kafka REST enabled for the Kafka service
- Topic created in the destination kafka service for the service logs
First, go to your PostgeSQL service overview page. From the service overview page, click 'Manage Integrations' to open the service integrations modal for managing and viewing the available integrations.
From the list of available service integrations, select 'Kafka Logs'.
Next, select the destination kafka service in which you would like to push the service logs into.
The last step is to provide the topic name into which you want to produce the service logs.
Next step is to verify that the integration setup works as expected. Go ahead and navigate to the destination kafka services topic message view.
In the topic messages view, switch to 'json' format and click 'fetch messages'. The service logs should now be consumed from the topic and presented in the view.
And that's it! The service logs from our postgreSQL service are now being produced into our Kafka service. If you want to remove or edit the target topic, head back to the source services 'manage integrations' modal. From there, you can manage your existing integrations.