Aiven supports integrating logs with a number of external monitoring systems that support rsyslog protocol, including Fluentd.

To integrate your service with Fluentd, a new endpoint needs to be added into the project that contains the service you want to integrate. This can be done using through Aiven console or command line using Aiven Client.

Before adding an endpoint, we first need to configure your log Fluentd aggregator server that is running td-agent. Begin by appending the following example configuration to /etc/td-agent/td-agent.conf 

<source>
  @type syslog
  port 41285
  bind 0.0.0.0
  <transport tcp>
  </transport>
  tag rsyslog
</source>

<match rsyslog.**>
  @type stdout
</match>


Make a note of the port in the above configuration, which will later be used when adding service integration.

This will forward syslog directly to /var/log/td-agent/td-agent.log file, but you can also configure your logs to flow to additional consumers such as ElasticSearch.
For a detailed description and additional configuration options, please see Fluentd syslog article.

Make sure to restart td-agent afterwards : sudo systemctl restart td-agent  and then verify that it has successfully started sudo systemctl status td-agent  

Add Fluentd endpoint using Aiven Console

Navigate to Aiven console, open your project and select Service Integrations on the left hand side and navigate to Syslog configuration.

When defining new endpoint, the following parameters can be applied

  • name - Create a name for this endpoint (e.g. Fluentd)
  • server - IPv4 or DNS name of the aggregator server
  • port - 41285 (or another port of your choice)
  • tls - disable (see below how to use TLS with avn-client)
  • format - set to rfc3164

Next, open your service in Aiven console and under Service Integrations option, click Manage Integrations which will bring up a list of available integrations for your service. Select Rsyslog from the provided list and click Use integration.

Finally, select an endpoint that you created in previous step and click Enable to enable service integration.


After enabling service integration, you can see that it has been activated in the Aiven console and the logs will be now integrated with Fluentd. It may take a few moments to setup the new log, and you can track the status in your service overview.

Your logs should now be visible in your server (or sent to another endpoint, depending on your setup). Run tail /var/log/td-agent/td-agent.log to verify that logs are now showing up on your server.

Add Fluentd endpoint using Aiven client

From the Aiven client, enter the following command, replacing values as specified below. 

avn service integration-endpoint-create --project your-project \
    -d Fluentd -t rsyslog \
    -c server=your-server-ip -c port=41285 \
    -c format=rfc3164 -c tls=false'

When defining new endpoint, the following parameters can be applied

  • name - Create a name for this endpoint (e.g. Fluentd)
  • server - DNS or IPv4 of your server
  • port - 41285 (or another port of your choice)
  • tls - set false
  • format - set to rfc3164

Add rsyslog integration to service

First you need the id of the endpoint previously created 

avn service integration-endpoint-list --project your-project
ENDPOINT_ID                           ENDPOINT_NAME   ENDPOINT_TYPE
====================================  ==============  =============
618fb764-5832-4636-ba26-0d9857222cfd  example-syslog  rsyslog

Using the ENDPOINT_ID from above and you can now link the service to the endpoint

avn service integration-create --project your-project \
    -t rsyslog -s your-service \
    -D 618fb764-5832-4636-ba26-0d9857222cfd

After enabling service integration, you can see that it has been activated in the Aiven console and the logs will be now integrated with Fluentd. It may take a few moments to setup the new log, and you can track the status in your service overview.

Your logs should now be visible in your server (or sent to another endpoint, depending on your setup). Run tail /var/log/td-agent/td-agent.log to verify that logs are now showing up on your server.

Additional articles may be found on our Aiven Support page.

If you have any questions with integration, please feel free to reach out to Support and let us know.

Did this answer your question?