In addition to using Aiven Elasticsearch to store the logs from your Aiven services, you can now integrate with an external monitoring system that supports the rsyslog protocol.

Creating rsyslog integration 

Add rsyslog integration endpoint

As the first step, you need to add the remote syslog you want to send the log to into the project that contains the service you want to integrate. This can be done using the Aiven Client

avn service integration-endpoint-create --project your-project \
    -d example-syslog -t rsyslog \
    -c server=logs.example.com -c port=514 \
    -c format=rfc5424 -c tls=true

When defining the remote syslog server the following parameters can be applied

required:

  • server -  DNS name or IPv4 address of the server
  • port - port to connect to
  • format - message format used by the server, this can be either rfc3164 (the old BSD style message format), rfc5424 (current syslog message format) or custom
  • tls - use TLS (as the messages are not filtered and may contain sensitive information, it is highly recommended to set this to true if the remote server supports it)

conditional (required if format == custom):

  • logline - syslog log line template for a custom format, supporting limited rsyslog style templating (using %tag%). Supported tags are: pri, procotol-version, timestamp, timestamp:::date-rfc3339, HOSTNAME, app-name, procid, msgid, msg and structured-data

optional:

  • sd - content of the structured data block of rfc5424 message
  • ca - (PEM format) Certificate Authority to use for verifying the servers certificate (typically not needed unless the server's certificate is issued by an internal CA or it uses a self-signed certificate)
  • key - (PEM format) client key if the server requires client authentication
  • cert - (PEM format) client cert to use  

Add rsyslog integration to service

First you need the id of the endpoint previously created 

avn service integration-endpoint-list --project your-project
ENDPOINT_ID                           ENDPOINT_NAME   ENDPOINT_TYPE
====================================  ==============  =============
618fb764-5832-4636-ba26-0d9857222cfd  example-syslog  rsyslog

Finally you can link the service to the endpoint

avn service integration-create --project your-project \
    -t rsyslog -s your-service \
    -D 618fb764-5832-4636-ba26-0d9857222cfd

Integrating with a third party syslog service 

papertrail

As papertrail identifies the client based on the server and port  you only need to copy the appropriate values from the "Log Destinations" page and use those as the values for server and port respectively. You do not need the ca-bundle as papertrail's servers use certificates signed by know CAs. You also need to set the format to rfc3164.

avn service integration-endpoint-create --project your-project \
    -d papertrail -t rsyslog \
    -c server=logsN.papertrailapp.com -c port=XXXXX \
    -c format=rfc3164 -c tls=true

loggly

In addition to the server and port you also need a customer token which you then need to give as part of the sd parameter when creating the endpoint.

avn service integration-endpoint-create --project your-project \
    -d loggly -t rsyslog \
    -c server=logs-01.loggly.com -c port=514 \
    -c format=rfc5424 -c tls=true \
    -c sd='TOKEN@NNNNN TAG="tag-of-your-choice"'

datadog

For datadog integration you need to use custom format with logline

avn service integration-endpoint-create --project your-project \ 
    -d datadog -t rsyslog \
    -c server=intake.logs.datadoghq.com -c port=10516 \
    -c tls=true -c format=custom \
    -c logline='<DATADOG_API_KEY> <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%'

(note, the logline value is one line)

Did this answer your question?