It is possible to send the logs of your Aiven services to Amazon AWS CloudWatch for storing, viewing and archiving them outside the Aiven platform.

Creating the AWS CloudWatch Logs integration

You need to start by creating a AWS CloudWatch Logs integration endpoint to the Aiven project you're using. It is possible to do this either inside the Aiven Web Console or by using the Aiven Client.

Create the integration endpoint using Aiven Web Console

You can enable the AWS CloudWatch Logs integration in the Service Integrations section of your project. You will need to give a name of the log group used in AWS CloudWatch as well as AWS credentials with appropriate access rights and the AWS region to be used. The access rights required for the credentials are logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents and logs:DescribeLogStreams. If a log group with given name does not exist in CloudWatch, it will be generated. Properties of the log group, like data retention, can be modified in the AWS console.

Add the integration endpoint to your service using Aiven Web Console

Go to the service you want to add the logs integration to and select Manage Integrations. Then find the AWS CloudWatch Logs integration from the list and click Use Integration. After this you will be asked to select the endpoint you want to use for the integration.

Create the integration endpoint using Aiven Client

avn service integration-endpoint-create --project your-project \
-d "AWS CloudWatch Logs" -t external_aws_cloudwatch_logs \
-c log_group_name=my-log-group \
-c access_key=YOURACCESSKEY \
-c secret_key=YOURSECRETKEY \
-c region=us-east-1

The required configuration parameters for the endpoint are as follows:

  • log_group_name - the name of the log group where log streams will get created, this field is optional and if left out a log group will be created automatically to AWS CloudWatch when the service integration is enabled
  • access_key - your AWS access key ID that has the proper access rights to CloudWatch Logs, namely logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents and logs:DescribeLogStreams
  • secret_key - your AWS secret access key
  • region - the AWS region the logs are stored, the region needs to have support for AWS CloudWatch Logs

Add the integration endpoint to your service using Aiven Client

To be able to send the logs to AWS CloudWatch Logs using the previously created endpoint, it needs to be attached to the service. For this you need the endpoint identifier. This you can get by listing the available endpoints for your project.

avn service integration-endpoint-list --project your-project

ENDPOINT_ID ENDPOINT_NAME ENDPOINT_TYPE
================================= ================= ================
02b4ee79-3c09-4608-87a1-4ee716... AWS CloudWatch... external_aws_...

Using this endpoint id you can attach the service to the endpoint.

avn service integration-create --project your-project \
-t external_aws_cloudwatch_logs -s your-service \
-D 02b4ee79-3c09-4608-87a1-4ee71609e0cb

Attaching the service to the endpoint will enable sending the service logs to AWS CloudWatch. Aiven platform will automatically create for you log streams to CloudWatch Logs to the log group you have provided in the integration endpoint configuration. If that is left out, the group is auto created and will be named using the name of your Aiven project prefixed by aiven-. The log streams are named based on the service used. Each instance of a service will get its own CloudWatch Logs stream.

To configure for example the data retention of your CloudWatch logs you need to go to the AWS CloudWatch console and modify the properties of the log groups. By default, the CloudWatch Logs data retention is set to never expire.

Did this answer your question?