It is possible to send the logs of your Aiven services to Amazon AWS CloudWatch for storing, viewing and archiving them outside the Aiven platform.
Creating the AWS CloudWatch Logs integration
You need to start by creating a AWS CloudWatch Logs integration endpoint to the Aiven project you're using. It is possible to do this either inside the Aiven Web Console or by using the Aiven Client.
Create the integration endpoint using Aiven Web Console
You can enable the AWS CloudWatch Logs integration in the Service Integrations section of your project. You will need to give a name of the log group used in AWS CloudWatch as well as AWS credentials with appropriate access rights and the AWS region to be used. The access rights required for the credentials are l
logs:DescribeLogStreams. If a log group with given name does not exist in CloudWatch, it will be generated. Properties of the log group, like data retention, can be modified in the AWS console.
Add the integration endpoint to your service using Aiven Web Console
Go to the service you want to add the logs integration to and select Manage Integrations. Then find the AWS CloudWatch Logs integration from the list and click Use Integration. After this you will be asked to select the endpoint you want to use for the integration.
Create the integration endpoint using Aiven Client
avn service integration-endpoint-create --project your-project \
-d "AWS CloudWatch Logs" -t external_aws_cloudwatch_logs \
-c log_group_name=my-log-group \
-c access_key=YOURACCESSKEY \
-c secret_key=YOURSECRETKEY \
The required configuration parameters for the endpoint are as follows:
log_group_name- the name of the log group where log streams will get created, this field is optional and if left out a log group will be created automatically to AWS CloudWatch when the service integration is enabled
access_key- your AWS access key ID that has the proper access rights to CloudWatch Logs, namely
secret_key- your AWS secret access key
region- the AWS region the logs are stored, the region needs to have support for AWS CloudWatch Logs
Add the integration endpoint to your service using Aiven Client
To be able to send the logs to AWS CloudWatch Logs using the previously created endpoint, it needs to be attached to the service. For this you need the endpoint identifier. This you can get by listing the available endpoints for your project.
avn service integration-endpoint-list --project your-project
ENDPOINT_ID ENDPOINT_NAME ENDPOINT_TYPE
================================= ================= ================
02b4ee79-3c09-4608-87a1-4ee716... AWS CloudWatch... external_aws_...
Using this endpoint id you can attach the service to the endpoint.
avn service integration-create --project your-project \
-t external_aws_cloudwatch_logs -s your-service \
Attaching the service to the endpoint will enable sending the service logs to AWS CloudWatch. Aiven platform will automatically create for you log streams to CloudWatch Logs to the log group you have provided in the integration endpoint configuration. If that is left out, the group is auto created and will be named using the name of your Aiven project prefixed by
aiven-. The log streams are named based on the service used. Each instance of a service will get its own CloudWatch Logs stream.
To configure for example the data retention of your CloudWatch logs you need to go to the AWS CloudWatch console and modify the properties of the log groups. By default, the CloudWatch Logs data retention is set to never expire.