It is possible to send the service logs of your Aiven services to Google Cloud Logging, formerly known as Google Stackdriver. Cloud Logging allows you to store, search, analyze, monitor, and alert on log data from your services running in the Aiven cloud.

Creating the Google Cloud Logging integration

You need to start by creating a Google Cloud Logging integration endpoint to the Aiven project that you are using. It is possible to do this either inside the Aiven Web Console or by using the Aiven Client. We'll go through both options here.

Create the integration endpoint using the Aiven Web Console

You can enable the Google Cloud Logging integration in the Service Integrations section of your project. You will need to configure the Google Cloud Platform project ID as well as the log ID used when sending the logs to Google Cloud Logging. Note that the value of log ID needs to be URL-encoded.

You will also need to provide service account credentials, that will be used when authenticating against Google Cloud Platform. You can find instructions on how to create this credential (private key) from the official Google documentation here.

The service account needs permission logging.logEntries.create. This permission is granted by the 2 roles: roles/logging.logWriter, and roles/editor. The official Google documentation also explains how to add roles to your service account.

Also, important to notice is that the credentials need to be provided in JSON format.

Add the integration endpoint to your service using the Aiven Web Console

Go to the service you want to add the logs integration to and select Manage Integrations. Then find the Google Cloud Logging integration from the list and click Use Integration. After this you will be asked to select the endpoint you want to use for the integration.

Create the integration endpoint using the Aiven client

avn service integration-endpoint-create --project your-project
-d "Google Cloud Logging" -t external_google_cloud_logging
-c project_id=your-gcp-project-id
-c log_id=my-aiven-service-logs
-c service_account_credentials='{"type": "service_account...'

The required configuration parameters for the endpoint are as follows:

  • project_id - The ID of the Google Cloud Platform project you want to send the service logs to.
  • log_id - The ID associated with the logs you are sending to Google Cloud Logging. Note that value needs to be URL-encoded.
  • service_account_credentials - Google Cloud Platform service account credentials to be used when sending out the logs. The account needs permission logging.logEntries.create. Also, important to notice is that the service account credentials need to be in JSON format.

Add the integration endpoint to your service using Aiven client

To be able to send the logs to Google Cloud Logging using the previously created endpoint, it needs to be attached to the service. For this you need the endpoint identifier. This you can get by listing the available endpoints for your project.

avn service integration-endpoint-list --project your-project

ENDPOINT_ID ENDPOINT_NAME ENDPOINT_TYPE
================================= ================= ================
02b4ee79-3c09-4608-87a1-4ee716... Google Cloud L... external_goog...

Using this endpoint id you can attach the service to the endpoint.

avn service integration-create --project your-project \
-t external_google_cloud_logging -s your-service \
-D 02b4ee79-3c09-4608-87a1-4ee71609e0cb

Attaching the service to the endpoint will enable sending the service logs to Google Cloud Logging. You can view the logs through the Google Cloud Platform console.

Did this answer your question?