Certain compliance standards require extended storage of logs, e.g. 1-3 years. This is possible in many SEIM providers as well as Aiven's fully managed Elasticsearch logging integration. This article walks through the steps required to setup extended log retention in GCP.

Aiven + Google Cloud Logging + Logs Storage

Enabling extended log retention Aiven services only involves three simple steps:

  1. Enable the Google Cloud Logging integration
  2. Create a Logs Storage Bucket
  3. Create a Google Cloud Log Router

1. Enable the Google Cloud Logging Integration

Follow Aiven's Google Cloud Logging quick start guide.

Once the integration is enabled, you will be able to view the logs in the Logs Viewer. The exported logs will be available with the following filter

resource.type="generic_node"
logName="projects/<<gcp-project>>/logs/<<integration-log-id>>"

e.g.
resource.type="generic_node"
logName="projects/avn-tooling/logs/demo-logs"

2. Create a Logs Storage Bucket

You can create a new Logs Storage Bucket in the Google Cloud Console. Set the name, and retention based on your requirements.

3. Create a Google Cloud Log Router

View the list of existing Logs Routers in the Google Cloud Console. Create a new sink and notice the types of, including other SEIM providers.

After selecting "Cloud Logging bucket" enter a name, select the bucket created in step 2, and use the filter from step 1.

4. View Logs In the Bucket

It is easy to limit the Logs Viewer scope to only include logs from your extended retention bucket. In the Logs Storage list, select "View logs in this bucket."

FAQ

Which Aiven services support this?

Integrate Google Cloud Logging with your Aiven PostgreSQL, Kafka, Elasticsearch, MySQL and Redis services.

Does this affect my billing?

Using the Google Cloud Logging integration does not affect Aiven billing.

Do note though that Google Cloud will still charge for the integration's logs based on their regular pricing. The costs incurred from Google Cloud Logging are solely your responsibility as the Google Cloud account owner.

Did this answer your question?