In addition to using Aiven for Elasticsearch to store the logs from your Aiven services in the same project (via logs integration), you can now integrate with a hosted Elasticsearch cluster in different project or one outside of the Aiven platform.
Creating external Elasticsearch integration
Add external Elasticsearch integration endpoint
As the first step, you need to add the remote Elasticsearch you want to send the logs to into the project that contains the service you want to integrate. Currently this can be done using the Aiven Client (web console support is going to be available later).
avn service integration-endpoint-create --project your-project \
-d example-eslogs -t external_elasticsearch_logs \
-c url=https://user:email@example.com \
-c index_prefix=logs -c index_days_max=3
When defining the external Elasticsearch log server the following parameters can be applied
- url - connection URL for the service
- index_prefix - prefix to use when creating the daily log indexes, the integration will automatically append a dash followed by the date in YYYY-MM-DD format to create a unique daily index (in the above example the indexes created would be logs-2019-05-20, logs-2019-05-21, etc)
- index_days_max - (default: 3) how many days logs are kept. The integration will automatically delete indexes that are older than the given value.
- ca - (PEM format) Certificate Authority to use for verifying the servers certificate (typically not needed unless the server's certificate is issued by an internal CA or it uses a self-signed certificate)
- timeout - time in seconds to wait for Elasticsearch request to complete. The default is 10 seconds. For a service under a heavy load or high latency you may need to increase this.
Add external Elasticsearch integration to service
First you need the id of the endpoint previously created
avn service integration-endpoint-list --project your-project
ENDPOINT_ID ENDPOINT_NAME ENDPOINT_TYPE
==================================== ============== =============
618fb764-5832-4636-ba26-0d9857222cfd example-eslogs external_e...
Finally you can link the service to the endpoint
avn service integration-create --project your-project \
-t external_elasticsearch_logs -s your-service \