External Elasticsearch log integration

In addition to using Aiven for Elasticsearch to store the logs from your Aiven services in the same project (via logs integration), you can now integrate with a hosted Elasticsearch cluster in different project or one outside of the Aiven platform.

Creating external Elasticsearch integration 

Add external Elasticsearch integration endpoint

As the first step, you need to add the remote Elasticsearch you want to send the logs to into the project that contains the service you want to integrate. Currently this can be done using the Aiven Client (web console support is going to be available later).

avn service integration-endpoint-create --project your-project \
    -d example-eslogs -t external_elasticsearch_logs \
    -c url=https://user:pswd@logs.example.com \
    -c index_prefix=logs -c index_days_max=3

When defining the external Elasticsearch log server the following parameters can be applied

required:

• url -  connection URL for the service 
• index_prefix - prefix to use when creating the daily log indexes, the integration will automatically append a dash followed by the date in YYYY-MM-DD format to create a unique daily index (in the above example the indexes created would be logs-2019-05-20, logs-2019-05-21, etc)

recommended (optional): 

• index_days_max - (default: 3) how many days logs are kept. The integration will automatically delete indexes that are older than the given value.

optional:

• ca - (PEM format) Certificate Authority to use for verifying the servers certificate (typically not needed unless the server's certificate is issued by an internal CA or it uses a self-signed certificate)
• timeout - time in seconds to wait for Elasticsearch request to complete. The default is 10 seconds. For a service under a heavy load or high latency you may need to increase this.  

Add external Elasticsearch integration to service

First you need the id of the endpoint previously created 

avn service integration-endpoint-list --project your-project
ENDPOINT_ID                           ENDPOINT_NAME   ENDPOINT_TYPE
====================================  ==============  =============
618fb764-5832-4636-ba26-0d9857222cfd  example-eslogs  external_e...

Finally you can link the service to the endpoint

avn service integration-create --project your-project \
    -t external_elasticsearch_logs -s your-service \
    -D 618fb764-5832-4636-ba26-0d9857222cfd