In addition to using Aiven for Elasticsearch to store the logs from your Aiven services in the same project (via logs integration), you can now integrate with a hosted Elasticsearch cluster in different project or one outside of the Aiven platform.

Creating external Elasticsearch integration 

As the first step, you need to add the remote Elasticsearch you want to send the logs to into the project that contains the service you want to integrate. Currently this can be done using the Aiven Client or the Web Console.

Using the Web Console

You can enable an external Elasticsearch in the Service Integrations  section of your project.

Enter the details of your service here and then navigate to the service that you want to integrate. Scroll down to Service Integrations  and select Elasticsearch  from the dialog and you will see the ID that you set when you added your external service in the next screen. Confirm that and start receiving your logs!

Add external Elasticsearch integration endpoint

Using the Aiven Client

avn service integration-endpoint-create --project your-project \
    -d example-eslogs -t external_elasticsearch_logs \
    -c url=https://user:pswd@logs.example.com \
    -c index_prefix=logs -c index_days_max=3

When defining the external Elasticsearch log server the following parameters can be applied

required:

  • url -  connection URL for the service 
  • index_prefix - prefix to use when creating the daily log indexes, the integration will automatically append a dash followed by the date in YYYY-MM-DD format to create a unique daily index (in the above example the indexes created would be logs-2019-05-20, logs-2019-05-21, etc)

recommended (optional): 

  • index_days_max - (default: 3) how many days logs are kept. The integration will automatically delete indexes that are older than the given value.

optional:

  • ca - (PEM format) Certificate Authority to use for verifying the servers certificate (typically not needed unless the server's certificate is issued by an internal CA or it uses a self-signed certificate)
  • timeout - time in seconds to wait for Elasticsearch request to complete. The default is 10 seconds. For a service under a heavy load or high latency you may need to increase this.  

Add external Elasticsearch integration to service

First you need the id of the endpoint previously created 

avn service integration-endpoint-list --project your-project
ENDPOINT_ID                           ENDPOINT_NAME   ENDPOINT_TYPE
====================================  ==============  =============
618fb764-5832-4636-ba26-0d9857222cfd  example-eslogs  external_e...

Finally you can link the service to the endpoint

avn service integration-create --project your-project \
    -t external_elasticsearch_logs -s your-service \
    -D 618fb764-5832-4636-ba26-0d9857222cfd

Got here by accident? Learn how Aiven simplifies working with Elasticsearch:

Did this answer your question?