In addition to using Aiven for Elasticsearch to store the logs from your Aiven services in the same project (via logs integration), you can now integrate with a hosted Elasticsearch cluster in different project or one outside of the Aiven platform.
Creating external Elasticsearch integration
As the first step, you need to add the remote Elasticsearch you want to send the logs to into the project that contains the service you want to integrate. Currently this can be done using the Aiven Client or the Web Console.
Using the Web Console
You can enable an external Elasticsearch in the Service Integrations section of your project.
Enter the details of your service here and then navigate to the service that you want to integrate. Scroll down to Service Integrations and select Elasticsearch from the dialog and you will see the ID that you set when you added your external service in the next screen. Confirm that and start receiving your logs!
Add external Elasticsearch integration endpoint
Using the Aiven Client
avn service integration-endpoint-create --project your-project \
-d example-eslogs -t external_elasticsearch_logs \
-c url=https://user:pswd@logs.example.com \
-c index_prefix=logs -c index_days_max=3When defining the external Elasticsearch log server the following parameters can be applied
required:
- url - connection URL for the service
- index_prefix - prefix to use when creating the daily log indexes, the integration will automatically append a dash followed by the date in YYYY-MM-DD format to create a unique daily index (in the above example the indexes created would be logs-2019-05-20, logs-2019-05-21, etc)
recommended (optional):
- index_days_max - (default: 3) how many days logs are kept. The integration will automatically delete indexes that are older than the given value.
optional:
- ca - (PEM format) Certificate Authority to use for verifying the servers certificate (typically not needed unless the server's certificate is issued by an internal CA or it uses a self-signed certificate)
- timeout - time in seconds to wait for Elasticsearch request to complete. The default is 10 seconds. For a service under a heavy load or high latency you may need to increase this.
Add external Elasticsearch integration to service
First you need the id of the endpoint previously created
avn service integration-endpoint-list --project your-project
ENDPOINT_ID ENDPOINT_NAME ENDPOINT_TYPE
==================================== ============== =============
618fb764-5832-4636-ba26-0d9857222cfd example-eslogs external_e...
Finally you can link the service to the endpoint
avn service integration-create --project your-project \
-t external_elasticsearch_logs -s your-service \
-D 618fb764-5832-4636-ba26-0d9857222cfdGot here by accident? Learn how Aiven simplifies working with Elasticsearch: