Due to popular demand, we've created a service integration that allows you to send your Kafka, PostgreSQL, Cassandra, Elasticsearch, Redis, InfluxDB, and Grafana logs to you managed Elasticsearch instance.
By doing so, you will be able to leverage the storage and searchability of Elasticsearch, giving you greater insight and command over your service logs.
Let's take a look and check out the five steps it takes to launch the integration, as well as an alternative process that'll allow you to do it in as little as four.
1: Choose the service you'd like to integrate with Elasticsearch
Begin from the service overview page and select the service whose logs you'd like to send to Elasticsearch. In this example, we have chosen 'kafka-uat.
2: Select Manage in the service integrations section*
Next, you will be taken to your service overview page. Scroll down to the Service integrations section and select Manage integrations; this will open a service integrations dialogue box.
3. Select Elasticsearch logs integration
Within the service integrations dialogue box, you will be shown the service integrations available to you. Select the Logs integration to send your service logs to an Elasticsearch instance.
4. Choose an existing Elasticsearch instance or create a new one
If you have an existing Elasticsearch instance, you can send your service logs there. Otherwise, you can create a new Elasticsearch instance to send them to: click Continue.
5. Configure your index prefix and retention limit
Once you select Continue, you will then be able to configure your Elasticsearch log integration; namely, your Elasticsearch index prefix and index retention limit. Click Enable. If you want to effectively disable index retention limit, you can set it to maximum value of 10000 days. Instead of index retention limits, you may want to consider using index patterns to manage number of indexes per prefix.
You can also choose to set up your Elasticsearch logs integration by navigating to the Logs tab of your service panel and selecting Enable Logs integration. This will take you directly to step 4.
To find out more information on the Elasticsearch logs integration and updates we made to our default console logs (pictured directly above), check out this blog post as well.