Note: Aiven for OpenSearch replaces Aiven for Elasticsearch after Elasticsearch version 7.10.2 due to licensing changes. In addition, OpenSearch Dashboards replaces Kibana. Starting from September 24th, 2021 until March 23rd, 2022, Aiven supports services running both OpenSearch and Elasticsearch 7.10.2. For more information on Aiven for OpenSearch, see our developer documentation.

The standard approach for index retention is to set the maximum number of indexes to store, and once that number is exceeded, the oldest index is deleted. However, this has some limitations. For example, you might want to set the maximum to 5 for one type of index and 8 for another type.

Aiven for Elasticsearch services give you the option to create glob-style patterns and set a unique maximum for each pattern.

Here are some examples of the patterns that you can use:

  • logs: matches logs but not logsfoo or foologs

  • logs*: matches logsfoo and logs_foo_bar but not foologs

  • *_logs_*: matches foo_logs_bar but not foologsbar

  • logs.?: matches logs.1 but not logs.11

You can also use * as a catch-all pattern that will match all indices. However, this iterates through all your patterns, so you should consider the impact carefully before using it.

For example, if you have log indexes logs.1 , logs.2, and so on, up to logs.34, along with new test.1, test.2, and test.3 indexes that are intended for testing purposes but are not in use yet.

If you then create a logs.* pattern with the maximum set to 8 and a * pattern with the maximum set to 3, the system iterates through your patterns one by one. First, it makes sure that it only keeps the 8 newest log indexes based on your logs.* pattern. Then, it runs the * pattern, which affects all your indexes, and since the test indices are the newest ones, it keeps those and deletes all your log indexes. Keep this in mind and be careful when setting a catch-all pattern.

Additionally, setting the maximum to 0 means that your pattern has no effect. The system ignores the maximum setting does not delete anything. You can use this if you want to disable the pattern temporarily.

If you use log integration with integration-specific retention times, note that Aiven for Elasticsearch applies both index patterns and integration retention times. If you use both log integrations and custom index patterns in your Elasticsearch service, we recommend that you use only one of them to clean up indexes:

  • Set the retention time for log integrations to the maximum value (10000)

  • Do not add index patterns for index prefixes managed by log integrations

You can set both, in which case the smaller setting takes effect.

To create cleanup patterns for Elasticsearch indexes:

  1. Log in to the Aiven web console and select your service.

  2. Click the Indexes tab.
    The top of this view lists the patterns that are currently in use.

  3. Click Add New Pattern.

  4. Enter the pattern that you want to use and the maximum index count for the pattern, then click Create.

Alternatively, you can use our API with a request similar to the following: 

curl -X PUT --data '{"user_config":{"index_patterns": [{"pattern": "logs*", "max_index_count": 2},{"pattern":"test.?", "max_index_count": 3}]}' header "content-type: application-json" --header "authorization: aivenv1 <YOUR TOKEN HERE>" https://api.aiven.io/v1beta/project/<project>/service/<service_name>

Learn how Aiven simplifies working with Elasticsearch:

Did this answer your question?