We've added a new feature that lets you control the number of indices to keep. Previously you were able to set a max number of indices, and as soon as you exceeded that number, the oldest index would be deleted.

While useful for many of our users, this of course had some limitations. What if you wanted to save max 5 of one type of index and max 8 of another type? That is what we are addressing with the launch of this new feature.

If you create a new Elasticsearch service, or apply maintenance updates to your existing service, you will have the option to create glob-style patterns and set a unique max count per pattern.

Examples

  • Pattern logs will match index logs but not logsfoo or foologs
  • Pattern logs* will match index logsfoo and logs_foo_bar but not foologs
  • Pattern *_logs_* will match index foo_logs_bar but not foologsbar
  • Pattern logs.? will match index logs.1 but not logs.11

Catch-all pattern

You can also create a catch-all pattern; *  which will match all indices.

Be aware that we will iterate over all your patterns, consider the following scenario:

You have a bunch of logs indices, logs.1 , logs.2  ... logs.34 . On top of that you have also created some new test-indices that you aren't really using yet: test.1 ,
test.2  and test.3 .

You then create logs.*  pattern with max count set to 8, and one * pattern with max count set to 3. We will then iterate over your patterns one by one, first making sure we only keep the 8 newest logs indices due to your logs.*  pattern. After that it will run the *  pattern, which will hit all your indices, and since the test indices are the
newest ones, it will keep those and delete all of your logs indices. Keep this in mind and be careful when setting a catch-all pattern.

Max index count 0

Additionally, setting max count to 0 means your pattern will have no effect what so ever, it will simply be ignored and nothing will be deleted. If you want to disable pattern temporarily, this is one way to achieve that.

Logs integration retention times

If you use logs integration with per-integration retention times, do note Aiven Elasticsearch will apply both index patterns and logs integration retention times. If you use both logs integration and custom index patterns in your Elasticsearch service, it is recommended to only set either one to clean up indexes to avoid unnecessary confusion:

  • Either set logs integration retention time to the maximum value (10000), or
  • Don't add index patterns for index prefixes managed by logs integration.

Both can be set, and smaller setting will take effect.

Sounds good. How do I create these patterns?

The easiest way to create Elasticsearch index cleanup patterns is to go to Aiven console, select your Elasticsearch service and then click the "Index" tab. At the top of that window, you will have the option to either create a new pattern, or delete any existing ones.

Alternatively you can use our API with a request similar to 

curl -X PUT --data '{"user_config":{"index_patterns": [{"pattern": "logs*", "max_index_count": 2},{"pattern":"test.?", "max_index_count": 3}]}' header "content-type: application-json" --header "authorization: aivenv1 <YOUR TOKEN HERE>" https://api.aiven.io/v1beta/project/<project>/service/<service_name>
Did this answer your question?