Note: Aiven for OpenSearch replaces Aiven for Elasticsearch after Elasticsearch version 7.10.2 due to licensing changes. In addition, OpenSearch Dashboards replaces Kibana. Starting from September 24th, 2021 until March 23rd, 2022, Aiven supports services running both OpenSearch and Elasticsearch 7.10.2. For more information on Aiven for OpenSearch, see our developer documentation.

When running low on disk space on Aiven Elasticsearch, there are two possible actions:

For example, for logs, it is often beneficial to create a separate, daily indexes, allowing easy and efficient clean-up of the oldest data.

If the service runs low on disk space, three different Elasticsearch mechanisms take effect:

  • cluster.routing.allocation.disk.watermark.low : defaults to 85% of disk space used. When this limit is exceeded, Elasticsearch starts avoiding allocating new shards to the server. On a single-server Elasticsearch, this has no effect. On a multi-server cluster, amount of data is not always equally distributed, in which case this will help balancing disk usage between servers.

  • cluster.routing.allocation.disk.watermark.high : defaults to 90% of disk space used. Elasticsearch will actively try to move shards to other servers with more available disk space. On a single-server Elasticsearch, this has no effect.

  • cluster.routing.allocation.disk.watermark.flood_stage : defaults to 95% of disk space used. Elasticsearch will mark all indexes hosted on the server exceeding this limit as read-only, allowing deletes (index.blocks.read_only_allow_delete ).

If low  or high thresholds are exceeded, and data is cleaned up, no further action is needed, as Elasticsearch will continue allowing writes.

If flood_state  is exceeded, you must manually unset read_only_allow_delete  for each affected index. This is done by updating index settings:

curl https://avnadmin:your-password@your-server-demoprj.aivencloud.com:12345/indexname/_settings -X PUT -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": null}' 

This needs to be done separately for each index that was marked as read-only by Elasticsearch.

Aiven does not automatically unset this option to ensure no flipping around read-only and read-write state.

Got here by accident? Learn how Aiven simplifies working with Elasticsearch:

Did this answer your question?