We recommend migrating to Aiven Elasticsearch with reindexing from remote cluster. The same process works for migrating from Aiven Elasticsearch to self-hosted Elasticsearch service.
To migrate (copy) data, set
reindex.remote.whitelist parameter with Aiven CLI client:
avn service update your-service-name-here -c elasticsearch.reindex_remote_whitelist=your.non-aiven-service.example.com:9200
Remember to replace port number with the port your non-Aiven Elasticsearch service is listening at. After running this command, all Elasticsearch instances will automatically restart.
Restarting may take several minutes, as we try to do a rolling restart to minimize service downtime. There is no need to power off the service.
After restart is done, you'll first need to create your (empty) destination index with any tool, for example with
curl -XPUT https://avnadmin:email@example.com:23125/logs-2024-09-21
you can reindex data from your remote Elasticsearch service. With
curl , this would be
curl -XPOST https://avnadmin:firstname.lastname@example.org:23125/_reindex \
-H 'Content-type: application/json' \
If you see something like
[your.non-aiven-service.example.com:9200] not whitelisted in reindex.remote.whitelist in the response, check the hostname and port you are using matches to what you set earlier.
Depending on the amount of data you have, reindexing may take significant amount of time.
As a complete process:
- Create an Aiven Elasticsearch service
reindex.remote.whitelistparameter to point to your source Elasticsearch service.
- Wait for the cluster to restart (typically a few minutes).
- Start migrating indexes
For each index:
- Stop writes to the index. If you are testing the process, no need to stop writes.
- Create index on your destination Elasticsearch service,
- Submit a reindex request,
- Wait for reindex to complete,
- Point clients to use that index from Aiven Elasticsearch for both reads and writes and resume any write activity,
- Optionally, delete the source index
For a larger number of indexes, we recommend automating this with a simple script executing these steps automatically.
As Aiven Elasticsearch does not support joining external Elasticsearch servers to the same cluster, online migration is not currently possible.
If you have any questions about this process, please don't hesitate to contact our support at email@example.com.