We recommend migrating to Aiven Elasticsearch with reindexing from remote cluster. The same process works for migrating from Aiven Elasticsearch to self-hosted Elasticsearch service.

To migrate (copy) data, set reindex.remote.whitelist  parameter with Aiven CLI client

avn service update your-service-name-here -c elasticsearch.reindex_remote_whitelist=your.non-aiven-service.example.com:9200 

Remember to replace port number with the port your non-Aiven Elasticsearch service is listening at. After running this command, all Elasticsearch instances will automatically restart.

Restarting may take several minutes, as we try to do a rolling restart to minimize service downtime. There is no need to power off the service.

After restart is done, you'll first need to create your (empty) destination index with any tool, for example with curl :

curl -XPUT https://avnadmin:yourpassword@es-123-demoprj.aivencloud.com:23125/logs-2024-09-21

 you can reindex data from your remote Elasticsearch service. With curl , this would be

curl -XPOST https://avnadmin:yourpassword@es-123-demoprj.aivencloud.com:23125/_reindex \
  -H 'Content-type: application/json' \
  -d '{"source":
          {"index": "logs-2024-09-21",
           "remote":
               {"username": "your-remote-username",
                "password": "your-remote-password",
                "host": "https://your.non-aiven-service.example.com:9200"
               }
          },
       "dest":
          {"index": "logs-2024-09-21"}
      }'

If you see something like [your.non-aiven-service.example.com:9200] not whitelisted in reindex.remote.whitelist  in the response, check the hostname and port you are using matches to what you set earlier.

Depending on the amount of data you have, reindexing may take significant amount of time.

As a complete process:

  1. Create an Aiven Elasticsearch service
  2. Set reindex.remote.whitelist  parameter to point to your source Elasticsearch service.
  3. Wait for the cluster to restart (typically a few minutes).
  4. Start migrating indexes

For each index:

  1. Stop writes to the index. If you are testing the process, no need to stop writes.
  2. Create index on your destination Elasticsearch service,
  3. Submit a reindex request,
  4. Wait for reindex to complete,
  5. Point clients to use that index from Aiven Elasticsearch for both reads and writes and resume any write activity,
  6. Optionally, delete the source index

For a larger number of indexes, we recommend automating this with a simple script executing these steps automatically.

As Aiven Elasticsearch does not support joining external Elasticsearch servers to the same cluster, online migration is not currently possible.

If you have any questions about this process, please don't hesitate to contact our support at support@aiven.io.

Did this answer your question?