For moving indexes from one service to another, elasticsearch-dump is a good option for doing so and be able to move it to you Elastisearch service in Aiven or have backup in AWS S3 for example. Before getting started, we recommend you to read the elasticsearch-dump tool information available on GitHub.

Aiven Database

To start with you'll need an Aiven Elasticsearch service. In order to create one you can follow the instructions in our Aiven Elasticsearch Getting Started guide.

Aiven Elasticsearch allows you to easily switch between different plan sizes but for the duration of the dumping process it usually makes sense to conservatively pick a plan size that is large enough for the task. This allows you to limit the downtime during the migration process by having a sufficiently powerful Elasticsearch plan.

Import data from your Elasticsearch to Aiven

To import the data we will use elasticdump command. It works by sending an input to an output. Both can be either an Elasticsearch URL or a file path (local or remote file storage). In this particular case we are using both URLs, one from Elastic Cloud and the other one from Aiven.

elasticdump \
 --input=https://elastic:secret@4ac0a9875c964442bf46ce9b5db6bbc5.asia-northeast1.gcp.cloud.es.io:9243/kibana_sample_data_logs/ \
 --output=https://avnadmin:secret@es-1ceb0262-test-marcelo-test.avns.net:23219/ \
 --type=data

When the dump is completed, you can now check that the index is available in the Elasticsearch service you send it to. You will able to find it under the indexes tab in your Aiven console.

Copying the data from Aiven Elasticsearch to S3

To copy the data we will also use elasticdump command. It works by sending an input to an output. In this particular case for the input we are using Aiven Elasticsearch URL and for the output a file path that will go directly in AWS S3 bucket. The S3 credentials are required as well for the operation to succeed.

elasticdump \
 --s3AccessKeyId "${access_key_id}" \
 --s3SecretAccessKey "${access_key_secret}" \
 --input=https://avnadmin:secret@es-1d6076d2-test-marcelo-test.avns.net:20354/kibana_sample_data_logs
--output "s3://aiventest-test/samplelogstest.json"


Now you have a copy of your Elasticsearch data in your AWS S3 bucket.

Did this answer your question?