We want to ensure that we are always able to connect to the service during the migration phase. We can accomplish this with a few simple tests during the migration phase.
Overview
We will complete the following steps to safely migrate Aiven services into a VPC.
Create VPC and set up Peering
Validate network peering connection with test service
Enable public access on all services to be migrated
Change application configs to use public access hostnames
Migrate service into VPC
Validate peering connections to all private hostnames and ports
Change application configs to use public access hostnames
Disable public access on all services
Note that steps 3, 4, 6, 7, 8 are optional but highly recommended. Following these steps ensures that networking configurations, and firewall rules are set up correctly.
Step 1: Initial Setup
Ensure that you have the VPC created and peering is active. This is simple to do and can be automated via terraform if needed. See Aiven's Using VPC Peering help article to set up the peering connection.
I will be using the google-us-east1 VPC for testing. Ensure that the VPC and peering are both in the "Active" state.
Step 2: Testing the Connection
First we want to ensure that we can connect to with a non-critical service to ensure that the networks are peered. In this case, I am going to create a small service inside the Aiven VPC just to test the network connection.
Ensure that your service is deployed into the Project VPC. I deployed a Jump Box into my VPC in GCP and set up SSH access.
From the command line, I want to make sure that I can resolve the DNS hostname, and connect to the service port. The following commands worked on Ubuntu 20.04 LTS
nslookup {host}nc -vz {host} {port}
At this point, I deleted the test service inside the VPC.
Step 3: Enable Public Access
Enable the public_access.kafka config on all of your services in the "Advanced Configuration" section. This will create a new hostname and port. It will still be publicly available once the service is moved into the VPC.
You will now see a new hostname and port under "Connection Information" by selecting the Public "Access Route"
Ensure that you can connect to each host/port combination as seen below.
Step 4: Configure and Redeploy Application
It is highly recommended to configure your applications to use the public access route during migration. This ensures access to the services while moving into a private network. In dual access mode, you can test all connections and ports before finally cutting over to use the private hostname(s) and underlying private IP's.
Step 5: Migrate Aiven Service into VPC
Use the "Cloud and VPC" section on the service overview to migrate your Aiven services into a VPC. Note the Public Internet tag.
Ensure that you select the region from the VPC tab. This is a dedicated VPC for your project.
Ensure that you see the Project VPC tag after migration. You can monitor the migration status on the service details page.
Step 6: Testing the Service Connections
After the migration, you can see that I am seeing some private IP's in the nslookup command. Ensure that you can connect to the private hostnames and ports, i.e. firewall rules and routing works.
Step 7: Configure and Redeploy Application
Now you can convert your application to use the private hostname again.
Step 8: Cleanup by Removing Public Access
Disable the public_access.kafka config on all of your services in the "Advanced Configuration" section. This will remove the public- prefixed hostname and port.
Conclusion
These steps allow you to perform public => VPC service migrations with zero downtime in a safe manner by testing connections every step of the way. As always, ensure that you client applications have failure and retry logic as the underlying servers and IP's change. This is usually not an issue in clustered services, e.g. Kafka and Elasticsearch, but might require additional configuration for services like PostgreSQL and Redis.