Note: AWS PrivateLink support is currently a beta feature. Please get in touch with support if you are interested in trying out this feature!

AWS PrivateLink brings Aiven services to the selected VPC in your AWS account. In a traditional setup using VPC peering, traffic is routed through an AWS VPC peering connection to your Aiven services. With PrivateLink it is possible to create an VPC Endpoint to your own VPC and access an Aiven service. The VPC Endpoint creates network interfaces (NICs for short) into the subnets and availability zones you choose, and receives private IP addresses belonging to your VPC's IP range. The VPC Endpoint is routed to your Aiven service located in one of Aiven's AWS accounts.

Setting it up

Privatelink can be enabled for Aiven services located in Project VPCs. Start by creating a VPC and launch the service(s) you wish to connect to that VPC. Unless you also plan on connecting to the Project VPC using VPC peering connections, any private IP range for the VPC can be used because there is no network routing between the VPCs. Therefore IP range overlap is not an issue.

Create an AWS PrivateLink VPC Endpoint Service

Once the service has been created, use the Aiven CLI to create a privatelink resource on the Aiven service. This creates an AWS Network Load Balancer dedicated to your Aiven service and attaches it to an AWS VPC Endpoint Service which we'll later use to connect your account's VPC Endpoint to.

# avn service privatelink aws create --principal arn:aws:iam::012345678901:user/mwf my-kafka
============== ================ ================================== ========
null null arn:aws:iam::012345678901:user/mwf creating

The command above creates a privatelink for the my-kafka service. The ARNs passed with --principal arguments are allowed to connect to the VPC Endpoint Service created together with the AWS Network Load Balancer. You can allow access for an entire AWS account, or a given user (as in the example) or role. Do only give permissions to roles that you trust, as the role can connect from any VPC of their choosing!

The privatelink resource is up to a few minutes in the initial creating state while the load balancer is being launched:

# avn service privatelink aws get my-kafka 
============== ================ ================================== ========
null null arn:aws:iam::012345678901:user/mwf creating

After the Load Balancer and VPC Endpoint Service have been created state changes to active and aws_service_id_ and aws_service_name are set:

# avn service privatelink aws get my-kafka

========================== ======================================================= ================================== ======
vpce-svc-0b16e88f3b706aaf1 arn:aws:iam::012345678901:user/mwf active

Create an AWS VPC Endpoint

At this point AWS resources in the Aiven account's Project VPC have been set up and ready to be connected from your account and VPC. To create a VPC Endpoint from with the AWS CLI:

# aws ec2 --region eu-west-1 create-vpc-endpoint --vpc-endpoint-type Interface --vpc-id $your_vpc_id --subnet-ids $space_separated_list_of_subnet_ids --security-group-ids $security_group_ids --service-name

Replace the --service-name with the value shown by avn service privatelink aws get aws_service_name field. Note that you should specify a subnet id for each availability zone in the region for fault tolerance. The security groups determine which instances are allowed to connect to the endpoint network interfaces created by AWS into the subnets you have specified.

Please note that the security group for the VPC endpoint must allow ingress in the port range 10000-31000 to accommodate the pool of Kafka broker ports used in our PrivateLink implementation.

Alternatively the VPC endpoint can be created in the AWS web console from VPC -> Endpoints -> Create endpoint. See AWS's documentation for details.

It will again take a while before the endpoint is ready to use as AWS provisions network interfaces to each of the subnets and connect them to the Aiven side VPC Endpoint Service. Once the AWS endpoint state changes to available, the connection is visible in Aiven:

# avn service privatelink aws connection list my-kafka

{"dns_name": "", "state": "active", "vpc_endpoint_id": "vpce-0d67e33206dc4f897"}

Enable PrivateLink access for Aiven service components

Lastly, PrivateLink access needs to be enabled on the Aiven service. Each service component can be controlled separately - for example you can enable PrivateLink access for Kafka, while allowing Kafka Connect to only be connected via VPC peering connections. Set user_config.privatelink_access.<service component> to true for the components you wish to enable. With Aiven CLI:

# avn service update -c privatelink_access.kafka=true my-kafka
# avn service update -c privatelink_access.kafka_connect=true my-kafka
# avn service update -c privatelink_access.kafka_rest=true my-kafka
# avn service update -c privatelink_access.schema_registry=true my-kafka

Access can also be set up in the Aiven web console from Service -> Overview tab -> Advanced configuration. Add the options from the list and toggle the switch to set the value to true.

It will take a minute or two before connectivity is available after enabling a service component. This is because AWS requires an AWS Load Balancer behind each VPC Endpoint Service, and target rules on the LB for the service nodes need at least two successful heartbeats before they transition from the initial state to healthy and are included in the LB's active forwarding rules.

Note that currently only one VPC Endpoint can be created for each Aiven service

Connection information

Once privatelink access has been enabled for a service component, a switch for the privatelink access route appears in Connection information on the Overview tab in the web console. The host and for some service components such as Kafka, port values will be different from the regular dynamic access route that is by default used to connect to the service. The same credentials can be used with any access route.

Update allowed principals list

To change the list of AWS accounts, IAM users or roles allowed to connect a VPC Endpoint, use the update command of the Aiven CLI:

# avn service privatelink aws update --principal arn:aws:iam::012345678901:role/my-privatelink-role my-kafka

Note that when adding an entry, --principal arguments should be included for existing entries as well

Delete a PrivateLink

Use the Aiven CLI to delete the AWS Load Balancer and VPC Service Endpoint:

# avn service privatelink aws delete my-kafka

========================== ======================================================= ================================== ========
vpce-svc-0b16e88f3b706aaf1 arn:aws:iam::007268611461:user/mwf deleting
Did this answer your question?