Starting from the 1st of March, it is possible to use a cross-account access role in Aiven S3 Sink Kafka Connector. Before that, the only option was to specify long-term AWS credentials ACCESS_KEY_ID and SECRET_ACCESS_KEY. The new approach assumes that Kafka Connector will request a short-term credential every time it has a task to store data to an S3 bucket. It is considered to be a more secure option.
1. Request a unique IAM user from Aiven Support.
In Aiven, customer security is always a priority, so we create a dedicated IAM user for each customer. As a result, we never share credentials and roles among customers, and cross-account role gives access to one Aiven project only. It is also highly recommended to request an External Id, a unique identifier that makes cross-account access even more secure.
Here is a sample answer:
IAM user arn:aws:iam::012345678901:user/sample-project-user
External Id 2f401145-06a0-4938-8e05-2d67196a0695
2. Create a cross-account access role
The example uses the AWS console to create a cross-account access role.
A. Log in to AWS Console. Go to IAM ->Roles -> Create role
B. Select Another AWS account as a type of trusted entity. Specify Account ID*. Select an option Require external ID.
* Account ID is a number from IAM user between 'aws:iam::' and ':user/' in a given an example, it is '012345678901'.
C. Select permissions that allow writing to an S3 bucket.
D. [Optional] Add tags.
E. Specify a name for the role. Example AivenKafkaConnectSink
It would be best to use a full IAM user name to limit access to one IAM user only. To do so you need to edit the newly created role: IAM -> Roles -> ROLE_NAME -> Trust relationships -> Edit trust relationship
In a policy document, the IAM user should be specified as Principal.
Finally, copy a newly created IAM role ARN. It will be needed in Kafka Connector configuration.
3. Create an AWS S3 Sink connector
First, configure your Aiven S3 Sink connector as usual. In a tab AWS fields aws.secret.access.key and aws.access.key.id are not mandatory anymore. However, for performance reasons, consider specifying the output bucket region in aws.s3.region.
Copy IAM role ARN from a previous step to an aws.sts.role.arn in a tab AWS STS. The aws.sts.role.external.id field is the External Id from a Step 1.
Another important field is an aws.sts.role.session.name where you can specify any id to identify a task. Session name exists to separate different tasks from the same project, and it will be visible in AWS Could Trail logs. For performance reasons, consider to specify aws.sts.config.endpoint. If the source S3 bucket is in 'eu-north-1', it is better to set 'https://sts.eu-north-1.amazonaws.com' as an STS endpoint.