Kafka Connect is a framework and a runtime for integrating Kafka with other systems. Kafka Connectors can either be source (for pulling data from other systems into Kafka) or sink connectors (for pushing data into other systems from Kafka).

Aiven supports the ability to run Kafka Connect as a separate service from your primary Kafka cluster.

Using an official Aiven Terraform Provider you can deploy a Kafka Connect service by setting up service integration endpoint.

To get started, please first see our Aiven Terraform integration guide to prepare Terraform script that will define your Aiven infrastructure.

Primary Kafka service

Kafka Connect service works together with the main Kafka cluster, and therefore we need to define Kafka service as well. If you previously setup Kafka service, you can import your existing infrastructure to Terraform.

# Kafka service
resource "avn_service" "kafka-service1" {
project = avn_project.kafka-con-project1.project
cloud_name = "google-europe-west1"
plan = "business-4"
service_name = "kafka-service1"
service_type = "kafka"
maintenance_window_dow = "monday"
maintenance_window_time = "10:00:00"
kafka_user_config {
kafka_version = "2.4"
}
}

Kafka Connect service

Kafka Connect service will be setup as a separate standalone service which will be visible in Aiven console and can be managed separately from the main Kafka service.

# Kafka connect service
resource "avn_service" "kafka_connect1" {
project = avn_project.kafka-con-project1.project
cloud_name = "google-europe-west1"
plan = "startup-4"
service_name = "kafka-connect1"
service_type = "kafka_connect"
maintenance_window_dow = "monday"
maintenance_window_time = "10:00:00"
kafka_connect_user_config {
kafka_connect {
consumer_isolation_level = "read_committed"
}
public_access {
kafka_connect = true
}
}
}

Service Integration

Finally we need to connect main Kafka service with Kafka Connect by setting up service integration endpoint which will point our new Kafka Connect to the Kafka cluster.

// Kafka connect service integration
resource "avn_service_integration" "i1" {
project = avn_project.kafka-con-project1.project
integration_type = "kafka_connect"
source_service_name = avn_service.kafka-service1.service_name
destination_service_name = avn_service.kafka_connect1.service_name
kafka_connect_user_config {
kafka_connect {
group_id = "connect"
status_storage_topic = "__connect_status"
offset_storage_topic = "__connect_offsets"
}
}
}

Creating Kafka Connectors

After Kafka Connect service integration is complete, it is time to create connectors that can either be source or sink connectors. Aiven supports over 20 open source connectors... and counting

In the example below, we will add a Kafka topic first and then setup ElasticSearch Sink Connector.

# Kafka topic
resource "aiven_kafka_topic" "kafka-topic1" {
project = avn_project.kafka-con-project1.project
service_name = avn_service.kafka-service1.service_name
topic_name = "test-kafka-topic1"
partitions = 3
replication = 2
}

# Kafka connector
resource "aiven_kafka_connector" "kafka-es-con1" {
project = avn_project.kafka-con-project1.project
service_name = avn_service.kafka-service1.service_name
connector_name = "kafka-es-con1"
config = {
"topics" = aiven_kafka_topic.kafka-topic1.topic_name
"connector.class" : "io.aiven.connect.elasticsearch.ElasticsearchSinkConnector"
"type.name" = "es-connector"
"name" = "kafka-es-con1"
"connection.url" = avn_service.es-service1.service_uri
}
}

You may configure ElasticSearch sink connector with Aiven ElasticSearch or an external service. In the example below, we will use Aiven ElasticSearch

# Elasticsearch service
resource "aiven_service" "es-service1" {
project = aiven_project.kafka-con-project1.project
cloud_name = "google-europe-west1"
plan = "hobbyist"
service_name = "es-service1"
service_type = "elasticsearch"
maintenance_window_dow = "monday"
maintenance_window_time = "10:00:00"
elasticsearch_user_config {
elasticsearch_version = "7"
}
}

Your Kafka Connect service should now be completely setup and your new connectors are up and running and processing data to/from your Kafka service.

Additional articles may be found on our Aiven Support page.

If you have any questions, please feel free to reach out to our Support and let us know.

Did this answer your question?