This article shows you an example of how to deploy and run a Confluent Docker Image for Schema Registry in an AWS EC2 instance. The idea behind the registry is to store all the schema of data written to Kafka. Each record has an identifier for the schema that is stored inside the registry when data is produced into Kafka. The consumers can use the identifier to pull the record out of the Schema Registry and deserialize the data.

This example uses an EC2 instance and we will learn how to deploy multi-container applications using Docker Compose.

Requirements

  • An Aiven for Apache Kafka service. For instructions on setting up a new Kafka service, see this article.

  • Deploying the Confluent Docker image for Schema Registry on a VPS.

Configuring SSL Authentication

  • Aiven for Kafka service uses SSL for authentication, and the next step is used to configure the Keystore and Truststore for securing the traffic between Kafka and Schema Registry. For instructions on creating the trusted files for authentication, see this article.

Deploying Confluent Image for Schema Registry with Docker Compose

  • Installing the Docker Compose on a Linux System

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  • Apply executable permissions to the binary:

sudo chmod +x /usr/local/bin/docker-compose
  • Docker Compose uses a YAML file to define multi-container applications. The default name for a Compose YAML file is docker-compose.yml. We are going to download the docker-compose file from the Confluent Github repository. The following example shows the Compose file that defines the Confluent Docker image for the Schema Registry with Aiven for Kafka.

    ---
    version: '2'
    services:

    schema-registry:
    image: confluentinc/cp-schema-registry:7.1.0
    hostname: schema-registry
    container_name: schema-registry
    ports:
    - "8081:8081"
    environment:
    SCHEMA_REGISTRY_HOST_NAME: schema-registry
    SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: SSL://kafka-sebin-dev-sandbox.aivencloud.com:12693
    SCHEMA_REGISTRY_KAFKASTORE_LISTENERS: http://0.0.0.0:8081
    SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: SSL
    SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_LOCATION: /certs/client.truststore.jks
    SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_PASSWORD: secret
    SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_TYPE: PKCS12
    SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_LOCATION: /certs/client.keystore.p12
    SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_PASSWORD: secret
    SCHEMA_REGISTRY_KAFKASTORE_SSL_KEY_PASSWORD: secret
    volumes:
    - /home/ubuntu/csr:/certs
  • volumes key is where you tell Docker to create new volumes and map the root directory for the docker image.

  • Deploying the image: sudo docker-compose up -d

  • Command to check the status of the image and tail logs

    sudo docker-compose ps
    sudo docker logs --tail=100 schema-registry
  • Now the schema registry starts listening on port 8081, let's create a schema using the API.

    POST /subjects/person/versions HTTP/1.1
    Host: ec2-3-17-161-17.us-east-2.compute.amazonaws.com
    Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

    {"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"namespace\":\"my.examples\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}",
    "schemaType": "AVRO"}

    HTTP/1.1 200 OK
    Date: Wed, 17 Nov 2021 09:29:35 GMT
    Content-Type: application/vnd.schemaregistry.v1+json
    Vary: Accept-Encoding, User-Agent
    Content-Length: 8

    {"id":4}

  • Use the API endpoint for managing schemas.

Did this answer your question?