Kafka

Getting started with Aiven for Apache Kafka

How to create a service and start using Aiven for Apache Kafka

Written by Hannu
Updated over a week ago

Querying Aiven for Apache Kafka with Lenses.io

Lenses is your DataOps solution for Kafka to support querying (with Lenses SQL), auditing and analysing your real-time data pipeline

Written by Christopher Gwilliams
Updated over a week ago

Kafka service user SSL certificate renewal

How Aiven renews SSL certificates for service users in Kafka services and how to take the new certificates into use

Written by Ivan
Updated over a week ago

What happens during a Kafka cluster upgrade?

Aiven for Apache Kafka upgrade procedure, downtime, duration, rolling back, impacts and risks

Written by Rob
Updated over a week ago

Aiven for Apache Kafka versions and upgrade instructions

Guidance on upgrading Kafka to the latest supported release

Written by Soumya
Updated over a week ago

Obtaining partition details about Kafka topics

Using the UI, CLI and API to obtain partition details about Kafka topics

Written by Craig Crawford
Updated over a week ago

Migrating to Aiven for Apache Kafka

How to migrate your Kafka instance to Aiven

Written by Jani Asikanius
Updated over a week ago

Kafka authentication types

What are SASL and SSL, and how to select the type of secure connection for your Apache Kafka service

Written by Christopher Gwilliams
Updated over a week ago

Managed Kafka POC checklist

Use this checklist to get the most out of your Aiven for Apache Kafka test

Written by David Esposito
Updated over a week ago

Kafka Streams with Aiven for Kafka

How to setup a Kafka Streams application with Aiven for Kafka and Schema Registry

Written by Jani Asikanius
Updated over a week ago

Python examples for testing Aiven for Apache Kafka

How to set up a producer and consumer in Python to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Node.js examples for testing Aiven for Apache Kafka

How to set up a producer and consumer in Node.js to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Java examples for testing Aiven for Apache Kafka

How to set up a producer and consumer in Java to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Kafka console tool examples for testing Aiven for Apache Kafka

How to use the Kafka console producer and consumer tools to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Ruby examples for testing Aiven for Apache Kafka

How to set up a producer and consumer in Ruby to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Go examples for testing Aiven for Apache Kafka

How to set up a producer in Go to test your Aiven for Apache Kafka service

Written by Mikko Röntynen
Updated over a week ago

Using Apache Flink with Aiven for Apache Kafka

How to set up Apache Flink with Aiven for Apache Kafka

Written by George
Updated over a week ago

Viewing and resetting consumer group offsets

How to use the kafka-consumer-groups.sh tool

Written by Heikki
Updated over a week ago

Kafka cluster upgrades

How to scale your Aiven for Apache Kafka cluster to suit your needs

Written by Oskari
Updated over a week ago

Consumer group graphs for Kafka dashboards

What Kafka consumer group telemetry and graphs are available in Aiven for Apache Kafka

Written by Jani Asikanius
Updated over a week ago

Kafka metrics through Prometheus integration

What metrics are available in Aiven for Apache Kafka

E

Written by Elina Finnilä
Updated over a week ago

Kafka NOT_LEADER_FOR_PARTITION errors

Why this error message appears in your logs and why it not cause for concern

Written by Sami
Updated over a week ago

Zookeeper access

Why we do not allow access to Zookeeper in Aiven for Apache Kafka services

Written by Sami
Updated over a week ago

Setting up a Confluent Docker Image for Schema Registry with Aiven for Kafka

How to setup and deploy a Confluent Docker Image for Schema Registry with your Aiven for Apache Kafka service

Written by Sebin
Updated over a week ago

Removing Kafka Topic Prefix when replicating with Aiven for Mirrormaker 2

Configure your Replication Flows to either prefix or not when replicating data between Kafka clusters using Aiven for Mirrormaker 2

Written by Christopher Gwilliams
Updated over a week ago

Configuring Java SSL to access Kafka

How to create the Java keystore and truststore that are required for accessing Aiven for Apache Kafka from a Java client

Written by Ervin Beta
Updated over a week ago

Missing Kafka Datadog metrics

Why the Kafka producer and consumer metrics that are described in the Datadog documentation are not available in Aiven for Apache Kafka

Written by Sami
Updated over a week ago

Kafka OutOfMemoryError exceptions

How to handle nodes in a Kafka or Kafka Connect cluster that are running low on memory and prompt OutOfMemoryError exceptions

Written by Thomas
Updated over a week ago

Best practices for Aiven for Apache Kafka

Tips on how to get the most out of your Kafka and Kafka Connect services

Written by Sami
Updated over a week ago

Kafka inter-broker protocol and default message format version

How the Kafka inter.broker.protocol.version and log.message.format.version are updated

Written by Sami
Updated over a week ago

Public access for Kafka in a VPC

How to enable public access to Aiven for Apache Kafka (Kafka, Connect and Rest) services running in a virtual private cloud (VPC)

Written by Christopher Gwilliams
Updated over a week ago

Configuring Aiven for Apache Kafka using Kafka CLI tools

How to use Kafka command-line interface (CLI) tools to edit configurations not exposed by the Aiven API or web console

Written by Christopher Gwilliams
Updated over a week ago

Topic Log Compaction

How to enable log compaction and how it works

Written by George
Updated over a week ago

Adding Kafka users and managing access control

Written by Heikki
Updated over a week ago

Kafka data retention

How to use log compaction and different data retention periods

Written by Oskari
Updated over a week ago

Creating Kafka topics automatically

How to enable the automatic creation of topics in Aiven for Apache Kafka and adjust the default properties for topics

Written by Mathias
Updated over a week ago

Setting Kafka parameters

How to configure parameters in the Aiven for Apache Kafka command-line client

Written by Mika
Updated over a week ago

Programmatic topic management

How to use the Java AdminClient interface to create and modify topics

Written by Heikki
Updated over a week ago

Disk usage in Kafka topic indexes

Why empty topics may use up over 20 MiB of disk space

Written by Mika
Updated over a week ago

Preventing full Kafka disks

What to do when the disk gets full in Aiven for Apache Kafka

Written by Mathias
Updated over a week ago

Bring your own Kafka connect cluster

How to integrate your own Kafka connect cluster with Aiven for Apache Kafka

Written by Jani Asikanius
Updated over a week ago

Configuring Datadog metrics with Aiven for Apache Kafka

How to send custom Kafka metrics to Datadog and filter them by topics or consumer groups

Written by Mika
Updated over a week ago

Setting up a JDBC sink connector with Aiven for Apache Kafka

How to set up and use a Java Database Connectivity (JDBC) sink connector with your Aiven for Apache Kafka service

Written by Jani Asikanius
Updated over a week ago

Kafka Tools

Additional Tools for Aiven for Apache Kafka

E

Written by Elina Finnilä
Updated over a week ago

Using Kafdrop Web UI with Aiven for Apache Kafka

Written by Christopher Gwilliams
Updated over a week ago

End of support for Confluent Schema Registry on Aiven

Aiven is switching to Karapace, an open-source implementation of Kafka REST and Schema Registry.

Written by Soumya
Updated over a week ago

Apache Kafka data visualization with Operatr

Steps to integrating Operatr with Aiven for Apache Kafka

Written by Jani Asikanius
Updated over a week ago

Setting up a Google BigQuery sink connector with Aiven for Apache Kafka

How to set up and use a Google BigQuery sink connector with your Aiven for Apache Kafka service

Written by Jani Asikanius
Updated over a week ago

Getting started with Aiven for Apache Kafka MirrorMaker 2

How to replicate data between Aiven-hosted Kafka clusters using Kafka MirrorMaker 2

Written by Ivan
Updated over a week ago

Migrating BigQuery Kafka connector from 1.2.0 to 1.6.5 (or later)

Dealing with the configuration breaking change

Written by Ivan
Updated over a week ago

Using kafkacat

Kafkacat is an useful utility for inspecting and producing messages into Kafka on the commandline.

Written by Heikki
Updated over a week ago

Kafkatool integration with Aiven for Apache Kafka

How to integrate Kafkatool with Aiven for Kafka

Written by Jani Asikanius
Updated over a week ago

Using Schema Registry with Aiven for Apache Kafka

Example on how to use Kafka Schema Registry available in Aiven for Apache Kafka

Written by Jani Asikanius
Updated over a week ago

Configuring Kafka REST

Written by Timo
Updated over a week ago

Is Kafka REST's performance at the same level as native Kafka connections?

Written by Mika
Updated over a week ago

Using KSQL with Aiven for Apache Kafka

How to run KSQL against Aiven hosted Kafka and Schema Registry

Written by Rauli
Updated over a week ago

Aiven for Apache Kafka Connect - connector list not currently available

What to do when you are seeing "connector list not currently available" in the Connectors tab in the web console

Written by Rob
Updated over a week ago

Kafka connect: Auto restart on failures

Written by Aiqin
Updated over a week ago

AWS IAM Assume Role credentials provider in Aiven S3 Sink Kafka Connector

Written by Olena Babenko
Updated over a week ago

Setting the logging level of an Aiven for Apache Kafka Connect cluster

Help debug issues in an Aiven for Apache Kafka Connect cluster by setting the logging level to be more verbose

Written by Rob
Updated over a week ago

Kafka Connect Connectors

Aiven supported connectors

Written by Hannu
Updated over a week ago

JDBC source connector with PostgreSQL

How to setup and use JDBC source connector for Kafka with PostgreSQL

Written by Jani Asikanius
Updated over a week ago

JDBC source connector with MySQL

How to setup and use JDBC source connector for Kafka with MySQL

Written by Jani Asikanius
Updated over a week ago

Setting up a JDBC sink connector with Aiven for Apache Kafka

How to set up and use Aiven's JDBC Sink connector with your Aiven for Apache Kafka service

Written by Sebin
Updated over a week ago

Using Debezium connector with RDS PostgreSQL and Aiven for Apache Kafka

How to use Debezium connector for change data capture (CDC) with RDS PostgreSQL and Aiven for Kafka

Written by Jani Asikanius
Updated over a week ago

Aiven for Apache Kafka - S3 sink connector by Aiven

How to setup and use the S3 sink connector by Aiven

Written by Jani Asikanius
Updated over a week ago

Aiven for Apache Kafka - GCS sink connector by Aiven

How to setup and use Kafka GCS sink connector

Written by Ivan
Updated over a week ago

Aiven for Apache Kafka - S3 sink connector by Confluent

How to set up and use the S3 sink connector by Confluent

Written by Heikki
Updated over a week ago

Aiven for Apache Kafka - Elasticsearch sink connector

How to setup and use Elasticsearch sink connector with Aiven for Apache Kafka

Written by Jani Asikanius
Updated over a week ago

Setting up Cassandra sink connector for Kafka

Consuming data from Kafka topic and storing in Cassandra

Written by Jani Asikanius
Updated over a week ago

Setting up Debezium with Aiven for MySQL

Get your data flowing from MySQL to Kafka

Written by Hannu
Updated over a week ago

Setting up Debezium with Aiven for PostgreSQL

Get your data flowing from PostgreSQL to Kafka

Written by Hannu
Updated over a week ago

Setting up Debezium for MongoDB

How to push data from MongoDB to Aiven for Apache Kafka using Debezium

Written by Jani Asikanius
Updated over a week ago

Kafka Connect: JDBC Source with SQL Server

Example configuration for SQL Server JDBC source

Written by Heikki
Updated over a week ago