Kafka

Querying Aiven Kafka with Lenses.io

Lenses is your DataOps solution for Aiven Kafka to support querying (with Lenses SQL), auditing and analysing your real-time data pipeline

Written by Christopher Gwilliams
Updated over a week ago

EOL for Aiven for Kafka v1.x, v2.0, v2.1, v2.2

Guidance on upgrading from old versions of Kafka reaching EOL.

Written by Soumya
Updated over a week ago

Getting started with Aiven for Apache Kafka

Written by Hannu
Updated over a week ago

Migrating your Kafka to Aiven

How to migrate your Kafka to Aiven

Written by Prem
Updated over a week ago

Managed Kafka POC Checklist

Use this checklist to get the most out of your Aiven for Apache Kafka test

Written by David Esposito
Updated over a week ago

Kafka Authentication Types

Know your SASL from your SSL and remove the fear of using Secure Connections

Written by Christopher Gwilliams
Updated over a week ago

Kafka Streams with Aiven for Kafka

How to setup a Kafka Streams application with Aiven for Kafka and Schema Registry

Written by Prem
Updated over a week ago

Using Apache Flink with Aiven Kafka

How to set up Apache Flink with Aiven Kafka

Written by George
Updated over a week ago

Viewing and Resetting Consumer Group Offsets

Usage instructions for kafka-consumer-groups.sh tool

Written by Heikki
Updated over a week ago

Kafka cluster upgrades

Scaling up your Kafka cluster

Written by Oskari
Updated over a week ago

Consumer group graph addition to Kafka dashboard

Kafka consumer group telemetry and graphs

Written by Prem
Updated over a week ago

Kafka metrics through Prometheus integration

All available metrics in Kafka

Written by Markus
Updated over a week ago

Kafka NOT_LEADER_FOR_PARTITION error

Why there is no need to worry when seeing this message in your logs

Written by Charlie
Updated over a week ago

Missing Kafka Datadog Metrics

Where are the Kafka producer and consumer metrics described in the Datadog documentation?

Written by Charlie
Updated over a week ago

Kafka OutOfMemoryError

Nodes in a Kafka or Kafka Connect cluster running low on memory will start throwing OutOfMemoryError exceptions

Written by Thomas
Updated over a week ago

Basic Kafka best practices

Tips on how to get the most out of your Kafka and Kafka Connect services

Written by Charlie
Updated over a week ago

Kafka inter broker protocol and default message format version

How are the Kafka inter.broker.protocol.version and log.message.format.version updated?

Written by Charlie
Updated over a week ago

Public Access for Kafka in a VPC

Enabling Public Access to Kafka (Kafka, Connect and Rest) Services Running in a VPC

Written by Christopher Gwilliams
Updated over a week ago

Configuring Aiven Kafka using Kafka CLI tools

Using Kafka CLI tools to edit configurations not exposed by the Aiven API/Console. For example, setting topic retention to less than 1 hour

Written by Christopher Gwilliams
Updated over a week ago

Topic Log Compaction

How to enable log compaction and how it works

Written by George
Updated over a week ago

Kafka Users and Access Control

Written by Heikki
Updated over a week ago

Kafka data retention

Utilizing log compaction and different data retention periods

Written by Oskari
Updated over a week ago

Kafka topic auto-create

This help article explains how to enable topic auto-create in Apache Kafka and adjust default properties for topics

Written by Mathias
Updated over a week ago

Setting max.message.bytes and other Kafka parameters

How to configure Aiven Kafka parameters?

Written by Mika
Updated over a week ago

Programmatic topic management

Using Java AdminClient interface to create and modify topics

Written by Heikki
Updated over a week ago

Kafka topic index disk usage

Why are empty topics using over 20 MiB disk space?

Written by Mika
Updated over a week ago

Kafka disk full prevention

What to do when the disk gets full in an Aiven Kafka

Written by Mathias
Updated over a week ago

Bring your own Kafka connect cluster

How to integrate your own Kafka connect cluster with Aiven for Kafka

Written by Prem
Updated over a week ago

How to configure Aiven Kafka Datadog metrics

Sending custom Kafka metrics to Datadog and filtering by topics/consumer groups

Written by Mika
Updated over a week ago

Setting up JDBC sink connector with Aiven for Kafka

How to setup and use JDBC sink connector with Aiven for Kafka

Written by Prem
Updated over a week ago

Kafka Tools

Additional Tools for Aiven Kafka

Written by Markus
Updated over a week ago

Using Kafdrop Web UI with Aiven Kafka

Written by Christopher Gwilliams
Updated yesterday

Kafka data visualization with Operatr

Steps to integrating Operatr with Aiven for Kafka

Written by Prem
Updated over a week ago

Setting up Google BigQuery sink connector with Aiven for Kafka

How to setup and use Google BigQuery sink connector with Aiven for Kafka

Written by Prem
Updated over a week ago

Getting started with Aiven for Apache Kafka MirrorMaker 2

How to replicate data between Aiven-hosted Kafka clusters using Kafka MirrorMaker 2

Written by Ivan
Updated over a week ago

Migrating BigQuery Kafka connector from 1.2.0 to 1.6.5 (or later)

Dealing with the configuration breaking change

Written by Ivan
Updated over a week ago

Using kafkacat

Kafkacat is an useful utility for inspecting and producing messages into Kafka on the commandline.

Written by Heikki
Updated over a week ago

Kafkatool integration with Aiven for Kafka

How to integrate Kafkatool with Aiven for Kafka

Written by Prem
Updated over a week ago

Using Schema Registry with Aiven Kafka

Example on how to use Kafka Schema Registry available in Aiven Kafka

Written by Prem
Updated over a week ago

Kafka MirrorMaker use within Aiven

Using Kafka MirrorMaker integration in Aiven to copy data across two Kafka clusters

Written by Ville
Updated over a week ago

Configuring Kafka REST

Written by Timo
Updated over a week ago

Is Kafka REST's performance at the same level as native Kafka connections?

Written by Mika
Updated over a week ago

Using KSQL with Aiven Kafka

How to run KSQL against Aiven hosted Kafka and Schema Registry

Written by Rauli
Updated over a week ago

Kafka connect: Auto restart on failures

Written by Aiqin
Updated over a week ago

AWS IAM Assume Role credentials provider in Aiven S3 Sink Kafka Connector

Written by Olena Babenko
Updated over a week ago

Setting the logging level of an Aiven Kafka Connect cluster

Help debug issues in an Aiven Kafka Connect cluster by setting the logging level to be more verbose

Written by Viet
Updated over a week ago

Kafka Connect Connectors

Aiven supported connectors

Written by Hannu
Updated over a week ago

JDBC source connector with PostgreSQL

How to setup and use JDBC source connector for Kafka with PostgreSQL

Written by Prem
Updated over a week ago

JDBC source connector with MySQL

How to setup and use JDBC source connector for Kafka with MySQL

Written by Prem
Updated over a week ago

Using Debezium connector with RDS PostgreSQL and Aiven Kafka

How to use Debezium connector for change data capture (CDC) with RDS PostgreSQL and Aiven Kafka

Written by Prem
Updated over a week ago

Aiven Kafka - S3 sink connector by Aiven

How to setup and use the S3 sink connector by Aiven

Written by Prem
Updated over a week ago

Aiven Kafka GCS sink connector

How to setup and use Aiven Kafka GCS sink connector

Written by Ivan
Updated over a week ago

Aiven Kafka - S3 sink connector by Confluent

How to setup and use the S3 sink connector by Confluent

Written by Heikki
Updated over a week ago

Aiven Kafka - Elasticsearch sink connector

How to setup and use Elasticsearch sink connector with Aiven for Kafka

Written by Prem
Updated over a week ago

Setting up Cassandra sink connector for Kafka

Consuming data from Kafka topic and storing in Cassandra

Written by Prem
Updated over a week ago

Setting up Debezium with Aiven for MySQL

Get your data flowing from MySQL to Kafka

Written by Hannu
Updated over a week ago

Setting up Debezium with Aiven for PostgreSQL

Get your data flowing from PostgreSQL to Kafka

Written by Hannu
Updated over a week ago

Setting up Debezium for MongoDB

How to push data from MongoDB to Aiven for Kafka using Debezium

Written by Prem
Updated over a week ago

Kafka Connect: JDBC Source with SQL Server

Example configuration for SQL Server JDBC source

Written by Heikki
Updated over a week ago