Aiven for Apache Kafka has a number of predefined plans that specify the number of brokers and the capacity of individual brokers. The predefined plans consist of 3, 6, 9, 15, or 30 brokers, but we can also create larger custom plans based on your requirements.
To increase the capacity of an existing Kafka cluster, two options are available:
Vertically scaling the performance of individual brokers
Horizontally scaling the cluster by adding more brokers
Both scaling options are available for all Aiven for Apache Kafka customers and can be performed as an online operation, keeping your cluster up and running during the upgrade. The benefits of each option are described below.
Vertically scaling a Kafka cluster means replacing existing broker nodes with higher capacity nodes while keeping the same number of brokers. If you cannot increase the partition or topic count of your Kafka cluster due to application constraints, this is usually the only available option.
In practice, vertical scaling means changing your service plan from, for example, Aiven for Apache Kafka Business-4 to Aiven for Apache Kafka Business-8. When you change the service plan, Aiven automatically starts adding new brokers with the new specifications to your existing cluster. Once the new brokers are online and the data is replicated to them from the older nodes, the old brokers are retired one by one.
Horizontal scaling means adding more brokers to an existing Kafka cluster. This allows sharing the load in the cluster to a larger number of individual nodes, allowing the cluster to serve more requests as a whole. Horizontal scaling also makes the cluster more resilient to failure in a single node: if one broker in a 3-node cluster fails, the remaining two nodes get a 50% load increase, which may cause availability issues in the cluster. If one broker in a 9-node cluster fails, the remaining 8 nodes will only see a load increase of roughly 13%.
When you upgrade an Aiven for Apache Kafka cluster from, for example, the 3-node Business-8 plan to the Premium-6x-8 plan, Aiven immediately launches six new brokers and adds them to your existing cluster. The existing cluster nodes stay online, and once the new brokers are online and included in the cluster configuration, Kafka starts placing partition replicas on them, once the data is copied to the new brokers the old nodes are removed. Note that it may take some time until the cluster is fully balanced.
Our recommendation is to utilize both the vertical and horizontal scaling capabilities of Aiven for Apache Kafka to achieve the best possible performance and fault tolerance. For production clusters, we recommend a minimum of 6 cluster nodes to make sure that the failure of a single cluster node does not cause a sharp increase in load for the remaining cluster nodes.
Learn how Aiven simplifies working with Apache Kafka: