When you upgrade to a new major version of Kafka, one of the steps outlined in the upgrade procedure is to modify the inter.broker.protocol.version and log.message.format.version configuration values.
While Aiven for Apache Kafka takes care of the upgrade for you, you may notice that these values remain unchanged immediately following an upgrade.
As an example, consider a service running Kafka 2.5.
When you upgrade this service to a new version of Kafka, new nodes join the cluster and topics are synchronized from the old nodes until the old ones can be removed. For the duration of the upgrade, the cluster contains a mixture of nodes running Kafka 2.5 and 2.6.
To maintain compatibility, Aiven makes sure that the inter.broker.protocol.version and log.message.format.version values are set to the minimum Kafka version present in the cluster.
This means that following the upgrade, these configuration values remain equal to the previous version of Kafka running in the cluster. Although you could safely increase the inter.broker.protocol.version value after the upgrade, you cannot increase the log.message.format.version because there are old messages in the old format on disk.
This is usually not a problem thanks to Kafka's bidirectional client compatibility unless you are looking to use new features allowed by a higher log.message.format.version. The solution here is another rolling update, which you can trigger by applying maintenance updates or upgrading or downgrading the plan.
When you trigger a maintenance update or upgrade after completing a major version upgrade, the new nodes that join the cluster only contain the nodes running Kafka 2.6.
The minimum Kafka version present is now also 2.6 and the inter.broker.protocol.version and log.message.format.version values are increased. This means that the messages of topics are finally created on the new nodes with the latest message format.
If you have any questions about your Aiven for Apache Kafka service, contact our support.