When upgrading to a new major version of Kafka one of the steps outlined in the upgrade procedure is to modify the inter.broker.protocol.version and log.message.format.version configuration values.
Fortunately Aiven takes care of the upgrade for you, but you may notice that immediately following an upgrade these values remain unchanged.
Consider a topic created on a service running Kafka 2.5.
When you upgrade the service to a new version of Kafka, new nodes join the cluster and topics are synced from the old nodes until they can be removed entirely. For the duration of the upgrade the cluster contains a mixture of nodes running Kafka 2.5 and 2.6.
To maintain compatibility, Aiven ensures the inter.broker.protocol.version and log.message.format.version values are set to the minimum Kafka version present in the cluster.
This means that following the upgrade these configuration values remain equal to the previous version of Kafka running in the cluster. Although it would be safe to increase the inter.broker.protocol.version after the upgrade, it is not safe to increase the log.message.format.version because there will be old messages in the old format on disk.
This is not usually a problem thanks to Kafka's bidirectional client compatibility unless you are looking to take advantage of new features allowed by a higher log.message.format.version. The solution here is another rolling update which is usually triggered by applying maintenance updates, or upgrading or downgrading the plan.
Therefore when you trigger a maintenance update/upgrade after completing a major version upgrade, then the new nodes that join the cluster only contain the nodes running Kafka 2.6.
The minimum Kafka version present is now also 2.6 and the inter.broker.protocol.version and log.message.format.version values are increased. This means that the messages of topics are finally created on the new nodes with the latest message format.
If you have any questions, please feel free to reach out to our Support and let us know.