Automatic connector task restart is not something that could be recommended for most of the cases.
However, sometimes a task can fail due to a rare problem like out of memory on a huge surge of data. Or we have observed a few cases where kafka connector stopped working after pg maintenance update, because of limitation in Postgress -i.e the lack of support to create replication slots before failover. This “failover slot” feature is not yet available on Postgre12. Though, there is active development for PostgreSQL 13 to support logical decoding on standbys, which is a major requirement to make failover possible.
Now we provide an option to turn the restarts on per connector basis just in case of edge situations. Here, take the creation of a new Debezium-PostgreSQL connector as an example:
- Navigate to your kafka connect connectors tab
- Select the Aiven tab
- Choose the Automatic restart to be True
Now if failure has been detected by our management platform, the connector with this Automatic restart setting to True will be restarted after a period of time.
- Service restarts not only failed tasks but connector as well. It depends on connector, e.g. Debezium behaves this way, but JDBC connector just fails.
- This feature applicable to all connectors.