During operation of an Aiven Kafka Connect cluster, we may encounter errors from one or more running connectors. Sometimes the stack trace printed in the logs is useful in determining the root cause of an issue. At other times, the information provided just isn't enough to work with, and we would want access to more detailed logs to debug an issue. This can be done by setting the logging level of an Kafka Connect's cluster. The logging levels are typically set to DEBUG or TRACE level.

The Kafka Connect API allows changing the logging level of a specific logger or connector. How we would call the API depends on the number of nodes of our Kafka Connect and the number of tasks of a connector we wish to set the logging level for. This is because, the API only changes logging on the worker that’s accessed, not across an entire distributed Connect cluster (source). Therefore, for a multi-node cluster, we would have to change the logging level in all of the worker nodes that run the connector's tasks that we are debugging.

For a dedicated Kafka Connect cluster using a startup plan, there is 1 worker node, while a business and premium plan has 3 and 6 nodes respectively.

Here are the steps to change the logging level of a connector running in an Aiven Kafka Connect cluster:

1. Determine the plan of the Aiven Kafka Connect cluster

For startup plans, skip step 2 and use the hostname of the cluster in steps 3. For business and premium plans, go to step 2.

2. Determine the IP address of each worker node

When we connect to the cluster by the hostname, it points to one of the three or six worker node IP addresses in a random fashion due to the DNS load balancing in place. Therefore, we would have to get the IP address of each worker node in order to change the logging level for the loggers/connectors in each of them. We can use Linux' host command to the the list of the IP addresses. For example:

$ host some-host.aivencloud.com

would yield something like:

some-host.aivencloud.com has address 192.168.0.1
some-host.aivencloud.com has address 192.168.0.3
some-host.aivencloud.com has address 192.168.0.6

3. Changing the logging level by calling the Kafka Connect API on each worker node

In these examples we will use curl. To get the list of loggers and connectors and their current logging level on a worker node, you can run:

$ curl -u avnadmin:password https://192.168.0.1:443/admin/loggers | jq (jq is to display the output as json)

You can either force curl to resolve the domain name to a particular IP with --resolve option:

$ curl -X PUT -H "Content-Type:application/json" -u avnadmin:password -d '{"level": "TRACE"}' https://some-host.aivencloud.com:443 --resolve some-host.aivencloud.com:443:192.168.0.1/admin/loggers/LOGGER

Or, alternatively, you can use the IP instead of the domain name and disable the certificate check (the certificate is valid for the domain name, but not for a bare IP):

$ curl -X PUT -H "Content-Type:application/json" -u avnadmin:password -d '{"level": "TRACE"}' https://192.168.0.1:443/admin/loggers/LOGGER --insecure

Note: When the worker is restarted, logging reverts back to using the logging properties defined in the log4j configuration file. In an Aiven Kafka Connect cluster the default logging level is INFO.

References

https://docs.confluent.io/5.5.0/connect/managing/monitoring.html

https://docs.confluent.io/platform/current/connect/logging.html

Did this answer your question?