Service Integrations provide additional functionality and features by connecting different Aiven services together.
Metrics integration enables Aiven users to send advanced telemetry data to an Aiven for InfluxDB metrics database and visualize it in Aiven for Grafana. Telemetry data is currently supported for Aiven for Apache Kafka, Aiven for PostgreSQL, Aiven for MySql, Aiven for Cassandra, Aiven for ElasticSearch, Aiven for Redis, etc. This article uses Kafka as an example but the same functionality works for other services as well.
In addition to the metrics integration, also log integration is supported. The log integration allows sending logs from any Aiven service to Aiven for Elasticsearch and to search the logs using Kibana frontend.
Aiven automatically provides basic host-level resource metrics (CPU, memory, disk and network) for every service under the "Metrics" tab in the service view of the Aiven web console.
The advanced telemetry feature brings much more detailed, service-specific metrics to the Aiven user, who can then drill down into the service behavior and identify issues and performance bottlenecks.
It's very easy to start using the advanced telemetry integration by following the steps outlined below: let's start!
To get started, you will need three services:
Aiven for Apache Kafka - The service that produces the telemetry data
Aiven InfluxDB / PostgreSQL - Database where the telemetry data is stored and can be queried from
Aiven Grafana - Dashboards for the telemetry data
Note 1: Currently these services need to be running in the same Aiven project, but this limitation will be removed in the future and you will be able to, for example, connect multiple Kafka services from different projects to the same InfluxDB / PostgreSQL and Grafana instances.
Note 2: This example will use InfluxDB as the collection point for the telemetry data. The same steps apply if you want to use PostgreSQL.
Let's launch the three services, starting with Kafka:
Next, we will launch an InfluxDB service:
And finally, the Grafana service:
After a few minutes we have all three new services running:
Note: The Service Integration feature is only available for Startup, Business and Premium plans for each service (i.e. not available for the Hobbyist service tier).
Once the services are launched, we will open the InfluxDB service overview page and find the Service Integrations section within it:
Currently, the status is "No active service integrations" as we have not yet activated any. Click the Manage integrations button and a list of available integrations opens up:
Here you can see which service integrations are available for an InfluxDB service and what are enabled.
First we will enable the Grafana dashboard for our InfluxDB service. Simply clicking on the Grafana icon and selecting the correct service is enough.
Now we have our Grafana dashboard up and running. Next we will enable receiving metrics from our Kafka service.
Again, simply clicking the Kafka icon and selecting the correct service is enough.
That is all that is needed to get the advanced Kafka telemetry data flowing to the InfluxDB service! Close the integrations dialog by pressing the Close button.
Now, let's open the Grafana dashboard to see the data by going back to the service list and selecting the Grafana service:
Within the Connection parameters section, click on the password icon to see the password that is needed to login to Grafana.
Copy the password so that we can paste it to the Grafana login dialog later. We'll also make a note that the default user name for Grafana is "avnadmin".
Next, click the link next to the host to open up a browser page to the Grafana login dialog:
Enter "avnadmin" as the username and paste the Grafana password into the password field and login into Grafana.
The Grafana default view opens showing that we have one dashboard named "Aiven Kafka - demo-kafka - Resources" available:
Click the dashboard name "Aiven Kafka - demo-kafka - Resources" to open up the Kafka metrics dashboard:
This dashboard is a predefined view that is automatically maintained by Aiven.
Note that it may take a minute to start getting data into to the dashboard view if you just enabled the integrations. The view can be refreshed by pressing the reload button at the top-right corner.
You can add custom dashboards by either defining them from scratch in Grafana or by saving a copy of the predefined dashboard under a different name that does not start with "Aiven".
N.B. any changes that you make to the predefined dashboard will eventually be automatically overwritten by the system.
How are the service integrations billed? Does it cost something extra?
The advanced telemetry data and predefined dashboards do not cost anything extra, but they require you to have an InfluxDB and a Grafana service, which will be billed hourly as regular Aiven services.
Can I define my own metrics dashboards?
Yes. You can use the predefined dashboards as a starting point and save them under a different name or you can build one from scratch. Dashboards whose title starts with the word "Aiven" are automatically managed, so it is better to name the dashboards differently from that.
Can I access the telemetry data directly in InfluxDB?
Yes. The InfluxDB service is a normal InfluxDB database service that can be accessed via any InfluxDB client software or your own application or script. You can perform complex queries over the data and so on.
Can I add alerts in Grafana for the telemetry data?
Yes. You can add alert thresholds for individual graphs and attach different alerting mechanisms to them for sending out alerts. Please refer to the Grafana documentation for more information.
Got here by accident? Learn how Aiven simplifies working with Grafana: