Service Integrations provide additional functionality and features by connecting different Aiven services together.
Metrics integration enables Aiven users to send advanced telemetry data to an Aiven InfluxDB metrics database and visualize it in Aiven Grafana. Telemetry data is currently supported for Aiven Kafka and PostgreSQL. This article uses Kafka as an example but the same functionality works for PostgreSQL as well.
In addition to the metrics integration, also log integration is supported. The log integration allows sending logs from any Aiven service to Aiven Elasticsearch and to search the logs using Kibana frontend.
Aiven automatically provides basic host-level resource metrics (CPU, memory, disk and network) for every service under the "Metrics" tab in the service view of the Aiven web console.
The advanced telemetry feature brings much more detailed, service-specific metrics to the Aiven user, who can then drill down into the service behavior and identify issues and performance bottlenecks.
It's very easy to start using the advanced telemetry integration by following the steps outlined below: let's start!
To get started, you will need three services:
- Aiven Kafka - The service that produces the telemetry data
- Aiven InfluxDB - Database where the telemetry data is stored and can be queried from
- Aiven Grafana - Dashboards for the telemetry data
Note: Currently these services need to be running in the same Aiven project, but this limitation will be removed in the future and you will be able to, for example, connect multiple Kafka services from different projects to the same InfluxDB and Grafana instances.
Let's launch the three services, starting with Kafka:
Next, we will launch an InfluxDB service:
And finally, the Grafana service:
After a few minutes we have all three new services running:
Note: The Service Integration feature is only available for Startup, Business and Premium plans for each service (i.e. not available for the Hobbyist service tier).
Once the services are launched, we will open the InfluxDB service overview page and find the Service Integrations section within it:
Currently, the status is "No active service integrations" as we have not yet activated any. Click the Manage... button and a list of available integrations opens up:
The list shows two ways of connecting the InfluxDB service to other services. One for pulling metrics from a Kafka service and another for displaying the metrics in a dashboard in the Grafana service.
Simply clicking the checkboxes next to the integrations will turn them on:
That is all that is needed to get the advanced Kafka telemetry data flowing to the InfluxDB service! Close the integrations dialog by pressing the Close button.
Now, let's open the Grafana dashboard to see the data by going back to the service list and selecting the Grafana service:
Within the Connection parameters section, click on the password CLICK_TO_REVEAL text to see the password that is needed to login to Grafana.
Copy the password so that we can paste it to the Grafana login dialog later. We'll also make a note that the default user name for Grafana is "avnadmin".
Next, click the link next to the host to open up a browser page to the Grafana login dialog:
Enter "avnadmin" as the username and paste the Grafana password into the password field and login into Grafana.
The Grafana default view opens showing that we have one dashboard named "Aiven Kafka - demo-kafka - Resources" available:
Click the dashboard name "Aiven Kafka - demo-kafka - Resources" to open up the Kafka metrics dashboard:
This dashboard is a predefined view that is automatically maintained by Aiven.
Note that it may take a minute to start getting data into to the dashboard view if you just enabled the integrations. The view can be refreshed by pressing the reload button at the top-right corner.
You can add custom dashboards by either defining them from scratch in Grafana or by saving a copy of the predefined dashboard under a different name that does not start with "Aiven".
N.B. any changes that you make to the predefined dashboard will eventually be automatically overwritten by the system.
How are the service integrations billed? Does it cost something extra?
The advanced telemetry data and predefined dashboards do not cost anything extra, but they require you to have an InfluxDB and a Grafana service, which will be billed hourly as regular Aiven services.
Can I define my own metrics dashboards?
Yes. You can use the predefined dashboards as a starting point and save them under a different name or you can build one from scratch. Dashboards whose title starts with the word "Aiven" are automatically managed, so it is better to name the dashboards differently from that.
Can I access the telemetry data directly in InfluxDB?
Yes. The InfluxDB service is a normal InfluxDB database service that can be accessed via any InfluxDB client software or your own application or script. You can perform complex queries over the data and so on.
Can I get similar advanced telemetry data for PostgreSQL, Elasticsearch, etc.?
The same functionality is available also for PostgreSQL. We will be adding more services to the metrics integration in the future.
Can I add alerts in Grafana for the telemetry data?
Yes. You can add alert thresholds for individual graphs and attach different alerting mechanisms to them for sending out alerts. Please refer to the Grafana documentation for more information.