Note: Aiven for OpenSearch replaces Aiven for Elasticsearch after Elasticsearch version 7.10.2 due to licensing changes. In addition, OpenSearch Dashboards replaces Kibana. Starting from September 24th, 2021 until March 23rd, 2022, Aiven supports services running both OpenSearch and Elasticsearch 7.10.2. For more information on Aiven for OpenSearch, see our developer documentation.

Logs are extremely important for understanding the health of your application as well as diagnosing problems that occur. This is ever important with the adoption of microservices and the usage of Kubernetes.

With hundreds, or maybe even thousands, of pods creating logs on dozens of nodes it is tedious if not impossible to install a log capturing agent on each pod for each different type of service.

Luckily, Kubernetes provides an abstraction that allows us to achieve what we want without knowing what is running on each pod or manually including agents on each pod: Daemonset. Briefly, a Daemonset, allows the scheduling of pods on some or all nodes based upon a user defined criteria.

Starting from scratch, in this tutorial I will walk you through the setup of a Kubernetes cluster, creating a basic application, and pushing those logs to an Aiven Elasticsearch cluster.

Getting the Dependencies

All of the code for this tutorial can be found at: https://github.com/aiven/k8s-logging-demo.

Let's start by cloning the repo:

git clone https://github.com/aiven/k8s-logging-demo.git

There are several local dependencies we will need including: Docker, Minikube, Kubectl, and Helm. The installation of each of these tools is slightly different depending on your operating system, so please follow the links above to install them.

Creating Our Cluster

Creating a Kubernetes cluster with Minikube is simple

minikube start

You can verify that your cluster is up and running by

kubectl get pods --all-namespaces

And you should see something like

We will be using a non default namespace so let's create that now

kubectl create namespace logging

Putting Stuff in Our Cluster

Now that we have a nice little Kubernetes cluster, let's go ahead and do something with it. We are going to build an API locally into a Docker image and then push that, along with FluentD to our cluster.

The API is a simple Python service using FastAPI.

FluentD is a data sourcing, aggregating and forwarding client that has hundreds of plugins to support all kinds of sources, transformations and outputs.

To build the API:

cd k8s-logging-demo
docker build -t local/log-demo .

We are going to install FluentD using a pre-built Helm Chart and in order to do that we have to add the repo and update the dependency.

cd k8s-logging-demo 
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm dependency update chart

We also need to tell Minikube about the proper Docker daemon to use. If we do not do this, Minikube will not know where to find the API image we just built.

eval $(minikube -p minikube docker-env)

The last part of the equation is getting an external store for our logs. To do this we will create an Elasticsearch instance in Aiven. Go ahead and create a free account, at which point you will be given $300 worth of free credits.

Create a new project in which to run Elasticsearch

Click Create a new service and select Elasticsearch. Then select the cloud provider and region of your choice. In the final step we choose the service plan -- in this case we will use a Hobbyist plan. A default name will be provided, but it is best to change it to something identifiable as it cannot be renamed later.

After a minute or so your Elasticsearch service will be ready to use. You can view all the connection information in the console by clicking on the service that you created.

Take note of the Host, Port, User and Password as we will need these to configure our Helm Chart.

We are now ready to deploy our Helm Chart--i.e.:

port=24947
host="https://my-test-my-project.aivencloud.com:$port"
user=avnadmin
password=123456789xx

helm install -n logging log-demo chart \
--set elasticsearch.hosts=$host \
--set elasticsearch.user=$user \
--set elasticsearch.pw=$password

Note that as seen in above code block the <ES Host> should be the concatenation of the host and port we captured from the Aiven Console. In my case it looks like https://something-unique-sa-demo.aivencloud.com:24947. The values set here are only a subset of the configurations for FluentD, the full set are detailed in the chart definition

You can check that things are building correctly by investigating the pods in the logging namespace

kubectl -n logging get pods

You should see something like

If all the pods aren't ready yet, just give it a few seconds and check again. Assuming the connection details you entered are correct, then they should become ready shortly.

Searching and Viewing Logs

The configuration that has been deployed captures all logs from every node (can be configured not to do this) and so if we head over to Kibana we should see that happening. Aiven automatically deploys Kibana alongside Elasticsearch and the connection info can also be found in the console.

Once logged into Kibana go to dev tools

And issue the following query, and you should see results

GET /_search
{
"query": {
"term": {
"kubernetes.namespace_name.keyword": {
"value": "kube-system"
}
}
}
}

Now lets utilize the API we built and see if we can see any logs from there. In a separate terminal, port forward the API service

kubectl port-forward -n logging service/api-service 8080:8080

Then we can issue a curl request to the API. The API only has one endpoint, an echo endpoint, which echos back what you send but also logs it as a warning message.

curl -X POST localhost:8080/echo -d'{"key": "HOLY MOLY"}'

Going back to Kibana, let's issue another request and see if we can find the log for the request we just made:

GET /_search 
{
"query": {
"match": {
"log": {
"query": "holy"
}
}
}
}

Most likely there will be several results, but the first one should be the log related to the API request we just issued.

Updates and Destruction

If at any point you want to make changes to any of the deployments, e.g. change the FluentD configuration, add an endpoint to the existing service or add a whole new service

helm upgrade -n logging log-demo <other parameters set during install>

Be aware that you may need to redeploy pods for changes to take effect. In order to tear down the installation

helm delete -n logging log-demo


Final Thoughts

This guide started from nothing and created a Kubernetes application with an observability layer.

The code and steps here could easily be expanded upon to use a different Kubernetes provider such as GKE or EKS and the Helm configuration could easily be expanded to include other use cases such as sending data to Kafka or capturing metrics as well.

Regardless of where the data comes from or where it goes or what kind it is, the Aiven platform has the tools and services to assist you on your journey.

Did this answer your question?