Logs

How to make Filebeat work on Kubernetes with AWS ElasticSearch

Ok, so this is one of those days, when you say to yourself - I need to get logs from my Kubernets cluster somewhere, where I can inspect them. You have a lot of options, and I would like to share my experience on one of them: how to get logs from your Kubernetes cluster pods to AWS ElasticSearch cluster using Filebeat.

Software versions

  1. AWS ElasticSearch cluster. Will use latest available now — v7.8
  2. Kubernetes cluster, will use AWS EKS v1.18.
  3. Download Filebeat installation v7.9.3 to your PC (choose your platform). Why this version ? Filebeat v7.10.0 index template is not compatible with AWS ES v7.8 and Filebeat v7.8.1 was not working correctly for me on the K8S cluster.

AWS ElasticSearch cluster preparation

You might ask yourself, why do I need to do it? Filebeat is a product of Elastic and they should work well together. This is correct, with one small exception, which is very important in our case. AWS ElasticSearch is using ElasticSearch OpenDistro and not vanilla ElasticSearch. Why this should matter to us? Well, OpenDistro doesn’t have ILM (Index Lifecycle Management), what it has instead is ISM (Index State Management). And we will need to prepare Filebeat index template to work with ISM and not ILM (which it does out of the box). If you ask, why I need ISM? With it you can have very important feature — index rotations and old index deletion, so you ES cluster will not get out of free space.

So lets do it, lets add our ISM policy to ES, which will handle Filebeat indexes lifecycle for us. You can find more information about ISM policies and example of one ISM policy here. Let’s apply one. We name that policy delete_old

curl -XPUT -H 'Content-Type: application/json' \
http://your-cluster.eu-west-1.es.amazonaws.com/_opendistro/_ism/policies/delete_old -d@policy.json

Check that this policy is available in your AWS ES Kibana.

Next we need to add template for Filebeat indexes to ES. You can get vanilla one from Filebeat, which you will have to amend later on:

curl -LO https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.9.3-darwin-x86_64.tar.gz
tar -zxvf filebeat-7.9.3-darwin-x86_64.tar.gz
cd filebeat-7.9.3-darwin-x86_64
./filebeat export template > filebeat-7.9.3.json

Let’s amend this template, remove ILM and add ISM and make sure we don’t use flattened data type as AWS ES v7.8 doesn’t support it yet. Fixed template can be found here.

Now it’s time to create template on AWS ES cluster:

curl -XPUT -H 'Content-Type: application/json' \
http://your-cluster.eu-west-1.es.amazonaws.com/_template/filebeat-7.9.3 -d@filebeat-7.9.3.json

That’s all preparation work you need to do on AWS ES cluster side.

Install Filebeat on Kubernetes

Now we can install Filebeat to our Kubernetes cluster. I have used this Helm chart. Working FileBeat configuration values.yaml can be found here. So lets download it and apply to EKS cluster:

helm2 upgrade --tls --install filebeat --namespace kube-system \
stable/filebeat --debug --version 4.0.0 --values values.yaml

After we do it we can see a filebeat instances started on K8S cluster:

kubectl -n kube-system get pods -l app.kubernetes.io/name=filebeat
NAME READY STATUS RESTARTS AGE
filebeat-2krhc 1/1 Running 0 2m
filebeat-4r6b6 1/1 Running 0 3m

And now final check: see your pods logs do appear in AWS ES Kibana dashboard.

Small benefits you will have

Small benefit from this installation is that you can control how logs are feed from your pods to ES with pod annotations. This basically mean if you log JSON, you can parse it and see as a parsed fields and not text in your AWS ES, so you can search easier.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store