We are using
Logs are collected and processed by a
The problem we faced is that those pods are consuming too much CPU – up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.
So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.
The found solution w a Filebeat collector, see
At first, we will deploy it manually do a Dev cluster, and then we’ll add an Aisible task to deploy it on our Production.
Contents
Logz.io Filebeat Helm chart
Add a repository:
Next, we need to fin our token and region – go to the account’s General Settings:
As we have the default region – us-east-1, see secrets.logzioRegion
from the Helm chart’s arguments.
Also, by default Logz.io client will be installed to the kube-system
namespace, but I’d like to place it to a dedicated NS to make it easier to monitor its used resources.
The Namespace is configured in the --set
, plus let’s add the --create-namespace
and --debug
options:
Check pods:
And logs in the Kibana on the
Ansible
There is a post about deploying a Helm chart with Ansible – Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS (Rus), at this time just a quick example.
Add a when
condition as we will deploy Logz.io to only the Production cluster:
- name: "Add Logzio chart repo" when: "eks_env.startswith('prod')" community.kubernetes.helm_repository: name: "logzio-helm" repo_url: "https://logzio.github.io/logzio-helm/filebeat" - name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace" when: "eks_env.startswith('prod')" community.kubernetes.helm: kubeconfig: "{{ kube_config_path }}" name: "logzio-k8s-logs" chart_ref: "logzio-helm/logzio-k8s-logs" release_namespace: "{{ eks_env }}-devops-logzio-ns" create_namespace: true values: secrets: logzioShippingToken: "{{ logzio_token }}" clusterName: "{{ eks_cluster_name }}" namespace: "{{ eks_env }}-devops-logzio-ns"
Specify the logzio_token
in the Ansible variables and encrypt it with the ansible-vault
, as we have our Ansible roles stored in our Github repository.
Fluentd vs Filebeat – CPU and performance
And finally CPU usage: old fluentd
(Ruby + C) on the left side vs new filebeat
(Golang) at the right side:
Similar posts
Also published on