We are using Logz.io to collect our Kubernetes cluster logs (also, there is a local Loki instance).

Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here – logzio-k8s.

The problem we faced is that those pods are consuming too much CPU – up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.

So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.

The found solution w a Filebeat collector, see Ship k8s logs with Helm via Filebeat – let’s try it.

At first, we will deploy it manually do a Dev cluster, and then we’ll add an Aisible task to deploy it on our Production.

Logz.io Filebeat Helm chart

Add a repository:

Next, we need to fin our token and region – go to the account’s General Settings:

As we have the default region – us-east-1, see How to look up your account region, then we can remove the secrets.logzioRegion from the Helm chart’s arguments.

Also, by default Logz.io client will be installed to the kube-system namespace, but I’d like to place it to a dedicated NS to make it easier to monitor its used resources.

The Namespace is configured in the values file so let’s override it with --set, plus let’s add the --create-namespace and --debug options:

Check pods:

And logs in the Kibana on the app.logs.io page:

Ansible

There is a post about deploying a Helm chart with Ansible – Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS (Rus), at this time just a quick example.

Add a when condition as we will deploy Logz.io to only the Production cluster:

- name: "Add Logzio chart repo"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm_repository:
    name: "logzio-helm"
    repo_url: "https://logzio.github.io/logzio-helm/filebeat"

- name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm:
    kubeconfig: "{{ kube_config_path }}"
    name: "logzio-k8s-logs"
    chart_ref: "logzio-helm/logzio-k8s-logs"
    release_namespace: "{{ eks_env }}-devops-logzio-ns"
    create_namespace: true
    values:
      secrets:
        logzioShippingToken: "{{ logzio_token }}"
        clusterName: "{{ eks_cluster_name }}"
      namespace: "{{ eks_env }}-devops-logzio-ns"

Specify the logzio_token in the Ansible variables and encrypt it with the ansible-vault, as we have our Ansible roles stored in our Github repository.

Fluentd vs Filebeat – CPU and performance

And finally CPU usage: old fluentd (Ruby + C) on the left side vs new filebeat (Golang) at the right side:

Also published on Medium.