We are using Logz.io to collect our Kubernetes cluster logs (also, there is a local Loki instance).

Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here – logzio-k8s.

The problem we faced is that those pods are consuming too much CPU – up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.

So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with Ansible via a Helm chart, but Fluentd has no ready charts yet.

The found solution w a Filebeat collector, see Ship k8s logs with Helm via Filebeat – let’s try it.

At first, we will deploy it manually do a Dev cluster, and then we’ll add an Aisible task to deploy it on our Production.

Contents

Logz.io Filebeat Helm chart

Add a repository:

[simterm]

$ helm repo add logzio-helm https://logzio.github.io/logzio-helm/filebeat
"logzio-helm" has been added to your repositories

[/simterm]

Next, we need to fin our token and region – go to the account’s General Settings:

As we have the default region – us-east-1, see How to look up your account region, then we can remove the secrets.logzioRegion from the Helm chart’s arguments.

Also, by default Logz.io client will be installed to the kube-system namespace, but I’d like to place it to a dedicated NS to make it easier to monitor its used resources.

The Namespace is configured in the values file so let’s override it with --set, plus let’s add the --create-namespace and --debug options:

[simterm]

$ helm install --namespace=dev-1-18-devops-logzio-ns \
--create-namespace --debug \
--set secrets.logzioShippingToken='AVG***Onq' \
--set secrets.clusterName='bttrm-eks-dev-1-18' \
--set namespace=dev-1-18-devops-logzio-ns \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

[/simterm]

Check pods:

[simterm]

$ kubectl -n dev-1-18-devops-logzio-ns get pod
NAME             READY   STATUS    RESTARTS   AGE
filebeat-2qt5s   1/1     Running   0          96s
filebeat-4xb44   1/1     Running   0          96s
filebeat-9prr9   1/1     Running   0          96s
filebeat-cth47   1/1     Running   0          96s
filebeat-fgmgx   1/1     Running   0          96s
filebeat-gb5ts   1/1     Running   0          96s
filebeat-hs9tr   1/1     Running   0          96s
filebeat-nskvg   1/1     Running   0          96s
filebeat-wfgbg   1/1     Running   0          96s

[/simterm]

And logs in the Kibana on the app.logs.io page:

Ansible

There is a post about deploying a Helm chart with Ansible – Ansible: модуль community.kubernetes и установка Helm-чарта с ExternalDNS (Rus), at this time just a quick example.

Add a when condition as we will deploy Logz.io to only the Production cluster:

- name: "Add Logzio chart repo"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm_repository:
    name: "logzio-helm"
    repo_url: "https://logzio.github.io/logzio-helm/filebeat"

- name: "Deploy Logzio Filebit chart to the {{ eks_env }}-devops-logzio-ns namespace"
  when: "eks_env.startswith('prod')"
  community.kubernetes.helm:
    kubeconfig: "{{ kube_config_path }}"
    name: "logzio-k8s-logs"
    chart_ref: "logzio-helm/logzio-k8s-logs"
    release_namespace: "{{ eks_env }}-devops-logzio-ns"
    create_namespace: true
    values:
      secrets:
        logzioShippingToken: "{{ logzio_token }}"
        clusterName: "{{ eks_cluster_name }}"
      namespace: "{{ eks_env }}-devops-logzio-ns"

Specify the logzio_token in the Ansible variables and encrypt it with the ansible-vault, as we have our Ansible roles stored in our Github repository.