Category Archives: Grafana

The open platform for beautiful analytics and monitoring

Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda

25 November 2023

Currently, we are able to collect our API Gateway logs from the CloudWatch Logs to Grafana Loki, see. Loki: collecting logs from CloudWatch Logs using Lambda Promtail. But in the process of migrating to Kubernetes, we have Application Load Balancers that can only write logs to S3, and we need to learn how to collect… Read More »

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana

19 August 2023

We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are no… Read More »

Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries

19 August 2023

So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to work… Read More »

VictoriaMetrics: deploying a Kubernetes monitoring stack

23 July 2023

Now we have VictoriaMetrics + Grafana on a regular EC2 instance, launched with Docker Compose, see the VictoriaMetrics: an overview and its use instead of Prometheus. It was kind of a Proof of Concept, and it’s time to launch it “in an adult way” – in Kubernetes and all the configurations stored in a GitHub… Read More »

VictoriaMetrics: an overview and its use instead of Prometheus

11 June 2023

I’ve heard a lot about VictoriaMetrics for a long time, and finally, it’s time to try it out. So, in a nutshell – VictoriaMetrics is “Prometheus on steroids” and is fully compatible with it – can use its configuration files, exporters, PromQL, etc. So for me who has always used Prometheus, the first question was… Read More »

Prometheus: GitHub Exporter – creating own exporter for GitHub API

11 June 2023

Recently, I got a new interesting task – to build a dashboard in Grafana that would display the status of our development process and its performance, that is, the efficiency of our DevOps processes. This is necessary because we are trying to build “true continuous deployment” so that the code automatically enters Production, and we… Read More »

Loki: collecting logs from CloudWatch Logs using Lambda Promtail

20 May 2023

Collecting logs in Grafana Loki with Kubernetes is very simple – we just launch Promtail in DaemonSet, configure it to read all data from /var/logs – and that’s it (in fact, we don’t specify anything at all – everything works out of the box from the Helm chart). But what about CloudWatch Logs? On my new project,… Read More »

GitLab: monitoring – Prometheus, metrics, and Grafana dashboard

12 March 2023

So, let’s continue our journey with migrating GitLab to Kubernetes. See previous parts: GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3 GitLab: міграція даних з GitLab cloud та процес backup-restore у self-hosted версії в Kubernetes In general, everything is working,… Read More »

Grafana Loki: alerts from the Loki Ruler and labels from logs

12 March 2023

For general information on Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Among other services that make up Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple: create… Read More »