Tag Archives: monitoring

Grafana Loki: LogQL and Recoding Rules for metrics from AWS Load Balancer logs

24 February 2024
 

 I didn’t plan this post at all as I thought I would do it quickly, but it didn’t work out quickly, and I need to dig a little deeper into this topic. So, what we are talking about: we have AWS Load Balancers, logs from which are collected to Grafana Loki, see. Grafana Loki: collecting… Read More »

VictoriaMetrics: pushing metrics without Prometheus Pushgateway

18 November 2023
 

 In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More »

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana

19 August 2023
 

  We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are… Read More »

Prometheus: GitHub Exporter – creating own exporter for GitHub API

11 June 2023
 

  Recently, I got a new interesting task – to build a dashboard in Grafana that would display the status of our development process and its performance, that is, the efficiency of our DevOps processes. This is necessary because we are trying to build “true continuous deployment” so that the code automatically enters Production, and… Read More »

Kubernetes: vertical Pods scaling with Vertical Pod Autoscaler

1 May 2023
 

  In addition to the Horizontal Pod Autoscaler (HPA), which creates additional pods if the existing ones start using more CPU/Memory than configured in the HPA limits, there is also the Vertical Pod Autoscaler (VPA), which works according to a different scheme: instead of horizontal scaling, i.e. increasing the number of Pods, it changes resources.requests of a Pod,… Read More »

Prometheus: running Pushgateway on Kubernetes with Helm and Terraform

28 April 2023
 

  We have a lot of AWS Lambda functions in the project, and developers want to be able to send metrics to our Prometheus to add their own alerts and graphs in Grafana. For this, the functions use the Prometheus library, which allows these metrics to be created (see Prometheus: Building a Custom Prometheus Exporter… Read More »

GitLab: gitlab-shell timeouts, and /metrics Connection refused

25 April 2023
 

  After running our self-hosted GitLab in production, we faced a bug: during git clone/pull/push operations, the request sometimes hung for 1-2 minutes. It looked like some kind of “floating” bug, that is, it could normally work 5 times, and then hangs once. The issues gitlab-shell timeouts For example, one time git clone works well:… Read More »

GitLab: monitoring – Prometheus, metrics, and Grafana dashboard

12 March 2023
 

 So, let’s continue our journey with migrating GitLab to Kubernetes. See previous parts: GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3 GitLab: міграція даних з GitLab cloud та процес backup-restore у self-hosted версії в Kubernetes In general, everything is working,… Read More »

Grafana Loki: alerts from the Loki Ruler and labels from logs

12 March 2023
 

  For general information on Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Among other services that make up Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple:… Read More »