Tag Archives: monitoring

Grafana Loki: LogQL and Recording Rules for metrics from AWS Load Balancer logs
0 (0)

24 February 2024

I didn’t plan this post at all as I thought I would do it quickly, but it didn’t work out quickly, and I need to dig a little deeper into this topic. So, what we are talking about: we have AWS Load Balancers, logs from which are collected to Grafana Loki, see. Grafana Loki: collecting… Read More: Grafana Loki: LogQL and Recording Rules for metrics from AWS… »

Loading

VictoriaMetrics: pushing metrics without Prometheus Pushgateway
0 (0)

18 November 2023

In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More: VictoriaMetrics: pushing metrics without Prometheus Pushgateway0 (0) »

Loading

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana
0 (0)

19 August 2023

We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are no… Read More: Grafana: values ​​from records in Loki logs, and dual-Y-axes panels… »

Loading

Prometheus: GitHub Exporter – creating own exporter for GitHub API
0 (0)

11 June 2023

Recently, I got a new interesting task – to build a dashboard in Grafana that would display the status of our development process and its performance, that is, the efficiency of our DevOps processes. This is necessary because we are trying to build “true continuous deployment” so that the code automatically enters Production, and we… Read More: Prometheus: GitHub Exporter – creating own exporter for GitHub API0… »

Loading

Kubernetes: vertical Pods scaling with Vertical Pod Autoscaler
0 (0)

1 May 2023

In addition to the Horizontal Pod Autoscaler (HPA), which creates additional pods if the existing ones start using more CPU/Memory than configured in the HPA limits, there is also the Vertical Pod Autoscaler (VPA), which works according to a different scheme: instead of horizontal scaling, i.e. increasing the number of Pods, it changes resources.requests of a Pod, which… Read More: Kubernetes: vertical Pods scaling with Vertical Pod Autoscaler0 (0) »

Loading

Prometheus: running Pushgateway on Kubernetes with Helm and Terraform
0 (0)

28 April 2023

We have a lot of AWS Lambda functions in the project, and developers want to be able to send metrics to our Prometheus to add their own alerts and graphs in Grafana. For this, the functions use the Prometheus library, which allows these metrics to be created (see Prometheus: Building a Custom Prometheus Exporter in… Read More: Prometheus: running Pushgateway on Kubernetes with Helm and Terraform0 (0) »

Loading

GitLab: gitlab-shell timeouts, and /metrics Connection refused
0 (0)

25 April 2023

After running our self-hosted GitLab in production, we faced a bug: during git clone/pull/push operations, the request sometimes hung for 1-2 minutes. It looked like some kind of “floating” bug, that is, it could normally work 5 times, and then hangs once. The issues gitlab-shell timeouts For example, one time git clone works well: [simterm]… Read More: GitLab: gitlab-shell timeouts, and /metrics Connection refused0 (0) »

Loading

GitLab: monitoring – Prometheus, metrics, and Grafana dashboard
0 (0)

12 March 2023

So, let’s continue our journey with migrating GitLab to Kubernetes. See previous parts: GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3 GitLab: міграція даних з GitLab cloud та процес backup-restore у self-hosted версії в Kubernetes In general, everything is working,… Read More: GitLab: monitoring – Prometheus, metrics, and Grafana dashboard0 (0) »

Loading

Grafana Loki: LogQL for logs and creating metrics for alerts
0 (0)

12 March 2023

Good – we learned how to launch Loki – Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and bolted-shipper, we also figured out how to configure alerts – Grafana Loki: alerts from the Loki Ruler and labels from logs. Now it’s time to figure out what we can do in Loki using its… Read More: Grafana Loki: LogQL for logs and creating metrics for alerts0… »

Loading

Grafana Loki: alerts from the Loki Ruler and labels from logs
0 (0)

12 March 2023

For general information on Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Among other services that make up Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple: create… Read More: Grafana Loki: alerts from the Loki Ruler and labels from… »

Loading