Category Archives: Monitoring

Hardware, services and network monitoring systems

Terraform: creating a module for collecting AWS ALB logs in Grafana Loki
0 (0)

24 February 2024

An example of creating a Terraform module to automate log collection from AWS Load Balancers in Grafana Loki. See how the scheme works in the Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda blog. In short, ALB writes logs to an S3 bucket, from where they are picked up by a Lambda… Read More: Terraform: creating a module for collecting AWS ALB logs in… »

Loading

Grafana Loki: LogQL and Recording Rules for metrics from AWS Load Balancer logs
0 (0)

24 February 2024

I didn’t plan this post at all as I thought I would do it quickly, but it didn’t work out quickly, and I need to dig a little deeper into this topic. So, what we are talking about: we have AWS Load Balancers, logs from which are collected to Grafana Loki, see. Grafana Loki: collecting… Read More: Grafana Loki: LogQL and Recording Rules for metrics from AWS… »

Loading

Karpenter: its monitoring, and Grafana dashboard for Kubernetes WorkerNodes
0 (0)

18 February 2024

We have an AWS Elastic Kubernetes Service cluster with Karpenter which is responsible for EC2 auto-scaling, see AWS: Getting started with Karpenter for autoscaling in EKS, and its installation with Helm. In general, there are no problems with it so far, but in any case we need to monitor it. For its monitoring, Karpenter provides… Read More: Karpenter: its monitoring, and Grafana dashboard for Kubernetes WorkerNodes0 (0) »

Loading

AWS: CloudWatch – Multi source query: collecting metrics from an external Prometheus
0 (0)

13 December 2023

Another interesting announcement from the last re:Invent is that CloudWatch has added the ability to collect metrics from external resources (see a very interesting report AWS re:Invent 2023 – Cloud operations for today, tomorrow, and beyond (COP227)). That is, we can now create graphs and/or alerts not only from the default metrics of CloudWatch itself,… Read More: AWS: CloudWatch – Multi source query: collecting metrics from an… »

Loading

Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda
0 (0)

25 November 2023

Currently, we are able to collect our API Gateway logs from the CloudWatch Logs to Grafana Loki, see. Loki: collecting logs from CloudWatch Logs using Lambda Promtail. But in the process of migrating to Kubernetes, we have Application Load Balancers that can only write logs to S3, and we need to learn how to collect… Read More: Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail… »

Loading

Kubernetes: Liveness and Readiness Probes – Best practices
0 (0)

18 November 2023

Some useful tips on using Liveness and Readiness Probes in Kubernetes – the difference between them, and how to properly configure these checks. To put it very briefly: livenessProbe: is used by Kubernetes to know when to perform a Pod restart readinessProbe: is used by Kubernetes to know when a container is ready to receive… Read More: Kubernetes: Liveness and Readiness Probes – Best practices0 (0) »

Loading

VictoriaMetrics: pushing metrics without Prometheus Pushgateway
0 (0)

18 November 2023

In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More: VictoriaMetrics: pushing metrics without Prometheus Pushgateway0 (0) »

Loading

VictoriaMetrics: VMAuth – Proxy, Authentication, and Authorization
0 (0)

27 August 2023

We continue to develop our monitoring stack. See the first part – VictoriaMetrics: creating a Kubernetes monitoring stack with its own Helm chart. What do we want to do next: give access to developers so that they can set Silence for alerts themselves in Alertmanager to avoid spamming Slack, see Prometheus: Alertmanager Web UI alerts Silence.… Read More: VictoriaMetrics: VMAuth – Proxy, Authentication, and Authorization0 (0) »

Loading

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana
0 (0)

19 August 2023

We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are no… Read More: Grafana: values ​​from records in Loki logs, and dual-Y-axes panels… »

Loading

Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries
0 (0)

19 August 2023

So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to work… Read More: Grafana Loki: performance optimization with Recording Rules, caching, and parallel… »

Loading