Category Archives: Monitoring

Hardware, services and network monitoring systems

Kubernetes: Liveness and Readiness Probes – Best practices
0 (0)

18 November 2023

Some useful tips on using Liveness and Readiness Probes in Kubernetes – the difference between them, and how to properly configure these checks. To put it very briefly: livenessProbe: is used by Kubernetes to know when to perform a Pod restart readinessProbe: is used by Kubernetes to know when a container is ready to receive… Read More: Kubernetes: Liveness and Readiness Probes – Best practices0 (0) »

Loading

VictoriaMetrics: pushing metrics without Prometheus Pushgateway
0 (0)

18 November 2023

In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More: VictoriaMetrics: pushing metrics without Prometheus Pushgateway0 (0) »

Loading

VictoriaMetrics: VMAuth – Proxy, Authentication, and Authorization
0 (0)

27 August 2023

We continue to develop our monitoring stack. See the first part – VictoriaMetrics: creating a Kubernetes monitoring stack with its own Helm chart. What do we want to do next: give access to developers so that they can set Silence for alerts themselves in Alertmanager to avoid spamming Slack, see Prometheus: Alertmanager Web UI alerts Silence.… Read More: VictoriaMetrics: VMAuth – Proxy, Authentication, and Authorization0 (0) »

Loading

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana
0 (0)

19 August 2023

We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are no… Read More: Grafana: values ​​from records in Loki logs, and dual-Y-axes panels… »

Loading

Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries
0 (0)

19 August 2023

So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to work… Read More: Grafana Loki: performance optimization with Recording Rules, caching, and parallel… »

Loading

AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity
0 (0)

19 August 2023

Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an increase… Read More: AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity0… »

Loading

VictoriaMetrics: deploying a Kubernetes monitoring stack
0 (0)

23 July 2023

Now we have VictoriaMetrics + Grafana on a regular EC2 instance, launched with Docker Compose, see the VictoriaMetrics: an overview and its use instead of Prometheus. It was kind of a Proof of Concept, and it’s time to launch it “in an adult way” – in Kubernetes and all the configurations stored in a GitHub… Read More: VictoriaMetrics: deploying a Kubernetes monitoring stack0 (0) »

Loading

VictoriaMetrics: an overview and its use instead of Prometheus
0 (0)

11 June 2023

I’ve heard a lot about VictoriaMetrics for a long time, and finally, it’s time to try it out. So, in a nutshell – VictoriaMetrics is “Prometheus on steroids” and is fully compatible with it – can use its configuration files, exporters, PromQL, etc. So for me who has always used Prometheus, the first question was… Read More: VictoriaMetrics: an overview and its use instead of Prometheus0 (0) »

Loading

Prometheus: GitHub Exporter – creating own exporter for GitHub API
0 (0)

11 June 2023

Recently, I got a new interesting task – to build a dashboard in Grafana that would display the status of our development process and its performance, that is, the efficiency of our DevOps processes. This is necessary because we are trying to build “true continuous deployment” so that the code automatically enters Production, and we… Read More: Prometheus: GitHub Exporter – creating own exporter for GitHub API0… »

Loading

Loki: collecting logs from CloudWatch Logs using Lambda Promtail
0 (0)

20 May 2023

Collecting logs in Grafana Loki with Kubernetes is very simple – we just launch Promtail in DaemonSet, configure it to read all data from /var/logs – and that’s it (in fact, we don’t specify anything at all – everything works out of the box from the Helm chart). But what about CloudWatch Logs? On my new project,… Read More: Loki: collecting logs from CloudWatch Logs using Lambda Promtail0 (0) »

Loading