Category Archives: Monitoring

Hardware, services and network monitoring systems

VictoriaMetrics: Churn Rate, High cardinality, metrics, and IndexDB

3 November 2025
 

 One day I received one of the default VictoriaMetrics alerts that are generated during the deployment of the Helm chart victoria-metrics-k8s-stack: I thought about writing a short post like “What is Churn Rate and how to fix it,” but in the end, I ended up diving deep into how VictoriaMetrics works with data in general… Read More »

Kubernetes: monitoring processes with process-exporter

1 November 2025
 

 We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »

AWS: Monitoring AWS OpenSearch Service cluster with CloudWatch

1 November 2025
 

 Let’s continue our journey with AWS OpenSearch Service. What we have is a small AWS OpenSearch Service cluster with three data nodes, used as a vector store for AWS Bedrock Knowledge Bases. Previous parts: AWS: Introduction to OpenSearch Service as a vector store AWS: Creating an OpenSearch Service cluster and configuring authentication and authorization Terraform:… Read More »

VictoriaLogs: “rate limit exceeded” and monitoring ingested logs

13 September 2025
 

  We use two systems for collecting logs in the project: Grafana Loki and VictoriaLogs, to which Promtail simultaneously writes all collected logs. We cannot get rid of Loki: although developers have long since switched to VictoriaLogs, some alerts are still created from metrics generated by Loki, so it is still present in the system.… Read More »

VictoriaMetrics: fixing the “no matches for kind VMAnomaly” error

10 July 2025
 

  It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »

VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes cluster

5 July 2025
 

 We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »

Python: introduction to the Celery, and its monitoring configurations

20 May 2025
 

 To put it very simply, Celery is something we can use to perform tasks outside of our main service. For example, there is a Backend API that has some kind of endpoint to which mobile devices send information that the user has created a new whatever in the application. The task of the Backend is… Read More »

PostgreSQL: using EXPLAIN and setting up “auto_explain” in AWS RDS

12 February 2025
 

  I have already mentioned the EXPLAIN feature in the PostgreSQL: AWS RDS Performance and monitoring blog post, but this is such an interesting and useful thing that it’s worth talking about it separately. In addition, AWS RDS for PostgreSQL has the ability to enable Execution Plans logging with EXPLAIN, which is also useful for… Read More »