Got a new task to monitor costs on OpenAI – to see how much each project spends per day, and send alerts to Slack if costs exceed a set threshold. I tried several existing exporters for the OpenAI API, but none provided cost metrics per project – so we’ll build one ourselves. To create the… Read More »
One day I received one of the default VictoriaMetrics alerts that are generated during the deployment of the Helm chart victoria-metrics-k8s-stack: I thought about writing a short post like “What is Churn Rate and how to fix it,” but in the end, I ended up diving deep into how VictoriaMetrics works with data in general… Read More »
We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »
Let’s continue our journey with AWS OpenSearch Service. What we have is a small AWS OpenSearch Service cluster with three data nodes, used as a vector store for AWS Bedrock Knowledge Bases. Previous parts: AWS: Introduction to OpenSearch Service as a vector store AWS: Creating an OpenSearch Service cluster and configuring authentication and authorization Terraform:… Read More »
We use two systems for collecting logs in the project: Grafana Loki and VictoriaLogs, to which Promtail simultaneously writes all collected logs. We cannot get rid of Loki: although developers have long since switched to VictoriaLogs, some alerts are still created from metrics generated by Loki, so it is still present in the system.… Read More »
It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »
We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »
In the previous article, we figured out what an MCP is in general, and created a basic MCP server that was connected to Windsurf – see AI: What is the MCP? Now, let’s try to create something more useful, for example, an MCP server that will connect to VictoriaLogs and receive some data. In fact,… Read More »
To put it very simply, Celery is something we can use to perform tasks outside of our main service. For example, there is a Backend API that has some kind of endpoint to which mobile devices send information that the user has created a new whatever in the application. The task of the Backend is… Read More »
I have already mentioned the EXPLAIN feature in the PostgreSQL: AWS RDS Performance and monitoring blog post, but this is such an interesting and useful thing that it’s worth talking about it separately. In addition, AWS RDS for PostgreSQL has the ability to enable Execution Plans logging with EXPLAIN, which is also useful for… Read More »