Continuing the topic of AWS Load Balancer logs: in the previous post, Golang: recording AWS Load Balancer logs to VictoriaLogs, we collected logs using our own logs collector in Golang. Now we need to get something useful out of these logs. Previously, when I had Loki on my project, we used its RecordingRules to create… Read More »
Got a new task to monitor costs on OpenAI – to see how much each project spends per day, and send alerts to Slack if costs exceed a set threshold. I tried several existing exporters for the OpenAI API, but none provided cost metrics per project – so we’ll build one ourselves. To create the… Read More »
One day I received one of the default VictoriaMetrics alerts that are generated during the deployment of the Helm chart victoria-metrics-k8s-stack: I thought about writing a short post like “What is Churn Rate and how to fix it,” but in the end, I ended up diving deep into how VictoriaMetrics works with data in general… Read More »
We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »
Let’s continue our journey with AWS OpenSearch Service. What we have is a small AWS OpenSearch Service cluster with three data nodes, used as a vector store for AWS Bedrock Knowledge Bases. Previous parts: AWS: Introduction to OpenSearch Service as a vector store AWS: Creating an OpenSearch Service cluster and configuring authentication and authorization Terraform:… Read More »
It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} … {“logger”:”setup”,”msg”:”cannot… Read More »
We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »
We continue the migration from Grafana Loki to VictoriaLogs, and the next task is to transfer Recording Rules from Loki to VictoriaLogs and update the alerts. Recording Rules and integration with VMAlert were brought to VictoriaLogs relatively recently, and I haven’t tested this scheme yet. Therefore, we will first do everything by hand to see… Read More »
In the previous post – AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki, we created a Grafana dashboard that displays NAT Gateway traffic usage statistics. What we were interested in there was which Kubernetes Pods use the most bytes, because it directly affects our AWS Costs. And everything appears to… Read More »
Continuing the topic about AWS: VPC Flow Logs, NAT Gateways, and Kubernetes Pods – a detailed overview. There we analyzed how to work with VPC Flow Logs in general, and learned how we can get information about traffic to/from Kubernetes Pods. But there is one problem when using Flow Logs with CloudWatch Logs – the… Read More »