On the project, we’ve gradually grown to the point where it’s time to have proper tracing – to build real observability, not just monitoring. A long time ago I did something similar with Jaeger – a monster, and it kind of stayed in my drafts from 2019 or 2020. Since right now our entire stack… Read More »
Today let’s talk about how to run OpenTelemetry in Kubernetes and integrate it with the VictoriaMetrics stack – VictoriaMetrics for metrics, VictoriaLogs for logs, and VictoriaTraces for traces. Actually, this post wasn’t planned at all, and once a draft did appear – it was supposed to be the third in the series, but in the… Read More »
Just a quick note, because I’ve needed to do something like this fairly often – and only today discovered how elegantly it’s done with vmalert. So, sometimes in an alert you want to display multiple $value entries, for example: – alert: OpenAI Budget Usage expr: | openai_budget_used_usd / openai_budget_total_usd * 100 > 80 … annotations:… Read More »
I think all Opsgenie users are aware that Atlassian is killing is shutting down the project. I’ve been using Opsgenie since 2018, got used to it, and overall it had everything I needed from an alerting system – a bit rough around the edges in places, but the necessary integrations worked and were easy enough… Read More »
Finally got around to monitoring. I was interested in running a standard stack with VictoriaMetrics + Grafana + Alertmanager not in the usual Kubernetes with a Helm chart, but simply on the host. However, the approach is the same as monitoring services in AWS/Kubernetes – on FreeBSD, we will have VictoriaMetrics for metrics, Grafana for… Read More »
We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »
In continuation of the topic with Підготовка до зими 2024-2025: ДБЖ, інвертори, та акумулятори (in Ukrainian). Surprise – there’s even a Prometheus exporter for the EcoFlow – berezhinskiy/ecoflow_exporter! It looks really cool. I launched it, looked at it, and ran to write this post. It can be run in a couple of clicks with Docker… Read More »
We have an AWS Elastic Kubernetes Service cluster with Karpenter which is responsible for EC2 auto-scaling, see AWS: Getting started with Karpenter for autoscaling in EKS, and its installation with Helm. In general, there are no problems with it so far, but in any case we need to monitor it. For its monitoring, Karpenter provides… Read More »
Another interesting announcement from the last re:Invent is that CloudWatch has added the ability to collect metrics from external resources (see a very interesting report AWS re:Invent 2023 – Cloud operations for today, tomorrow, and beyond (COP227)). That is, we can now create graphs and/or alerts not only from the default metrics of CloudWatch itself,… Read More »
In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More »