On the project, we’ve gradually grown to the point where it’s time to have proper tracing – to build real observability, not just monitoring. A long time ago I did something similar with Jaeger – a monster, and it kind of stayed in my drafts from 2019 or 2020. Since right now our entire stack… Read More »
Today let’s talk about how to run OpenTelemetry in Kubernetes and integrate it with the VictoriaMetrics stack – VictoriaMetrics for metrics, VictoriaLogs for logs, and VictoriaTraces for traces. Actually, this post wasn’t planned at all, and once a draft did appear – it was supposed to be the third in the series, but in the… Read More »
While I’m working on a series of posts about setting up and using Claude Code, here’s a quick example of building my own AI Agent for VictoriaMetrics and Kubernetes, “wrapping” it into a Claude Code Plugin, and creating my own Claude Code Marketplace where similar plugins for developers on my project will live. The general… Read More »
We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes. The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10-11 gigabytes, which causes the Pod… Read More »
AWS EKS Terraform module version v21.0.0 added support for the AWS Provider Version 6. Documentation – here>>>. The main changes in the AWS EKS module are the replacement of IRSA with EKS Pod Identity for the Karpenter sub-module: Native support for IAM roles for service accounts (IRSA) has been removed; EKS Pod Identity is now… Read More »
We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it. But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates… Read More »
I actually started to write about creating my own Kubernetes Operator, but decided to make a separate topic about what a Kubernetes CustomResourceDefinition is, and how creating a CRD works at the level of the Kubernetes API and the etcd. That is, to start with how Kubernetes actually works with resources, and what happens when… Read More »
Perhaps everyone has used operators in Kubernetes, for example, PostgreSQL operator, VictoriaMetrics Operator. But what’s going on under the hood? How and to what are CustomResourceDefinition (CRD) applied, and what is an “operator”? And finally, what is the difference between a Kubernetes Operator and a Kubernetes Controller? In the previous part – Kubernetes: Kubernetes APIs,… Read More »
How exactly do resources.requests and resources.limits in a Kubernetes manifest works “under the hood”, and how exactly will Linux allocate and limit resources for containers? So, in Kubernetes for Pods, we can set two main parameters for CPU and Memory – the spec.containers.resources.requests and spec.containers.resources.limits fields: resources.requests: affects how and where a Pod will be… Read More »
It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} … {“logger”:”setup”,”msg”:”cannot… Read More »