Kubernetes: PVC in StatefulSet, and the “Forbidden updates to statefulset spec” error

22 July 2025
 

 We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it. But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates… Read More »

Kubernetes: what are the Kubernetes Operator and CustomResourceDefinition

21 July 2025
 

 Perhaps everyone has used operators in Kubernetes, for example, PostgreSQL operator, VictoriaMetrics Operator. But what’s going on under the hood? How and to what are CustomResourceDefinition (CRD) applied, and what is an “operator”? And finally, what is the difference between a Kubernetes Operator and a Kubernetes Controller? In the previous part – Kubernetes: Kubernetes APIs,… Read More »

Kubernetes: Pod resources.requests, resources.limits, and Linux cgroups

20 July 2025
 

 How exactly do resources.requests and resources.limits in a Kubernetes manifest works “under the hood”, and how exactly will Linux allocate and limit resources for containers? So, in Kubernetes for Pods, we can set two main parameters for CPU and Memory – the spec.containers.resources.requests and spec.containers.resources.limits fields: resources.requests: affects how and where a Pod will be… Read More »

TCP/IP: OSI and TCP/IP models, TCP packets, Linux sockets and ports

19 July 2025
 

  It’s difficult to describe in one post what thousands of books have been written about in a thousand pages, but today we’ll try to quickly review the basics of how hosts communicate on a network. First, let’s talk about the OSI and TCP/IP models, then about packet structure and connection establishment, and finally, we’ll… Read More »

VictoriaMetrics: fixing the “no matches for kind VMAnomaly” error

10 July 2025
 

  It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »

Arch Linux: linux-firmware-nvidia: /usr/lib/firmware/nvidia/ exists in filesystem

9 July 2025
 

  Even though the changes were announced in emails from Arch Linux, for some reason, many people have had problems with the latest update: in the Arch Linux subreddits on Reddit, people are complaining like “Everything is broken, what should I do now?” Let’s see how to complete the upgrade and what exactly has changed.… Read More »

Kubernetes: 503 errors with AWS ALB possible causes and solutions

9 July 2025
 

  After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation.… Read More »

VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes cluster

5 July 2025
 

 We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »