Category Archives: Troubleshooting

Various problems solutions

Kubernetes: PVC in StatefulSet, and the “Forbidden updates to statefulset spec” error

22 July 2025
 

 We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it. But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates… Read More »

VictoriaMetrics: fixing the “no matches for kind VMAnomaly” error

10 July 2025
 

  It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »

Arch Linux: linux-firmware-nvidia: /usr/lib/firmware/nvidia/ exists in filesystem

9 July 2025
 

  Even though the changes were announced in emails from Arch Linux, for some reason, many people have had problems with the latest update: in the Arch Linux subreddits on Reddit, people are complaining like “Everything is broken, what should I do now?” Let’s see how to complete the upgrade and what exactly has changed.… Read More »

Kubernetes: 503 errors with AWS ALB possible causes and solutions

9 July 2025
 

  After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation.… Read More »

VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes cluster

5 July 2025
 

 We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »

Terraform: using import, and some hidden pitfalls

15 June 2025
 

 Terraform has two ways to bring existing resources under Terraform management – using the Terraform CLI and the terraform import command, or using the import resource. Why might we need to import resources? if we already have a manually configured (the “clickops”) service that we want to bring under Terraform management (for example, the common… Read More »

Python: introduction to the Celery, and its monitoring configurations

20 May 2025
 

 To put it very simply, Celery is something we can use to perform tasks outside of our main service. For example, there is a Backend API that has some kind of endpoint to which mobile devices send information that the user has created a new whatever in the application. The task of the Backend is… Read More »

Nexus: Configuring Docker proxy repository, and ContainerD in Kubernetes

17 May 2025
 

 I wrote about launching Nexus in the Nexus: launch in Kubernetes, and PyPI caching repository configuration post, now I want to add Docker image caching to PyPI, especially since Docker Hub introduces new limits from April 1, 2025 – see Docker Hub usage and limits. We’ll do it as usual: first run manually locally on… Read More »