Tag Archives: Kubernetes

VictoriaMetrics: fixing the “no matches for kind VMAnomaly” error

10 July 2025
 

  It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »

Kubernetes: 503 errors with AWS ALB possible causes and solutions

9 July 2025
 

  After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation.… Read More »

VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes cluster

5 July 2025
 

 We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »

Kubernetes: find a directory with a mounted volume in a Pod on its host

18 May 2025
 

 We have an AWS Elastic Kubernetes Service with the VictoriaMetrics stack deployed (see VictoriaMetrics: deploying a Kubernetes monitoring stack). I need to migrate the data from the old VMSingle Pod to the new one on the new cluster, and to do this, I need to find VMSingle’s data on an EC2. Note: regarding the migration… Read More »

Nexus: Configuring Docker proxy repository, and ContainerD in Kubernetes

17 May 2025
 

 I wrote about launching Nexus in the Nexus: launch in Kubernetes, and PyPI caching repository configuration post, now I want to add Docker image caching to PyPI, especially since Docker Hub introduces new limits from April 1, 2025 – see Docker Hub usage and limits. We’ll do it as usual: first run manually locally on… Read More »

Nexus: launch in Kubernetes, and PyPI caching repository configuration

17 May 2025
 

 In Kubernetes, we run GitHub Runner for the build and deploy of our Backend API, see GitHub Actions: Running Actions Runner Controller in Kubernetes. But over time, we noticed that there was too much traffic on the NAT Gateway – see VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki.… Read More »

VictoriaLogs: creating Recording Rules with VMAlert

11 January 2025
 

  We continue the migration from Grafana Loki to VictoriaLogs, and the next task is to transfer Recording Rules from Loki to VictoriaLogs and update the alerts. Recording Rules and integration with VMAlert were brought to VictoriaLogs relatively recently, and I haven’t tested this scheme yet. Therefore, we will first do everything by hand to… Read More »

Kubernetes: a single AWS Load Balancer for different Kubernetes Ingresses

21 December 2024
 

  One day, we looked at our AWS costs on AWS Load Balancers, and understood that we needed to decrease the spends a bit. What we wanted was to have one LoadBalancer, and through it to route requests to different Kubernetes Ingresses and Services in different Namespaces. The first thing that came to mind was… Read More »

VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki

7 December 2024
 

  In the previous post – AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki, we created a Grafana dashboard that displays NAT Gateway traffic usage statistics. What we were interested in there was which Kubernetes Pods use the most bytes, because it directly affects our AWS Costs. And everything appears… Read More »