Tag Archives: VictoriaMetrics

VictoriaMetrics: fixing the “no matches for kind VMAnomaly” error

10 July 2025
 

  It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} …… Read More »

VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes cluster

5 July 2025
 

 We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »

VictoriaLogs: creating Recording Rules with VMAlert

11 January 2025
 

  We continue the migration from Grafana Loki to VictoriaLogs, and the next task is to transfer Recording Rules from Loki to VictoriaLogs and update the alerts. Recording Rules and integration with VMAlert were brought to VictoriaLogs relatively recently, and I haven’t tested this scheme yet. Therefore, we will first do everything by hand to… Read More »

VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki

7 December 2024
 

  In the previous post – AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki, we created a Grafana dashboard that displays NAT Gateway traffic usage statistics. What we were interested in there was which Kubernetes Pods use the most bytes, because it directly affects our AWS Costs. And everything appears… Read More »

AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki

7 December 2024
 

 Continuing the topic about AWS: VPC Flow Logs, NAT Gateways, and Kubernetes Pods – a detailed overview. There we analyzed how to work with VPC Flow Logs in general, and learned how we can get information about traffic to/from Kubernetes Pods. But there is one problem when using Flow Logs with CloudWatch Logs – the… Read More »

VictoriaMetrics Cloud: integration with AWS Data Firehose for CloudWatch metrics

2 October 2024
 

  I will write about VictoriaMetrics Cloud itself separately, but now I want to check how you can write CloudWatch Metrics via AWS Firehose to VictoriaMetrics Cloud. In fact, the AWS Data Firehose service itself allows you to transfer streaming data from various sources to Amazon services such as AWS S3, Redshift, Open Search, or… Read More »

VictoriaLogs: an overview, run in Kubernetes, LogsQL, and Grafana

9 September 2024
 

 VictoriaLogs is a relatively new system for collecting and analyzing logs, similar to Grafana Loki, but – like VictoriaMetrics compared to vanilla Prometheus – less demanding on CPU/Memory resources. Personally, I’ve been using Grafana Loki for about 5 years, but sometimes I have concerns about it – both in terms of documentation and the overall… Read More »

VictoriaMetrics: pushing metrics without Prometheus Pushgateway

18 November 2023
 

 In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More »

VictoriaMetrics: VMAuth – Proxy, Authentication, and Authorization

27 August 2023
 

  We continue to develop our monitoring stack. See the first part – VictoriaMetrics: creating a Kubernetes monitoring stack with its own Helm chart. What do we want to do next: give access to developers so that they can set Silence for alerts themselves in Alertmanager to avoid spamming Slack, see Prometheus: Alertmanager Web UI alerts… Read More »

Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries

19 August 2023
 

  So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to… Read More »