It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution.
So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs:
...
{"logger":"controller-runtime.source.EventHandler","msg":"if kind is a CRD, it should be installed before calling Start","kind":"VMAnomaly.operator.victoriametrics.com","error":"no matches for kind \"VMAnomaly\" in version \"operator.victoriametrics.com/v1\""}
...
{"logger":"setup","msg":"cannot setup manager","error":"cannot start controller manager: failed to wait for vmanomaly caches to sync kind source: *v1.VMAnomaly: timed out waiting for cache to be synced for
Kind *v1.VMAnomaly"}
...
Most likely due to the addition of a new VMAnomaly resource in v0.60.0 of the operator.
Let’s verify it by checking the existing CRDs:
$ kk get crd | grep victoriametrics vlclusters.operator.victoriametrics.com 2025-06-17T13:13:06Z vlogs.operator.victoriametrics.com 2025-06-10T09:39:22Z vlsingles.operator.victoriametrics.com 2025-06-17T13:13:07Z vmagents.operator.victoriametrics.com 2025-06-10T09:39:23Z vmalertmanagerconfigs.operator.victoriametrics.com 2025-06-10T09:39:23Z vmalertmanagers.operator.victoriametrics.com 2025-06-10T09:39:23Z vmalerts.operator.victoriametrics.com 2025-06-10T09:39:22Z vmauths.operator.victoriametrics.com 2025-06-10T09:39:22Z vmclusters.operator.victoriametrics.com 2025-06-10T09:39:23Z vmnodescrapes.operator.victoriametrics.com 2025-06-10T09:39:22Z vmpodscrapes.operator.victoriametrics.com 2025-06-10T09:39:23Z vmprobes.operator.victoriametrics.com 2025-06-10T09:39:22Z vmrules.operator.victoriametrics.com 2025-06-10T09:39:22Z vmscrapeconfigs.operator.victoriametrics.com 2025-06-10T09:39:23Z vmservicescrapes.operator.victoriametrics.com 2025-06-10T09:39:22Z vmsingles.operator.victoriametrics.com 2025-06-10T09:39:23Z vmstaticscrapes.operator.victoriametrics.com 2025-06-10T09:39:23Z vmusers.operator.victoriametrics.com 2025-06-10T09:39:22Z
As expected, there is no VMAnomaly.
The best option would be simply to install all the CRDs from a dedicated Helm chart victoriametrics-operator-crds (which under the hood simply installs CRDs from the file victoria-metrics-operator/charts/crds/crds/crd.yaml)
But historically, we already have CRDs set manually (maybe there was no separate chart back then?), so the easiest option is to just update them directly from the file:
$ kk apply -f https://raw.githubusercontent.com/VictoriaMetrics/helm-charts/refs/heads/master/charts/victoria-metrics-operator/charts/crds/crds/crd.yaml ... customresourcedefinition.apiextensions.k8s.io/vmanomalies.operator.victoriametrics.com created ...
Now let’s check the CRD in the cluster:
$ kk get crd | grep anomal vmanomalies.operator.victoriametrics.com 2025-07-09T14:27:43Z
Done, everything works.
