We use two systems for collecting logs in the project: Grafana Loki and VictoriaLogs, to which Promtail simultaneously writes all collected logs. We cannot get rid of Loki: although developers have long since switched to VictoriaLogs, some alerts are still created from metrics generated by Loki, so it is still present in the system. And… Read More »
AWS EKS Terraform module version v21.0.0 added support for the AWS Provider Version 6. Documentation – here>>>. The main changes in the AWS EKS module are the replacement of IRSA with EKS Pod Identity for the Karpenter sub-module: Native support for IAM roles for service accounts (IRSA) has been removed; EKS Pod Identity is now… Read More »
A fairly common error when upgrading module versions, when there are restrictions on module or provider versions, and they do not match each other. The Issue In this case, I merged Pull Requests from Renovate and didn’t notice that terraform-aws-modules/terraform-aws-lambda needed hashicorp/aws provider version 6: And first I upgraded Lambda to version 8. After that… Read More »
We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it. But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can’t just change the size through values.yaml file, because it will lead to the error“Forbidden: updates… Read More »
It’s not the first time I’ve encountered similar errors when upgrading VictoriaMetrics, so it’s time to note a solution. So, after upgrading the victoria-metrics-k8s-stack 0.55.0 => 0.56.0, errors appeared in the Operator logs: … {“logger”:”controller-runtime.source.EventHandler”,”msg”:”if kind is a CRD, it should be installed before calling Start”,”kind”:”VMAnomaly.operator.victoriametrics.com”,”error”:”no matches for kind \”VMAnomaly\” in version \”operator.victoriametrics.com/v1\””} … {“logger”:”setup”,”msg”:”cannot… Read More »
Even though the changes were announced in emails from Arch Linux, for some reason, many people have had problems with the latest update: in the Arch Linux subreddits on Reddit, people are complaining like “Everything is broken, what should I do now?” Let’s see how to complete the upgrade and what exactly has changed. The… Read More »
After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation. Let’s… Read More »
We have VictoriaMetrics and VictoriaLogs running on an AWS Elastic Kubernetes Service cluster. We do major upgrades to EKS by creating a new cluster, and therefore we have to transfer monitoring data from the old VMSingle instance to the new one. For VictoriaMetrics, there is the vmctl tool which can migrate data through the APIs… Read More »
Terraform has two ways to bring existing resources under Terraform management – using the Terraform CLI and the terraform import command, or using the import resource. Why might we need to import resources? if we already have a manually configured (the “clickops”) service that we want to bring under Terraform management (for example, the common… Read More »
We have an automation for AWS IAM that creates EKS Access Entries to give AWS IAM Users access to an EKS cluster. I don’t remember if I wrote it myself or if some LLM generated it (although judging by the code, I did 🙂 ), but later I discovered an unpleasant feature of how this… Read More »