Disruption budgets were introduced in version 0.36, and it looks like a very interesting tool to limit Karpenter from recreating WorkerNodes. For example, in my case, we don’t want EC2 instances to be killed during business hours in the US because we have customers there, so we currently have consolidationPolicy=whenEmpty to prevent “unnecessary” deletion… Read More »
VictoriaLogs is a relatively new system for collecting and analyzing logs, similar to Grafana Loki, but – like VictoriaMetrics compared to vanilla Prometheus – less demanding on CPU/Memory resources. Personally, I’ve been using Grafana Loki for about 5 years, but sometimes I have concerns about it – both in terms of documentation and the overall… Read More »
We have a new EKS cluster 1.30 on our project, where we want to completely remove the old IRSA with OIDC and start using EKS Pod Identities – see AWS: EKS Pod Identities – a replacement for IRSA? Simplifying IAM access management. And everything seems to work fine, but when I started deploying our… Read More »
So, we have an AWS Elastic Kubernetes Service cluster with Authentication mode EKS API and ConfigMap, which we enabled during upgrade of the EKS Terraform module from version 19.21 to 20.0. Before switching EKS Authentication mode completely to the API, we need to transfer all users and roles to Access Entries of the EKS cluster… Read More »
It seems like a common task to update a version of a Terraform module, but terraform-aws-modules/eks version 20.0 had some pretty big changes with breaking changes. The changes relate to authentication and authorization in AWS IAM and AWS EKS, which I analyzed in the post AWS: Kubernetes and Access Management API, the new authentication in… Read More »
We have an API service with Gunicorn in Kubernetes that periodically returns 502, 503, 504 errors. I started debugging it, and found a weird thing: there were no messages in the logs about the received SIGTERM, so I first went to deal with Kubernetes – why doesn’t it send it? The Issue So, here’s… Read More »
In Kubernetes, in addition to metrics and logs from containers, we can get information about the operation of components using Kubernetes Events. Events usually store information about the status of Pods (creation, evict, kill, ready or not-ready status of pods), WorkerNodes (status of servers), Kubernetes Scheduler (inability to start a pod, etc.). Kubernetes Events… Read More »
We have an AWS EKS cluster with WorkerNodes/EC2 created with Karpenter. The process of creating the infrastructure, cluster, and launching Karpenter is described in previous posts: Terraform: Building EKS, part 1 – VPC, Subnets and Endpoints Terraform: Building EKS, part 2 – an EKS cluster, WorkerNodes, and IAM Terraform: Building EKS, part 3 –… Read More »
Sometimes, when deploying Helm charts, the error “UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress” may appear: It can occur because the previous deployment failed due to errors in the chart, or the connection between the build machine and the Kubernetes cluster was lost. Check the release status with ls –all: $ helm -n dev-backend-api-ns… Read More »
We have a relatively large spending on AWS NAT Gateway Processed Bytes, and it became interesting to know what exactly is processed through it. It would seem that everything is simple – just turn on VPC Flow Logs and see what’s what. But when it comes to AWS Elastic Kubernetes Service and NAT Gateways, things… Read More »