After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation.… Read More »
To put it very simply, Celery is something we can use to perform tasks outside of our main service. For example, there is a Backend API that has some kind of endpoint to which mobile devices send information that the user has created a new whatever in the application. The task of the Backend is… Read More »
In Kubernetes, we run GitHub Runner for the build and deploy of our Backend API, see GitHub Actions: Running Actions Runner Controller in Kubernetes. But over time, we noticed that there was too much traffic on the NAT Gateway – see VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki.… Read More »
I have already mentioned the EXPLAIN feature in the PostgreSQL: AWS RDS Performance and monitoring blog post, but this is such an interesting and useful thing that it’s worth talking about it separately. In addition, AWS RDS for PostgreSQL has the ability to enable Execution Plans logging with EXPLAIN, which is also useful for… Read More »
We are migrating our Backend API from DynamoDB to AWS RDS PostgreSQL, and several times RDS crashed. Actually, given that we took db.t3.small with two vCPUs and two gigabytes of memory to save money, it was quite expected, but I wondered why everything was crashing. A few days later, I started to debug the… Read More »
One day, we looked at our AWS costs on AWS Load Balancers, and understood that we needed to decrease the spends a bit. What we wanted was to have one LoadBalancer, and through it to route requests to different Kubernetes Ingresses and Services in different Namespaces. The first thing that came to mind was… Read More »
We run GitHub Runners in Kubernetes to build and deploy our Backend API, see GitHub Actions: running the Actions Runner Controller in Kubernetes. But over time, we noticed that there was too much traffic on the NAT Gateway – see VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki.… Read More »
In the previous post – AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki, we created a Grafana dashboard that displays NAT Gateway traffic usage statistics. What we were interested in there was which Kubernetes Pods use the most bytes, because it directly affects our AWS Costs. And everything appears… Read More »
Continuing the topic about AWS: VPC Flow Logs, NAT Gateways, and Kubernetes Pods – a detailed overview. There we analyzed how to work with VPC Flow Logs in general, and learned how we can get information about traffic to/from Kubernetes Pods. But there is one problem when using Flow Logs with CloudWatch Logs – the… Read More »
Disruption budgets were introduced in version 0.36, and it looks like a very interesting tool to limit Karpenter from recreating WorkerNodes. For example, in my case, we don’t want EC2 instances to be killed during business hours in the US because we have customers there, so we currently have consolidationPolicy=whenEmpty to prevent “unnecessary” deletion… Read More »