After migration to a new EKS cluster, we started getting alerts about 503 errors sometimes. The errors were happened in three cases: sometimes without any deployment, when all Pods were Running && Ready sometimes during deployment – but only on Dev, because there is only one Pod for API and sometimes during Karpenter Consolidation.… Read More »
To put it very simply, Celery is something we can use to perform tasks outside of our main service. For example, there is a Backend API that has some kind of endpoint to which mobile devices send information that the user has created a new whatever in the application. The task of the Backend is… Read More »
In Kubernetes, we run GitHub Runner for the build and deploy of our Backend API, see GitHub Actions: Running Actions Runner Controller in Kubernetes. But over time, we noticed that there was too much traffic on the NAT Gateway – see VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki.… Read More »
One day, we looked at our AWS costs on AWS Load Balancers, and understood that we needed to decrease the spends a bit. What we wanted was to have one LoadBalancer, and through it to route requests to different Kubernetes Ingresses and Services in different Namespaces. The first thing that came to mind was… Read More »
We run GitHub Runners in Kubernetes to build and deploy our Backend API, see GitHub Actions: running the Actions Runner Controller in Kubernetes. But over time, we noticed that there was too much traffic on the NAT Gateway – see VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki.… Read More »
So, we’re back to the topic of AWS VPC Flow Logs, VictoriaLogs, and the Grafana dashboard. In the post VictoriaLogs: a Grafana dashboard for AWS VPC Flow Logs – migrating from Grafana Loki, we created a cool dashboard to display various statistics on AWS NAT Gateway traffic. But there is a small drawback: all… Read More »
In the previous post – AWS: VPC Flow Logs – logs to S3 and Grafana dashboard with Loki, we created a Grafana dashboard that displays NAT Gateway traffic usage statistics. What we were interested in there was which Kubernetes Pods use the most bytes, because it directly affects our AWS Costs. And everything appears… Read More »
Continuing the topic about AWS: VPC Flow Logs, NAT Gateways, and Kubernetes Pods – a detailed overview. There we analyzed how to work with VPC Flow Logs in general, and learned how we can get information about traffic to/from Kubernetes Pods. But there is one problem when using Flow Logs with CloudWatch Logs – the… Read More »
Disruption budgets were introduced in version 0.36, and it looks like a very interesting tool to limit Karpenter from recreating WorkerNodes. For example, in my case, we don’t want EC2 instances to be killed during business hours in the US because we have customers there, so we currently have consolidationPolicy=whenEmpty to prevent “unnecessary” deletion… Read More »
I will write about VictoriaMetrics Cloud itself separately, but now I want to check how you can write CloudWatch Metrics via AWS Firehose to VictoriaMetrics Cloud. In fact, the AWS Data Firehose service itself allows you to transfer streaming data from various sources to Amazon services such as AWS S3, Redshift, Open Search, or… Read More »