Category Archives: Troubleshooting

Various problems solutions

AWS: Kubernetes and External Secrets Operator for AWS Secrets Manager
0 (0)

24 August 2024

We have a new EKS cluster 1.30 on our project, where we want to completely remove the old IRSA with OIDC and start using EKS Pod Identities – see AWS: EKS Pod Identities – a replacement for IRSA? Simplifying IAM access management. And everything seems to work fine, but when I started deploying our Backend… Read More: AWS: Kubernetes and External Secrets Operator for AWS Secrets Manager0… »

Terraform: EKS and Karpenter version upgrade 19.21 to 20.0
0 (0)

13 July 2024

It seems like a common task to update a version of a Terraform module, but terraform-aws-modules/eks version 20.0 had some pretty big changes with breaking changes. The changes relate to authentication and authorization in AWS IAM and AWS EKS, which I analyzed in the post AWS: Kubernetes and Access Management API, the new authentication in… Read More: Terraform: EKS and Karpenter version upgrade 19.21 to 20.00 (0) »

AWS: RDS IAM database authentication, EKS Pod Identities, and Terraform
0 (0)

7 July 2024

We’re preparing to migrate our Backend API database from DynamoDB to AWS RDS with PostgreSQL, and finally decided to try out AWS RDS IAM database authentication, which appeared in 2021. IAM database authentication, as the name implies, allows us to authenticate to RDS using AWS IAM instead of the login-password from the database server itself.… Read More: AWS: RDS IAM database authentication, EKS Pod Identities, and Terraform0… »

Kubernetes: containers, and the “lost” SIGTERM signals
0 (0)

23 June 2024

We have an API service with Gunicorn in Kubernetes that periodically returns 502, 503, 504 errors. I started debugging it, and found a weird thing: there were no messages in the logs about the received SIGTERM, so I first went to deal with Kubernetes – why doesn’t it send it? The Issue So, here’s what… Read More: Kubernetes: containers, and the “lost” SIGTERM signals0 (0) »

Helm: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
0 (0)

24 May 2024

Sometimes, when deploying Helm charts, the error “UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress” may appear: It can occur because the previous deployment failed due to errors in the chart, or the connection between the build machine and the Kubernetes cluster was lost. Check the release status with ls –all: $ helm -n dev-backend-api-ns… Read More: Helm: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress0 (0) »

Arch Linux: fixing the “yay: error while loading shared libraries: libalpm.so.13” error
0 (0)

17 March 2024

After the latest Arch Linux upgrade, Yay reported the following error: yay: error while loading shared libraries: libalpm.so.13: cannot open shared object file: No such file or directory First, let’s check if we have the libalpm library file, and which version: $ sudo find / -type f -name “*.so*” | grep libalpm /usr/lib/libalpm.so.14.0.0 So, we… Read More: Arch Linux: fixing the “yay: error while loading shared libraries:… »

Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda
0 (0)

25 November 2023

Currently, we are able to collect our API Gateway logs from the CloudWatch Logs to Grafana Loki, see. Loki: collecting logs from CloudWatch Logs using Lambda Promtail. But in the process of migrating to Kubernetes, we have Application Load Balancers that can only write logs to S3, and we need to learn how to collect… Read More: Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail… »

Terraform: building EKS, part 3 – Karpenter installation
0 (0)

16 September 2023

This is the third part of deploying an AWS Elastic Kubernetes Service cluster with Terraform, in which we will add Karpenter to our cluster. I’ve decided to post this separately because it’s quite a long post. And in the next and final (hopefully!), the fourth part, we will add the rest – all kinds of… Read More: Terraform: building EKS, part 3 – Karpenter installation0 (0) »

AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity
0 (0)

19 August 2023

Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an increase… Read More: AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity0… »

Arch Linux: failed to mount on real root
0 (0)

25 July 2023

I updated my Arch Linux yesterday, and for the first time in 9 years of using this system, I encountered an error when after a reboot the system could not mount a disk: ERROR: device UUID not found. mount: /new_root: can’t find UUID. ERROR: Failed to mount UUID on real root. You are now being… Read More: Arch Linux: failed to mount on real root0 (0) »