Category Archives: Troubleshooting

Various problems solutions

GitHub Actions: running the Actions Runner Controller in Kubernetes

16 October 2024
 

  We use GitHub Actions for deployments, and eventually came to the point where we wanted to run its Runners on our own Kubernetes cluster because: self-hosted GitHub Runners are cheaper – in fact, you pay only for the servers that run the jobs we need to run SQL migrations on AWS RDS in AWS… Read More »

AWS: Kubernetes and External Secrets Operator for AWS Secrets Manager

24 August 2024
 

  We have a new EKS cluster 1.30 on our project, where we want to completely remove the old IRSA with OIDC and start using EKS Pod Identities – see AWS: EKS Pod Identities – a replacement for IRSA? Simplifying IAM access management. And everything seems to work fine, but when I started deploying our… Read More »

Terraform: EKS and Karpenter version upgrade 19.21 to 20.0

13 July 2024
 

 It seems like a common task to update a version of a Terraform module, but terraform-aws-modules/eks version 20.0 had some pretty big changes with breaking changes. The changes relate to authentication and authorization in AWS IAM and AWS EKS, which I analyzed in the post AWS: Kubernetes and Access Management API, the new authentication in… Read More »

AWS: RDS IAM database authentication, EKS Pod Identities, and Terraform

7 July 2024
 

  We’re preparing to migrate our Backend API database from DynamoDB to AWS RDS with PostgreSQL, and finally decided to try out AWS RDS IAM database authentication, which appeared in 2021. IAM database authentication, as the name implies, allows us to authenticate to RDS using AWS IAM instead of the login-password from the database server… Read More »

Helm: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

24 May 2024
 

 Sometimes, when deploying Helm charts, the error “UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress” may appear: It can occur because the previous deployment failed due to errors in the chart, or the connection between the build machine and the Kubernetes cluster was lost. Check the release status with ls –all: $ helm -n dev-backend-api-ns… Read More »

Arch Linux: fixing the “yay: error while loading shared libraries: libalpm.so.13” error

17 March 2024
 

 After the latest Arch Linux upgrade, Yay reported the following error: yay: error while loading shared libraries: libalpm.so.13: cannot open shared object file: No such file or directory First, let’s check if we have the libalpm library file, and which version: $ sudo find / -type f -name “*.so*” | grep libalpm /usr/lib/libalpm.so.14.0.0 So, we… Read More »

Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda

25 November 2023
 

 Currently, we are able to collect our API Gateway logs from the CloudWatch Logs to Grafana Loki, see. Loki: collecting logs from CloudWatch Logs using Lambda Promtail. But in the process of migrating to Kubernetes, we have Application Load Balancers that can only write logs to S3, and we need to learn how to collect… Read More »

Terraform: building EKS, part 3 – Karpenter installation

16 September 2023
 

 This is the third part of deploying an AWS Elastic Kubernetes Service cluster with Terraform, in which we will add Karpenter to our cluster. I’ve decided to post this separately because it’s quite a long post. And in the next and final (hopefully!), the fourth part, we will add the rest – all kinds of… Read More »

AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity

19 August 2023
 

  Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an… Read More »