Category Archives: Troubleshooting

Various problems solutions

Grafana Loki: collecting AWS LoadBalancer logs from S3 with Promtail Lambda

25 November 2023

 Currently, we are able to collect our API Gateway logs from the CloudWatch Logs to Grafana Loki, see. Loki: collecting logs from CloudWatch Logs using Lambda Promtail. But in the process of migrating to Kubernetes, we have Application Load Balancers that can only write logs to S3, and we need to learn how to collect… Read More »

Terraform: building EKS, part 3 – Karpenter installation

16 September 2023

 This is the third part of deploying an AWS Elastic Kubernetes Service cluster with Terraform, in which we will add Karpenter to our cluster. I’ve decided to post this separately because it’s quite a long post. And in the next and final (hopefully!), the fourth part, we will add the rest – all kinds of… Read More »

AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity

19 August 2023

  Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an… Read More »

AWS: CDK and Python – building an EKS cluster, and general impressions of CDK

22 July 2023

  So, Terraform is great, but so far in our project, we have decided to create the first AWS EKS clusters using the AWS CDK, because firstly, it is already on the project, and secondly, it is very interesting to try a new tool. Today we will see what came out of it, and how… Read More »

GitLab: gitlab-shell timeouts, and /metrics Connection refused

25 April 2023

  After running our self-hosted GitLab in production, we faced a bug: during git clone/pull/push operations, the request sometimes hung for 1-2 minutes. It looked like some kind of “floating” bug, that is, it could normally work 5 times, and then hangs once. The issues gitlab-shell timeouts For example, one time git clone works well:… Read More »

Kubernetes: Minikube, and a LoadBalancer in the Pending status

5 October 2022

  After running Pritunl in Minikube, it is not possible to connect to the VPN: … 2022-10-03 13:50:32 TCP/UDP: Preserving recently used remote address: [AF_INET] 2022-10-03 13:50:32 UDP link local: (not bound) 2022-10-03 13:50:32 UDP link remote: [AF_INET] … Check its Kubernetes Service: [simterm] $ kubectl -n pritunl-local get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)… Read More »

Arch Linux: pacman – error “HTTP server doesn’t seem to support byte ranges”

30 May 2022

 During system upgrade with pacman -Syu, I got internet disruption, and on the next upgrade I got the error from pacman. It sounds like “HTTP server doesn’t seem to support byte ranges. Cannot resume“: [simterm] … ==> Retrieving sources… -> Downloading clightd-5.6.tar.gz… ** Resuming transfer from byte position 12288 % Total % Received % Xferd… Read More »

Kubernetes: Helm – “x509: certificate signed by unknown authority”, and ServiceAccount for Pod

29 September 2021

 We have Github runners in our AWS Elastic Kubernetes service cluster, that are used to build Docker images and deploy them with Helm or ArgoCD. On the first helm install run in a Github runner’s Pod, we are getting the “x509: certificate signed by unknown authority” error: [simterm] # helm –kube-apiserver=https://kubernetes.default.svc.cluster.local list Error: Kubernetes cluster unreachable:… Read More »