Category Archives: Troubleshooting

Various problems solutions

Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness

1 April 2021
 

 Faced with a very interesting thing during removal of a Kubernetes Namespace. After a kubectl delete namespace NAMESPACE is executed, the namespace hangs in the Terminating state, and any attempt to forcibly remove it didn’t help. First, let’s see how such a force-removal can be done, and then will check the real cause and a… Read More »

AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB Ingress controller

4 March 2021
 

 One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS). Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller. Errors in the controller’s logs were: E0304 07:12:38.595113       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key… Read More »

Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors

25 February 2021
 

 We have a PHP application running with Kubernetes in pods with two dedicated containers – NGINX и PHP-FPM. The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections. So, in this post, we will take a closer look at the pods’… Read More »

Linux: setlocale: LC_ALL: cannot change locale (en_US.utf8) and Cyrillic symbols

4 February 2021
 

 Locales is a set of environment variables that are used to determine how to display data and time (for example, first of the week), symbols encoding (for example, how to display cyrillic symbols), default files order when one executing the ls command, and so on. Those variables are: LANG: Determines the default locale in the… Read More »

ArgoCD: an overview, SSL configuration, and an application deploy

22 November 2020
 

 ArgoCD helps to deliver applications to Kubernetes by using the GitOps approach, i.e. when a Git-repository is used as a source of trust, thus all manifest, configs and other data are stored in a repository. It can b used with Kubernetes manifest, kustomize, ksonnet, jsonnet, and what we are using in our project – Helm-charts.… Read More »

Linux: LEMP set up – NGINX, PHP, MySQL, SSL, monitoring, logs, and a WordPress blog migration

6 November 2020
 

 Finally got time to migrate the RTFM.CO.UA blog to a new server with Debian 10. This time manually, without any automation will set up a LEMP stack Wrote a similar at 2016 – Debian: установка LEMP — NGINX + PHP-FPM + MariaDB (Rus), but in time the post is more complete of the process and… Read More »

Kubernetes: running SQL migrations with Kubernetes Job and Helm hook

27 October 2020
 

 We have a project running in Kubernetes that needs to run SQL migrations during deployment. To run migrations need to clone a Github repository and run actually migrations stored in it. Currently, this is done with Kubernetes initContainers , and there are two of them – the first one with git clones the repository with… Read More »

Kubernetes: Evicted pods and Pods Quality of Service

3 October 2020
 

 We have a Kubernetes cluster running on AWS Elastic Kubernetes Service. In this cluster, we have an application which is usually working fine but sometimes our monitoring system notifies about unhealthy pods: Check pods: [simterm] $ kk -n eks-prod-1-web-projectname-admin-backend-ns get pod NAME READY STATUS RESTARTS AGE bttrm-web-projectname-admin-backend-64648597fc-9j29n 1/1 Running 0 43m bttrm-web-projectname-admin-backend-64648597fc-kptjj 1/1 Running 0… Read More »

Slack: Cannot read property ‘appTeams’ of undefined

23 September 2020
 

 I have Slack desktop running on Arch Linux. It was installed from AUR and usually works fine. But today after rebooting my laptop Slack won’t start again sying it has the ‘appTeams’ property error: [simterm] $ slack Initializing local storage instance at path: /home/setevoy/.config/Slack/local-settings.json Failed to get path for key, this may be expected: /home/setevoy/.config/autostart… Read More »

AWS Elastic Kubernetes Service: load-testing and high-load tuning – problems and solutions

4 September 2020
 

 Actually, this post was planned as a short note about using NodeAffinity for Kubernetes Pod: But then, as often happens, after starting writing about one thing, I faced another, and then another one, and as a result – I made this long-read post about Kubernetes load-testing. So, I’ve started about NodeAffinity, but then wondered how… Read More »