Yearly Archives: 2021

Kubernetes: Cluster Autoscaler – failed to renew lease
0 (0)

7 April 2021

We have a Kubernetes Cluster Autoscaler for AWS EC2 WorkerNode groups scaling. On our Dev cluster sometimes it stop working with the following message in its logs: [simterm] … E0331 08:57:52.264549 1 leaderelection.go:320] error retrieving resource lock kube-system/cluster-autoscaler: Get https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-autoscaler: context deadline exceeded I0331 08:58:14.468096 1 leaderelection.go:277] failed to renew lease kube-system/cluster-autoscaler: timed out waiting… Read More: Kubernetes: Cluster Autoscaler – failed to renew lease0 (0) »

Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness
0 (0)

1 April 2021

Faced with a very interesting thing during removal of a Kubernetes Namespace. After a kubectl delete namespace NAMESPACE is executed, the namespace hangs in the Terminating state, and any attempt to forcibly remove it didn’t help. First, let’s see how such a force-removal can be done, and then will check the real cause and a… Read More: Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness0 (0) »

Opsgenie: integration with AWS RDS and alerting
0 (0)

18 March 2021

Let’s configure Opsgenie with AWS RDS. The idea is to get notifications from RDS about events and send them to Opsgenie which will send them to our Slack. To do so, we need to configure AWS Simple Notification Service and AWS RDS Event subscriptions. The official documentation is here>>>. Opsgenie confiuration Go to the Integrations… Read More: Opsgenie: integration with AWS RDS and alerting0 (0) »

Windows: Windows Subsystem for Linux and Ubuntu Linux installation
0 (0)

14 March 2021

Windows Subsystem for Linux (WSL) is a Windows subsystem that helps to run various Linux distribution directly from the Windows OS. WSL version 2 which is the default version now uses the Hyper-V virtualization under the hood to install and run Linux and share files and devices between both systems. Windows Subsystem for Linux To… Read More: Windows: Windows Subsystem for Linux and Ubuntu Linux installation0 (0) »

Kubernetes: what is Endpoints
0 (0)

13 March 2021

Usually, we don’t see Endpoints objects when using Kubernetes Services, as they are working under the hood, similarly to ReplicaSets which are “hidden” behind Kubernetes Deployments. Kubernetes Service So, Service is a Kubernetes abstraction that uses labels to chose pods to route traffic to, see the Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services, and Ingress… Read More: Kubernetes: what is Endpoints0 (0) »

AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB Ingress controller
0 (0)

4 March 2021

One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS). Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller. Errors in the controller’s logs were: E0304 07:12:38.595113 1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key… Read More: AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB… »

Jenkins: running workers in Kubernetes and Docker images build
0 (0)

27 February 2021

We have a Jenkins instance that is running jobs in Docker containers on its host. Eventually, we’ve faced with an issue when the current AWS Ec2 instance t2.2xlarge (8 CPU, 32 RAM) during peak workload periods was too overloaded – not enough CPU time, not enough memory. So, the first solution could be to proceed… Read More: Jenkins: running workers in Kubernetes and Docker images build0 (0) »

Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors
0 (0)

25 February 2021

We have a PHP application running with Kubernetes in pods with two dedicated containers – NGINX и PHP-FPM. The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections. So, in this post, we will take a closer look at the pods’… Read More: Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors0 (0) »

Yandex.Tank: load testing tool – an overview, configuration, and examples
0 (0)

10 February 2021

Besides the Apache Bench and JMeter there is another utility – Yandex Tank. It’s used by our QA team and now it’s time for me to take a closer look on it to test one issue with our application running on a Kubernetes cluster. In this post a short overview of its capabilities and configuration.… Read More: Yandex.Tank: load testing tool – an overview, configuration, and examples0… »

Linux: setlocale: LC_ALL: cannot change locale (en_US.utf8) and Cyrillic symbols
0 (0)

4 February 2021

Locales is a set of environment variables that are used to determine how to display data and time (for example, first of the week), symbols encoding (for example, how to display cyrillic symbols), default files order when one executing the ls command, and so on. Those variables are: LANG: Determines the default locale in the… Read More: Linux: setlocale: LC_ALL: cannot change locale (en_US.utf8) and Cyrillic symbols0… »