Category Archives: Kubernetes

Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management.

Kubernetes: ExternalDNS – records retrieval failed: failed to list hosted zones: Throttling: status code: 400

9 April 2021
 

 We have an ExternalDNS service running, see the Kubernetes: update AWS Route53 DNS from an Ingress post, which started sending a lot of messages like: msg=”failed to list resource records sets for zone /hostedzone/Z2VM3W5SRY4I9J: Throttling: \n\tstatus code: 400 And even AWS Console in the Route53 says “Throttling error that was caused because API rate was… Read More »

Kubernetes: Cluster Autoscaler – failed to renew lease

7 April 2021
 

 We have a Kubernetes Cluster Autoscaler for AWS EC2 WorkerNode groups scaling. On our Dev cluster sometimes it stop working with the following message in its logs: [simterm] … E0331 08:57:52.264549 1 leaderelection.go:320] error retrieving resource lock kube-system/cluster-autoscaler: Get https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cluster-autoscaler: context deadline exceeded I0331 08:58:14.468096 1 leaderelection.go:277] failed to renew lease kube-system/cluster-autoscaler: timed out waiting… Read More »

Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness

1 April 2021
 

 Faced with a very interesting thing during removal of a Kubernetes Namespace. After a kubectl delete namespace NAMESPACE is executed, the namespace hangs in the Terminating state, and any attempt to forcibly remove it didn’t help. First, let’s see how such a force-removal can be done, and then will check the real cause and a… Read More »

Kubernetes: what is Endpoints

13 March 2021
 

 Usually, we don’t see Endpoints objects when using Kubernetes Services, as they are working under the hood, similarly to ReplicaSets which are “hidden” behind Kubernetes Deployments. Kubernetes Service So, Service is a Kubernetes abstraction that uses labels to chose pods to route traffic to, see the Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services, and Ingress… Read More »

AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB Ingress controller

4 March 2021
 

 One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS). Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller. Errors in the controller’s logs were: E0304 07:12:38.595113       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key… Read More »

Jenkins: running workers in Kubernetes and Docker images build

27 February 2021
 

 We have a Jenkins instance that is running jobs in Docker containers on its host. Eventually, we’ve faced with an issue when the current AWS Ec2 instance t2.2xlarge (8 CPU, 32 RAM) during peak workload periods was too overloaded – not enough CPU time, not enough memory. So, the first solution could be to proceed… Read More »

Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors

25 February 2021
 

 We have a PHP application running with Kubernetes in pods with two dedicated containers – NGINX и PHP-FPM. The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections. So, in this post, we will take a closer look at the pods’… Read More »

Logz.io: collection logs from Kubernetes – fluentd vs filebeat

1 February 2021
 

 We are using Logz.io to collect our Kubernetes cluster logs (also, there is a local Loki instance). Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here – logzio-k8s. The problem we faced is that those pods are consuming… Read More »

ArgoCD: a Helm chart deployment, and working with Helm Secrets via AWS KMS

22 November 2020
 

 In the previous post ArgoCD: an overview, SSL configuration, and an application deploy we did a quick overview on how to work with the ArgoCD in general, and now let’s try to deploy a Helm chart. The most interesting part of this is how to enable the Helm Secrets. Had some pain with this, but… Read More »

ArgoCD: an overview, SSL configuration, and an application deploy

22 November 2020
 

 ArgoCD helps to deliver applications to Kubernetes by using the GitOps approach, i.e. when a Git-repository is used as a source of trust, thus all manifest, configs and other data are stored in a repository. It can b used with Kubernetes manifest, kustomize, ksonnet, jsonnet, and what we are using in our project – Helm-charts.… Read More »