Category Archives: Troubleshooting

Various problems solutions

Kubernetes: spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’

5 May 2021
 

 During applications deploy from a Helm chart described in the Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS we are getting the “spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’” error. Let’s reproduce it and find solutions with kubectl and Helm to solve it. The “spec.ports[0].nodePort: Forbidden: may not be… Read More »

Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS

27 April 2021
 

 Let’s proceed with our journey with Istio. Previous parts: Istio: an overview and running Service Mesh in Kubernetes Istio: external AWS Application LoadBalancer and Istio Ingress Gateway Besides Istio, in this post, we will also configure ExternalDNS, see the Kubernetes: update AWS Route53 DNS from an Ingress for details. Everything described below is a kind… Read More »

Istio: a cause and solution of the “SQLSTATE Connection refused”

23 April 2021
 

 During starting a pod we got errors “SQLSTATE[HY000] [2002] Connection refused’” in two different applications – a РНР/Yii and NodeJS. In the РHР/Yii it’s coming when we are running a pre-install hook during deployment with Helm and on the MySQL Migration Job execution: Yii Migration Tool (based on Yii v2.0.38)Exception ‘yii\db\Exception’ with message ‘SQLSTATE[HY000] [2002]… Read More »

Kubernetes: metrics-server – “401 Unauthorized” amd “kubelet stopped posting node status”

23 April 2021
 

 On our AWS Elastic Kubernetes Service Dev cluster, we got a couple of namespaees hanging i nthe Termination state. “401 Unauthorized”, response: “Unauthorized” Remembering a similar issue where the root cause was the metrics-server (see the Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness post for details), the first thing I did was to check… Read More »

Kubernetes: NodeLocal DNS and the “lookup istiod.istio-system.svc on lookup: no such host” error

19 April 2021
 

 In our Deployments, we are using the NodeLocal DNS as a local DNS cache to reduce requests number to the AWS VPC DNS, see the Kubernetes: load-testing and high-load tuning – problems and solutions for details. Currently, a manifest looks like the next: … dnsPolicy: “None” dnsConfig: nameservers: – 169.254.20.10 … The problem is, that… Read More »

Kubernetes: ExternalDNS – records retrieval failed: failed to list hosted zones: Throttling: status code: 400

9 April 2021
 

 We have an ExternalDNS service running, see the Kubernetes: update AWS Route53 DNS from an Ingress post, which started sending a lot of messages like: msg=”failed to list resource records sets for zone /hostedzone/Z2VM3W5SRY4I9J: Throttling: \n\tstatus code: 400 And even AWS Console in the Route53 says “Throttling error that was caused because API rate was… Read More »

Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness

1 April 2021
 

 Faced with a very interesting thing during removal of a Kubernetes Namespace. After a kubectl delete namespace NAMESPACE is executed, the namespace hangs in the Terminating state, and any attempt to forcibly remove it didn’t help. First, let’s see how such a force-removal can be done, and then will check the real cause and a… Read More »

AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB Ingress controller

4 March 2021
 

 One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS). Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller. Errors in the controller’s logs were: E0304 07:12:38.595113       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key… Read More »

Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors

25 February 2021
 

 We have a PHP application running with Kubernetes in pods with two dedicated containers – NGINX и PHP-FPM. The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections. So, in this post, we will take a closer look at the pods’… Read More »