Category Archives: Troubleshooting

Various problems solutions

Tableau: install Tableau Bridge to access a database server in a private network
0 (0)

9 August 2021

To access a database server, that has no public access (as it must be – access only inside an AWS VPC), Tableau suggests using its tool called Tableau Bridge. The idea is to have a Bridge service running in a network, which has access to a database server via its Private IP. Also, Bridge will… Read More »

Loading

Let’s Encrypt: SSL and the “SERVFAIL looking up CAA for domain.com” error
0 (0)

29 July 2021

One of mine website stopped working with the “Connection reset.” error. NGINX configs seem to be correct, and other sites on the same server are working. NGINX also gave nothing, no errors, PHP-FPM also are good. Let’s check the website with the curl: [simterm] $ curl -Iv https://example.setevoy.org.ua/ * Trying 139.59.205.180:443… * Connected to example.setevoy.org.ua… Read More »

Loading

AWS: Simple Email Service Bounce rate and monitoring with and Prometheus
0 (0)

14 July 2021

Recently, AWS blocked our AWS Simple Email Service because of its low bounce rate. This can be checked in the AWS SES > Reputation Dashboard, our account currently has Under review status: After we’ve connected AWS Tech Support, they enabled it back, but we must solve the issue asap, and have to monitor AWS SES… Read More »

Loading

ArgoCD: Okta integration, and user groups
0 (0)

17 May 2021

In the previous post ArgoCD: users, access, and RBAC we’ve checked how to manage users and their permissions in ArgoCD, now let’s add an SSO authentification. The idea is that we don’t add user accounts locally in the ArgoCD’s ConfigMap, but instead will use our Okta users databases and Okta will perform their authentication. And… Read More »

Loading

Kubernetes: spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’
0 (0)

5 May 2021

During applications deploy from a Helm chart described in the Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS we are getting the “spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’” error. Let’s reproduce it and find solutions with kubectl and Helm to solve it. The “spec.ports[0].nodePort: Forbidden: may not be… Read More »

Loading

Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS
0 (0)

27 April 2021

Let’s proceed with our journey with Istio. Previous parts: Istio: an overview and running Service Mesh in Kubernetes Istio: external AWS Application LoadBalancer and Istio Ingress Gateway Besides Istio, in this post, we will also configure ExternalDNS, see the Kubernetes: update AWS Route53 DNS from an Ingress for details. Everything described below is a kind… Read More »

Loading

Istio: a cause and solution of the “SQLSTATE Connection refused”
0 (0)

23 April 2021

During starting a pod we got errors “SQLSTATE[HY000] [2002] Connection refused’” in two different applications – a РНР/Yii and NodeJS. In the РHР/Yii it’s coming when we are running a pre-install hook during deployment with Helm and on the MySQL Migration Job execution: Yii Migration Tool (based on Yii v2.0.38)Exception ‘yii\db\Exception’ with message ‘SQLSTATE[HY000] [2002]… Read More »

Loading

Kubernetes: metrics-server – “401 Unauthorized” amd “kubelet stopped posting node status”
0 (0)

23 April 2021

On our AWS Elastic Kubernetes Service Dev cluster, we got a couple of namespaees hanging i nthe Termination state. “401 Unauthorized”, response: “Unauthorized” Remembering a similar issue where the root cause was the metrics-server (see the Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness post for details), the first thing I did was to check… Read More »

Loading

Kubernetes: NodeLocal DNS and the “lookup istiod.istio-system.svc on lookup: no such host” error
0 (0)

19 April 2021

In our Deployments, we are using the NodeLocal DNS as a local DNS cache to reduce requests number to the AWS VPC DNS, see the Kubernetes: load-testing and high-load tuning – problems and solutions for details. Currently, a manifest looks like the next: … dnsPolicy: “None” dnsConfig: nameservers: – 169.254.20.10 … The problem is, that… Read More »

Loading

Kubernetes: ExternalDNS – records retrieval failed: failed to list hosted zones: Throttling: status code: 400
0 (0)

9 April 2021

We have an ExternalDNS service running, see the Kubernetes: update AWS Route53 DNS from an Ingress post, which started sending a lot of messages like: msg=”failed to list resource records sets for zone /hostedzone/Z2VM3W5SRY4I9J: Throttling: \n\tstatus code: 400 And even AWS Console in the Route53 says “Throttling error that was caused because API rate was… Read More »

Loading