Category Archives: Troubleshooting

Various problems solutions

AWS: Cost Explorer – costs checking on the CloudWatch Logs example

7 September 2021
 

 This post isn’t an overview of the AWS Cost Explorer service, but just a quick example of how to detect costs for your AWS account and its services. So, when I came back from the vacation, I noticed that we’ve paid more than usually for the AWS CloudWatch for August – almost 50 dollars per… Read More »

Tableau: install Tableau Bridge to access a database server in a private network

9 August 2021
 

 To access a database server, that has no public access (as it must be – access only inside an AWS VPC), Tableau suggests using its tool called Tableau Bridge. The idea is to have a Bridge service running in a network, which has access to a database server via its Private IP. Also, Bridge will… Read More »

Let’s Encrypt: SSL and the “SERVFAIL looking up CAA for domain.com” error

29 July 2021
 

 One of mine website stopped working with the “Connection reset.” error. NGINX configs seem to be correct, and other sites on the same server are working. NGINX also gave nothing, no errors, PHP-FPM also are good. Let’s check the website with the curl: The cause Because the error above are reported from the SSL_connect call,… Read More »

AWS: Simple Email Service Bounce rate and monitoring with and Prometheus

14 July 2021
 

 Recently, AWS blocked our AWS Simple Email Service because of its low bounce rate. This can be checked in the AWS SES > Reputation Dashboard, our account currently has Under review status: After we’ve connected AWS Tech Support, they enabled it back, but we must solve the issue asap, and have to monitor AWS SES… Read More »

ArgoCD: Okta integration, and user groups

17 May 2021
 

 In the previous post ArgoCD: users, access, and RBAC we’ve checked how to manage users and their permissions in ArgoCD, now let’s add an SSO authentification. The idea is that we don’t add user accounts locally in the ArgoCD’s ConfigMap, but instead will use our Okta users databases and Okta will perform their authentication. And… Read More »

Kubernetes: spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’

5 May 2021
 

 During applications deploy from a Helm chart described in the Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS we are getting the “spec.ports[0].nodePort: Forbidden: may not be used when `type` is ‘ClusterIP’” error. Let’s reproduce it and find solutions with kubectl and Helm to solve it. The “spec.ports[0].nodePort: Forbidden: may not be… Read More »

Istio: shared Ingress/AWS ALB, Helm chart with conditions, Istio, and ExternalDNS

27 April 2021
 

 Let’s proceed with our journey with Istio. Previous parts: Istio: an overview and running Service Mesh in Kubernetes Istio: external AWS Application LoadBalancer and Istio Ingress Gateway Besides Istio, in this post, we will also configure ExternalDNS, see the Kubernetes: update AWS Route53 DNS from an Ingress for details. Everything described below is a kind… Read More »

Istio: a cause and solution of the “SQLSTATE Connection refused”

23 April 2021
 

 During starting a pod we got errors “SQLSTATE[HY000] [2002] Connection refused’” in two different applications – a РНР/Yii and NodeJS. In the РHР/Yii it’s coming when we are running a pre-install hook during deployment with Helm and on the MySQL Migration Job execution: Yii Migration Tool (based on Yii v2.0.38)Exception ‘yii\db\Exception’ with message ‘SQLSTATE[HY000] [2002]… Read More »

Kubernetes: metrics-server – “401 Unauthorized” amd “kubelet stopped posting node status”

23 April 2021
 

 On our AWS Elastic Kubernetes Service Dev cluster, we got a couple of namespaees hanging i nthe Termination state. “401 Unauthorized”, response: “Unauthorized” Remembering a similar issue where the root cause was the metrics-server (see the Kubernetes: namespace hangs in Terminating and metrics-server non-obviousness post for details), the first thing I did was to check… Read More »

Kubernetes: NodeLocal DNS and the “lookup istiod.istio-system.svc on lookup: no such host” error

19 April 2021
 

 In our Deployments, we are using the NodeLocal DNS as a local DNS cache to reduce requests number to the AWS VPC DNS, see the Kubernetes: load-testing and high-load tuning – problems and solutions for details. Currently, a manifest looks like the next: … dnsPolicy: “None” dnsConfig: nameservers: – 169.254.20.10 … The problem is, that… Read More »