Category Archives: Troubleshooting

Various problems solutions

Kubernetes: Evicted pods and Pods Quality of Service

3 October 2020
 

 We have a Kubernetes cluster running on AWS Elastic Kubernetes Service. In this cluster, we have an application which is usually working fine but sometimes our monitoring system notifies about unhealthy pods: Check pods: [simterm] $ kk -n eks-prod-1-web-projectname-admin-backend-ns get pod NAME READY STATUS RESTARTS AGE bttrm-web-projectname-admin-backend-64648597fc-9j29n 1/1 Running 0 43m bttrm-web-projectname-admin-backend-64648597fc-kptjj 1/1 Running 0… Read More »

Slack: Cannot read property ‘appTeams’ of undefined

23 September 2020
 

 I have Slack desktop running on Arch Linux. It was installed from AUR and usually works fine. But today after rebooting my laptop Slack won’t start again sying it has the ‘appTeams’ property error: [simterm] $ slack Initializing local storage instance at path: /home/setevoy/.config/Slack/local-settings.json Failed to get path for key, this may be expected: /home/setevoy/.config/autostart… Read More »

AWS Elastic Kubernetes Service: load-testing and high-load tuning – problems and solutions

4 September 2020
 

 Actually, this post was planned as a short note about using NodeAffinity for Kubernetes Pod: But then, as often happens, after starting writing about one thing, I faced another, and then another one, and as a result – I made this long-read post about Kubernetes load-testing. So, I’ve started about NodeAffinity, but then wondered how… Read More »

Kubernetes: a cluster’s monitoring with the Prometheus Operator

13 August 2020
 

 Continuing with the Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles, where we configured Prometheus manually to see how it’s working – now, let’s try to use Prometheus Operator installed via Helm chart. So, the task is spin up a Prometheus server and all necessary exporter in an AWS Elastic Kubernetes… Read More »

Kubernetes: HorizontalPodAutoscaler – an overview with examples

12 August 2020
 

 Kubernetes HorizontalPodAutoscaler automatically scales Kubernetes Pods under ReplicationController, Deployment, or ReplicaSet controllers basing on its CPU, memory, or other metrics. It was shortly discussed in the Kubernetes: running metrics-server in AWS EKS for a Kubernetes Pod AutoScaler post, now let’s go deeper to check all options available for scaling. For HPA you can use three… Read More »

Prometheus: yet-another-cloudwatch-exporter – collecting AWS CloudWatch metrics

23 July 2020
 

 Currently, to collect metrics from the AWS CloudWatch we are using AWS’s own cloudwatch-exporter, see the Prometheus: CloudWatch exporter — сбор метрик из AWS и графики в Grafana post (in Rus), but it has a few gaps: it’s written in Java, so uses CPU/memory of the monitoring host doesn’t scrapes AWS tags from resources uses… Read More »

Kubernetes: 503 no endpoints available for service – causes and solutions

15 June 2020
 

 We have a Redis service running behind a Service with the ClusterIP type. This Redis must accessible by pods from the same namespace (a Gorush service). The problem is that those pod can’t connect to the Redis service using its gorush-server-redis-svc:6379 name and reporting “Can’t connect redis server: connection refused“: [simterm] $ kk -n gorush-test… Read More »

Docker: configure tzdata and timezone during build

17 May 2020
 

 During a Docker image build – it stops asking to configure the tzdata. Dockerfile at this moment is the next: FROM ubuntu:18.04 RUN apt update && apt install -y python-pip python-dev ssh python-boto3 RUN pip install ansible==2.4.3.0 Let’s reproduce – run the build: [simterm] admin@jenkins-production:~$ docker build -t proj/proj-ansible:1.1 . Sending build context to Docker… Read More »

Helm: helm-secrets – sensitive data encryption with AWS KMS and use it with Jenkins

16 May 2020
 

 So, as a follow-up to the Helm: Kubernetes package manager – an overview, getting started post – let’s discuss about sensitive data in our Helm charts. What I want is to store a chart files in a repository, but even if such a repo will be a private Github repo – I still don’t want… Read More »

AWS Elastic Kubernetes Service: a cluster creation automation, part 2 – Ansible, eksctl

1 May 2020
 

 The first part – AWS Elastic Kubernetes Service: a cluster creation automation, part 1 – CloudFormation. To remind the whole idea is to create an automation process to create an EKS cluster: Ansible uses the cloudformation module to create an infrastructure by using an Outputs of the CloudFormation stack created – Ansible from a template will… Read More »