Category Archives: Troubleshooting

Various problems solutions

HTTP: redirects, POST and GET requests, and the “lost” data

23 November 2019
 

 We have a web application which has to accept POST-requests from clients. In front of this application, there is some proxy service, no matter which – initially, we faced the issues on an AWS’s Application Load Balancer, then I reproduced them with NGINX, and it will “work” for any other proxying system. Besides proxying –… Read More »

Grafana: Loki – the LogQL’s Prometheus-like counters, aggregation functions and dnsmasq’s requests graphs

17 November 2019
 

 The last time I configured Loki for logs collecting and monitoring was in February 2019 – almost a year ago, see the Grafana Labs: Loki – logs collecting and monitoring system post, when Loki was in its Beta state. Now we faced with outgoing traffic issues in our Production environments and can’t find who guilty for… Read More »

dnsmasq: AWS – “Temporary failure in name resolution”, logs, debug and dnsmasq cache size

28 October 2019
 

 We are using AWS VPC DNS and sometimes facing with errors like “php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution“. The only advice from AWS tech. support was to configure a local dnsmasq service to act as a local DNS cache, but I did this already year ago and this issue happens once in 1-2-3… Read More »

Arch Linux: package-query: error while loading shared libraries: libalpm.so.11

26 October 2019
 

 Didn’t install upgrades for a couple of weeks, today packages to be upgraded became over 100, so I run it. Upgrades on my Arch Linux are installed with a simple alias in the.bashrc: … alias osupgrade=”yaourt -Syua –noconfirm” … (seems eventually it’s time to change it to the alias osupgrade=”yay -Syua –noconfirm”) The error Usually,… Read More »

OpenVPN: the No route to host and ERR_ADDRESS_UNREACHABLE errors – the cause and a solution

14 October 2019
 

 We have an OpenVPN Access Server running, see its setup in the OpenVPN: OpenVPN Access Server set up and AWS VPC peering configuration post. The VPN server is hosted in a first AWS VPC, while a Bitwarden service – in another one VPC. Between those VPCs we have a VPC peering configured, and the OpenVPN… Read More »

Debian: logrotate won’t rotate logs with an “unknown group ‘syslog'” error

9 October 2019
 

 We have an AWS EC2 with Debian and logrotate. One day its root partition was exhausted and when I started investigating it – found, that we have a bunch of files like /var/log/syslog.N.gz. At the same time by default logrotate creates a config file to rotate syslog log files: [simterm] root@monitoring-dev:~# cat /etc/logrotate.d/syslog # Ansible… Read More »

PostgreSQL: FATAL: Peer authentication failed for user “name”

11 September 2019
 

 A quick note mainly for myself on how to enable remote access in PostgreSQL and create its database dump. An EC2-server moved to our AWS account with one website. During an attempt to connect to the PostgreSQL running on this host – the “FATAL: Peer authentication failed for user “name”” appears. And the same error… Read More »

Kubernetes: part 4 – AWS EKS authentification, aws-iam-authenticator and AWS IAM

3 September 2019
 

  Let’s proceed with our AWS Elastic Kubernetes Service, EKS. Previous parts: Kubernetes: part 1 – architecture and main components overview Kubernetes: part 2 – a cluster set up on AWS with AWS cloud-provider and AWS LoadBalancer Kubernetes: part 3 – AWS EKS overview and manual EKS cluster set up. In the previous – Kubernetes:… Read More »

SonarQube: solving “Unrecoverable indexation failures” and Elasticsearch “Disk watermark exceeded” errors

29 August 2019
 

 We are using SonarQube started from a Jenkins jobs. See more at SonarQube: running tests from Jenkins Pipeline in Docker. So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors: java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39) at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122) … And host… Read More »