We continue to develop our monitoring stack. See the first part – VictoriaMetrics: creating a Kubernetes monitoring stack with its own Helm chart. What do we want to do next: give access to developers so that they can set Silence for alerts themselves in Alertmanager to avoid spamming Slack, see Prometheus: Alertmanager Web UI alerts… Read More »
On all my previous projects where Kubernetes was, for its WorkerNodes scaling I’ve used the Cluster Autoscaler (CAS) because actually there were no other options before. In general, CAS worked well, but in November 2020 AWS released its own solution for scaling nodes in EKS – the Karpenter, and if at first time reviews… Read More »
Kubernetes allows very flexible control over how its Pods will be located on servers, i.e. WorkerNodes. This can be useful if you need to run a pod on a specific node configuration, for example – a WorkerNode must have a GPU, or an SSD instead of an HDD. Another example is when you need… Read More »
We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are… Read More »
To improve the performance of Grafana Loki, it is necessary to install several almost identical instances of Memcached, see Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries. The monitoring stack itself is deployed from one Helm chart, which dependencies – Promtail, Loki, etc – are added through the dependency field of… Read More »
So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to… Read More »
Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an… Read More »
I updated my Arch Linux yesterday, and for the first time in 9 years of using this system, I encountered an error when after a reboot the system could not mount a disk: ERROR: device UUID not found. mount: /new_root: can’t find UUID. ERROR: Failed to mount UUID on real root. You are now being… Read More »
Now we have VictoriaMetrics + Grafana on a regular EC2 instance, launched with Docker Compose, see the VictoriaMetrics: an overview and its use instead of Prometheus. It was kind of a Proof of Concept, and it’s time to launch it “in an adult way” – in Kubernetes and all the configurations stored in a… Read More »
Let’s continue with AWS CDK and Python. I’m not writing because I like CDK, but because there are quite a few examples on the Internet for AWS CDK with Python, so let them at least be here. So, we have a cluster – AWS: CDK and Python – building an EKS cluster, and general… Read More »