Tracing allows you to track requests between components, that is, for example, when using AWS and Kubernetes we can trace the entire path of a request from AWS Load Balancer to Kubernetes Pod and to DynamoDB or RDS. This helps us both to track performance issues – where and which requests are taking a long… Read More »
I didn’t plan this post at all as I thought I would do it quickly, but it didn’t work out quickly, and I need to dig a little deeper into this topic. So, what we are talking about: we have AWS Load Balancers, logs from which are collected to Grafana Loki, see. Grafana Loki: collecting… Read More »
In the Prometheus: running Pushgateway on Kubernetes with Helm and Terraform post I wrote about how to add Pushgateway to Prometheus, which allows using the Push model instead of Pull, that is, an Exporter can send metrics directly to the database instead of waiting for Prometheus or VMAgent to come to it. With VictoriaMetrics, it’s… Read More »
We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are… Read More »
Recently, I got a new interesting task – to build a dashboard in Grafana that would display the status of our development process and its performance, that is, the efficiency of our DevOps processes. This is necessary because we are trying to build “true continuous deployment” so that the code automatically enters Production, and… Read More »
In addition to the Horizontal Pod Autoscaler (HPA), which creates additional pods if the existing ones start using more CPU/Memory than configured in the HPA limits, there is also the Vertical Pod Autoscaler (VPA), which works according to a different scheme: instead of horizontal scaling, i.e. increasing the number of Pods, it changes resources.requests of a Pod,… Read More »
We have a lot of AWS Lambda functions in the project, and developers want to be able to send metrics to our Prometheus to add their own alerts and graphs in Grafana. For this, the functions use the Prometheus library, which allows these metrics to be created (see Prometheus: Building a Custom Prometheus Exporter… Read More »
After running our self-hosted GitLab in production, we faced a bug: during git clone/pull/push operations, the request sometimes hung for 1-2 minutes. It looked like some kind of “floating” bug, that is, it could normally work 5 times, and then hangs once. The issues gitlab-shell timeouts For example, one time git clone works well:… Read More »
So, let’s continue our journey with migrating GitLab to Kubernetes. See previous parts: GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube GitLab: Helm chart of values, dependencies, and deployment in Kubernetes with AWS S3 GitLab: міграція даних з GitLab cloud та процес backup-restore у self-hosted версії в Kubernetes In general, everything is working,… Read More »
Good – we learned how to launch Loki – Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and bolted-shipper, we also figured out how to configure alerts – Grafana Loki: alerts from the Loki Ruler and labels from logs. Now it’s time to figure out what we can do in Loki using… Read More »