Category Archives: Loki

Loki – Prometheus-inspired logging for cloud natives

Grafana: values ​​from records in Loki logs, and dual-Y-axes panels in Grafana

19 August 2023
 

  We have a function in AWS Lambda, that is writing logs to CloudWatch Logs, from where with the lambda-promtail we are getting them to a Grafana Loki instance to use them in Grafana graphs. What the task is: in the logs, we have records about “Init duration” and “Max Memory Used” by Lambdas. There are… Read More »

Grafana Loki: performance optimization with Recording Rules, caching, and parallel queries

19 August 2023
 

  So, we have Loki installed from the chart in simple-scale mode, see Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Loki is runnings on an AWS Elastic Kubernetes Service cluster, installed with Loki Helm chart, AWS S3 is used as a long-term store, and BoltDB Shipper is used to… Read More »

AWS: Grafana Loki, InterZone traffic in AWS, and Kubernetes nodeAffinity

19 August 2023
 

  Traffic in AWS is generally quite an interesting and sometimes complicated thing, I once wrote about it in the AWS: Cost optimization – services expenses overview and traffic costs in AWS. Now, it’s time to return to this topic again. So, what’s the problem: in AWS Cost Explorer, I’ve noticed that we have an… Read More »

VictoriaMetrics: deploying a Kubernetes monitoring stack

23 July 2023
 

  Now we have VictoriaMetrics + Grafana on a regular EC2 instance, launched with Docker Compose, see the VictoriaMetrics: an overview and its use instead of Prometheus. It was kind of a Proof of Concept, and it’s time to launch it “in an adult way” – in Kubernetes and all the configurations stored in a… Read More »

Loki: collecting logs from CloudWatch Logs using Lambda Promtail

20 May 2023
 

  Collecting logs in Grafana Loki with Kubernetes is very simple – we just launch Promtail in DaemonSet, configure it to read all data from /var/logs – and that’s it (in fact, we don’t specify anything at all – everything works out of the box from the Helm chart). But what about CloudWatch Logs? On my new… Read More »

Grafana Loki: alerts from the Loki Ruler and labels from logs

12 March 2023
 

  For general information on Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper. Among other services that make up Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple:… Read More »

Grafana Loki: alerts from Ruler and labels from logs

7 January 2023
 

  For general information about Grafana Loki, see the Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper post. Among other components of the Loki, there is a separate service called ruler that is responsible for working with alerts that can be generated directly from logs. The idea is very simple:… Read More »

Grafana Loki: architecture and running in Kubernetes with AWS S3 storage and boltdb-shipper

7 January 2023
 

  The last time I worked with Loki when it was still in Beta, and it looked much simpler then than it does now. In the new project, there is no logging system at all, and since we all love the Grafana stack, we also decided to use Loki for logging. Although to be honest,… Read More »

Grafana: Loki – the LogQL’s Prometheus-like counters, aggregation functions and dnsmasq’s requests graphs

17 November 2019
 

 The last time I configured Loki for logs collecting and monitoring was in February 2019 – almost a year ago, see the Grafana Labs: Loki – logs collecting and monitoring system post, when Loki was in its Beta state. Now we faced with outgoing traffic issues in our Production environments and can’t find who guilty for… Read More »