Grafana Labs: Loki – logs collecting and monitoring system

Grafana 6.0 was still in Beta on Feb 2019, when the new feature was introduced – Loki, a  log aggregation system available via another new ability – Explore.

It’s similar to well-known ELK/EFK stack but more simple to set up and use and is intended to be used mostly with clouds and systems like Prometheus and Kubernetes.

It’s still in the early development stage and main disadvantage which I found during an attempt to set up the Proof of Concept described in this post is the lack of documentation and examples.

Thus – in this and a couple of next posts I’ll try to show a few its setup and usage examples.

In any case – I liked Loki, especially because you will no need to setup whole dedicated ELK stack: the same Grafana can be used for your monitoring dashboards and you’ll all your logs available via the same Grafana instance. So I’ll try to implement it on my current project.

Or will go back to the ELK solution…)

Project’s home page – here>>>, post in Grafana Lab’s blog – here>>>, and the Github repository here>>>.

There is also a good discussion on the Reddit here>>>.

Loki has thelogcli tool – but I didn’t check it yet.

Loki-stack consists of three main components:

• promtail – agent to collect logs on a host and push them to a Loki instance
• loki – logs aggregation and processing server
• Grafana – for querying and displaying logs

Loki Proof of Concept setup

Now let’s try to set up it, add some NGINX logs and to check how it will look like in Grafana.

Stack creation

Create Loki’s home directory:

Next, create a Docker Compose file – loki-compose.yml:

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:master
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:master
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/docker-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"
    networks:
      - loki

Start containers:

In the promtail‘s output, you’ll see information about logs watched by promatil:

…
promtail_1 | level=info ts=2019-02-05T14:07:30.777577365Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”
promtail_1 | level=info ts=2019-02-05T14:07:30.779215957Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log
promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/alternatives.log – &{Offset:19515 Whence:0}
promtail_1 | level=info ts=2019-02-05T14:07:30.782000937Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log
promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/auth.log – &{Offset:3869 Whence:0}
promtail_1 | level=info ts=2019-02-05T14:07:30.782899496Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log
promtail_1 | level=info ts=2019-02-05T14:07:30.783409465Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log
promtail_1 | level=info ts=2019-02-05T14:07:30.78359605Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log
promtail_1 | level=info ts=2019-02-05T14:07:30.783648219Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log
promtail_1 | level=info ts=2019-02-05T14:07:30.784386768Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log
promtail_1 | level=info ts=2019-02-05T14:07:30.785801248Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog
promtail_1 | level=info ts=2019-02-05T14:07:30.785871765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log
promtail_1 | level=info ts=2019-02-05T14:07:30.785943765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog
promtail_1 | level=info ts=2019-02-05T14:07:30.78598882Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog
promtail_1 | level=info ts=2019-02-05T14:07:30.786064399Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log
…

Similarly to Prometheus exporters – you can access data via the /metrics URI:

Now you can access your Grafana instance via http://IP:3000 URL:

Login with the default credentials – admin:admin.

Adding Loki data source

Grafana will suggest you to add a data source:

Add the Loki data source:

Set Loki’s URL as http://loki:3100 (as we are running our Loki’s stack in the Docker’s own network):

And now you can navigate to the Explore:

Check some already collected logs via the Log labels:

Adding new logs 

Logs to be tailed are configured in the promtail‘s /etc/promtail/local-config.yaml or /etc/promtail/docker-config.yamlconfigs.

Now let’s install NGINX:

Check if log-files appeared on the host:

Loki’s configuration file stored inside the container at this moment in the /etc/loki directory:

promtail‘s – in /etc/promtail:

Let’s create own configuration for promtail– /opt/loki/loki-promtail-conf.yml:

server:

  http_listen_port: 9080
  grpc_listen_port: 0

positions:

  filename: /tmp/positions.yaml

client:

  url: http://loki:3100/api/prom/push

scrape_configs:

  - job_name: system
    entry_parser: raw
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        __path__: /var/log/*log

  - job_name: nginx
    entry_parser: raw
    static_configs:
    - targets:
        - localhost
      labels:
        job: nginx
        __path__: /var/log/nginx/*log

Here in the scrape_configs we added a new job – nginx with the __path__ to the directory and files to be tailed by the promtail.

Next, let’s update our Compose file and add the new config file mapping from our host inside the promtail‘s container as /etc/promtail/docker-config.yaml:

...
  promtail:
    image: grafana/promtail:master
    volumes:
      - /opt/loki/loki-promtail-conf.yml:/etc/promtail/docker-config.yaml
      - /var/log:/var/log
    command: -config.file=/etc/promtail/docker-config.yaml
    networks:
      - loki
...

Restart Compose stack and check logs in the promtail‘s output to see which logs are watched by the promtail now:

…
promtail_1 | level=info ts=2019-02-05T15:08:10.53723788Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”
promtail_1 | level=info ts=2019-02-05T15:08:10.538575763Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log
promtail_1 | level=info ts=2019-02-05T15:08:10.539914281Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log
promtail_1 | level=info ts=2019-02-05T15:08:10.541431662Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log
promtail_1 | level=info ts=2019-02-05T15:08:10.541911162Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log
promtail_1 | level=info ts=2019-02-05T15:08:10.542352516Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log
promtail_1 | level=info ts=2019-02-05T15:08:10.542780189Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log
promtail_1 | level=info ts=2019-02-05T15:08:10.543222436Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log
promtail_1 | level=info ts=2019-02-05T15:08:10.54394579Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”nginx\”}”
promtail_1 | level=info ts=2019-02-05T15:08:10.54473202Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog
promtail_1 | level=info ts=2019-02-05T15:08:10.545166611Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log
promtail_1 | level=info ts=2019-02-05T15:08:10.54564307Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog
promtail_1 | level=info ts=2019-02-05T15:08:10.546097865Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog
promtail_1 | level=info ts=2019-02-05T15:08:10.54651553Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/alternatives.log – &{Offset:0 Whence:0}
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/auth.log – &{Offset:0 Whence:0}
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/access.log – &{Offset:0 Whence:0}
promtail_1 | level=info ts=2019-02-05T15:08:10.548790452Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/access.log
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/error.log – &{Offset:0 Whence:0}
promtail_1 | level=info ts=2019-02-05T15:08:10.54959973Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/error.log
…

key=”{job=\”nginx\”}” appeared – all good.

Let it work for a couple of minutes and check inside Grafana:

Logs present and we are done for now.

Check the next post – Grafana Labs: Loki – distributed system, labels, and filters to see an example of how to set up a distributed system and tags and filters usage example.

Also published on Medium.