Grafana 6.0 was still in Beta on Feb 2019, when the new feature was introduced – Loki, a log aggregation system available via another new ability –
It’s similar to well-known
It’s still in the early development stage and main disadvantage which I found during an attempt to set up the Proof of Concept described in this post is the lack of documentation and examples.
Thus – in this and a couple of next posts I’ll try to show a few its setup and usage examples.
In any case – I liked Loki, especially because you will no need to setup whole dedicated ELK stack: the same Grafana can be used for your monitoring dashboards and you’ll all your logs available via the same Grafana instance. So I’ll try to implement it on my current project.
Or will go back to the ELK solution…)
Project’s home page –
There is also a good discussion on the Reddit
Loki has thelogcli
Loki-stack consists of three main components:
promtail
– agent to collect logs on a host and push them to a Loki instanceloki
– logs aggregation and processing serverGrafana – for querying and displaying logs
Loki Proof of Concept setup
Now let’s try to set up it, add some NGINX logs and to check how it will look like in Grafana.
Stack creation
Create Loki’s home directory:
Next, create a Docker Compose file – loki-compose.yml
:
version: "3" networks: loki: services: loki: image: grafana/loki:master ports: - "3100:3100" command: -config.file=/etc/loki/local-config.yaml networks: - loki promtail: image: grafana/promtail:master volumes: - /var/log:/var/log command: -config.file=/etc/promtail/docker-config.yaml networks: - loki grafana: image: grafana/grafana:master ports: - "3000:3000" networks: - loki
Start containers:
In the promtail
‘s output, you’ll see information about logs watched by promatil
:
…
promtail_1 | level=info ts=2019-02-05T14:07:30.777577365Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”
promtail_1 | level=info ts=2019-02-05T14:07:30.779215957Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log
promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/alternatives.log – &{Offset:19515 Whence:0}
promtail_1 | level=info ts=2019-02-05T14:07:30.782000937Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log
promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/auth.log – &{Offset:3869 Whence:0}
promtail_1 | level=info ts=2019-02-05T14:07:30.782899496Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log
promtail_1 | level=info ts=2019-02-05T14:07:30.783409465Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log
promtail_1 | level=info ts=2019-02-05T14:07:30.78359605Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log
promtail_1 | level=info ts=2019-02-05T14:07:30.783648219Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log
promtail_1 | level=info ts=2019-02-05T14:07:30.784386768Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log
promtail_1 | level=info ts=2019-02-05T14:07:30.785801248Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog
promtail_1 | level=info ts=2019-02-05T14:07:30.785871765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log
promtail_1 | level=info ts=2019-02-05T14:07:30.785943765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog
promtail_1 | level=info ts=2019-02-05T14:07:30.78598882Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog
promtail_1 | level=info ts=2019-02-05T14:07:30.786064399Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log
…
Similarly to Prometheus exporters – you can access data via the /metrics
URI:
Now you can access your Grafana instance via http://IP:3000 URL:
Login with the default credentials – admin:admin.
Adding Loki data source
Grafana will suggest you to add a data source:
Add the Loki data source:
Set Loki’s URL as http://loki:3100 (as we are running our Loki’s stack in the Docker’s own network):
And now you can navigate to the Explore:
Check some already collected logs via the Log labels:
Adding new logs
Logs to be tailed are configured in the promtail
‘s /etc/promtail/local-config.yaml
/etc/promtail/docker-config.yaml
Now let’s install NGINX:
Check if log-files appeared on the host:
Loki’s configuration file stored inside the container at this moment in the /etc/loki
directory:
promtail
‘s – in /etc/promtail
:
Let’s create own configuration for promtail
– /opt/loki/loki-promtail-conf.yml
:
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /tmp/positions.yaml client: url: http://loki:3100/api/prom/push scrape_configs: - job_name: system entry_parser: raw static_configs: - targets: - localhost labels: job: varlogs __path__: /var/log/*log - job_name: nginx entry_parser: raw static_configs: - targets: - localhost labels: job: nginx __path__: /var/log/nginx/*log
Here in the scrape_configs
we added a new job – nginx with the __path__
to the directory and files to be tailed by the promtail
.
Next, let’s update our Compose file and add the new config file mapping from our host inside the promtail
‘s container as /etc/promtail/docker-config.yaml
:
... promtail: image: grafana/promtail:master volumes: - /opt/loki/loki-promtail-conf.yml:/etc/promtail/docker-config.yaml - /var/log:/var/log command: -config.file=/etc/promtail/docker-config.yaml networks: - loki ...
Restart Compose stack and check logs in the promtail
‘s output to see which logs are watched by the promtail
now:
…
promtail_1 | level=info ts=2019-02-05T15:08:10.53723788Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”
promtail_1 | level=info ts=2019-02-05T15:08:10.538575763Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log
promtail_1 | level=info ts=2019-02-05T15:08:10.539914281Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log
promtail_1 | level=info ts=2019-02-05T15:08:10.541431662Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log
promtail_1 | level=info ts=2019-02-05T15:08:10.541911162Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log
promtail_1 | level=info ts=2019-02-05T15:08:10.542352516Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log
promtail_1 | level=info ts=2019-02-05T15:08:10.542780189Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log
promtail_1 | level=info ts=2019-02-05T15:08:10.543222436Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log
promtail_1 | level=info ts=2019-02-05T15:08:10.54394579Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”nginx\”}”
promtail_1 | level=info ts=2019-02-05T15:08:10.54473202Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog
promtail_1 | level=info ts=2019-02-05T15:08:10.545166611Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log
promtail_1 | level=info ts=2019-02-05T15:08:10.54564307Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog
promtail_1 | level=info ts=2019-02-05T15:08:10.546097865Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog
promtail_1 | level=info ts=2019-02-05T15:08:10.54651553Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/alternatives.log – &{Offset:0 Whence:0}
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/auth.log – &{Offset:0 Whence:0}
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/access.log – &{Offset:0 Whence:0}
promtail_1 | level=info ts=2019-02-05T15:08:10.548790452Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/access.log
promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/error.log – &{Offset:0 Whence:0}
promtail_1 | level=info ts=2019-02-05T15:08:10.54959973Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/error.log
…
key=”{job=\”nginx\”}” appeared – all good.
Let it work for a couple of minutes and check inside Grafana:
Logs present and we are done for now.
Check the next post – Grafana Labs: Loki – distributed system, labels, and filters to see an example of how to set up a distributed system and tags and filters usage example.
Similar posts
Also published on