Grafana Loki: Like Prometheus, but for logs
Ganesh Vernekar, November 2019
@_codesome
About me
Presentation Title | 02
Ganesh Vernekar
Software Engineer, Grafana Labs
Prometheus Maintainer
Graduated this year from IIT Hyderabad
Twitter: @_codesome
Email: ganesh@grafana.com
03/18 Project started
12/18 Launched at KubeCon NA
12/18 #1 on HN for ~12hrs!
03/19 Labels from logs
04/19 Lazy Queries; LogQL filtering
04/19 KubeCon EU: context, live tailing
06/19 v-.1.0 Beta release!
https://github.com/grafana/loki
Grafana Loki: Prometheus, but for logs | 04
goo.gl/5DEVH6
Loki is horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus
#0 Simple to scale
#1 Integrated with existing observability tools
#2 Airplane friendly and Cloud native
Grafana Loki: Prometheus, but for logs | 05
#0 Simple to scale
Grafana Loki: Prometheus, but for logs
| 6
Grafana Loki: Prometheus, but for logs | 07
Existing log aggregation systems do full text indexing and support complex queries
DEwMGIwZ => {
time: “2018-01-31 15:41:04”,
job: “frontend”,
env: “dev”,
line: “POST /api/prom/push...”
}
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”
Grafana Loki: Prometheus, but for logs | 08
Existing log aggregation systems do full text indexing and support complex queries
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”
NodeN
…
Node1
Node0
{job=“frontend”, env=“dev”} => {
time: “2018-01-31 15:41:04”,
line: “POST /api/prom/push HTTP/1.1 502 0"
}
Grafana Loki: Prometheus, but for logs | 09
Loki doesn’t index the text of the logs, instead grouping entries into “streams” and indexing those with labels.
#1 Integrated with existing observability tools
Grafana Loki: Prometheus, but for logs
| 10
Grafana Loki: Prometheus, but for logs | 11
1. Alert
2. Dashboard
3. Adhoc Query
4. Log Aggregation
5. Distributed Tracing
Fix!
Prometheus’ data model is very simple:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Identifiers are bags of (label, value) pairs:
{job=“foo”, instance=“bar”, ... }
Timestamps are millisecond int64, values are float64
Grafana Loki: Prometheus, but for logs | 12
Grafana Loki: Prometheus, but for logs | 13
Apps
Apps
Apps
apps
k8s
#0 Prometheus talks to k8s to discover list of targets
#1 Target information is “relabelled” to build labels
#2 Metrics are pulled from apps
#3 Target labels added to series labels
Loki’s data model is very similar:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Timestamps are nanosecond floats, values are byte arrays.
Identifiers are the same - label sets.
Grafana Loki: Prometheus, but for logs | 14
Grafana Loki: Prometheus, but for logs | 15
prom
tail
Apps
Apps
Apps
apps
k8s
Grafana Loki: Prometheus, but for logs | 16
Grafana Loki: Prometheus, but for logs | 17
1. Alert
2. Dashboard
3. Adhoc Query
4. Log Aggregation
5. Distributed Tracing
Fix!
#2 Airplane friendly and Cloud Native
Grafana Loki: Prometheus, but for logs
| 18
Grafana Loki: Prometheus, but for logs | 19
Airplane friendly
prom
tail
Apps
Apps
Apps
Grafana Loki: Prometheus, but for logs | 20
Scale out
prom
tail
Apps
Apps
Apps
Grafana Loki: Prometheus, but for logs | 21
Scale out
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
Grafana Loki: Prometheus, but for logs | 22
Microservices
promtail
#0 Simple to scale
#1 Integrated with existing observability tools
#2 Cloud native and airplane friendly
Grafana Loki: Prometheus, but for logs | 23
Bonus
Loki as a Prometheus datasource!
(From v0.4.0)
Grafana Loki: Prometheus, but for logs | 25
DEMO
Thanks! Questions?