Building scalable OSS observability on AWS with Mimir, Loki, Tempo, and Pyroscope
TRACK 01 - SESSION 02
Suraj Nath
Sr. Software Engineer, Grafana Labs
—
All about LGTM Stack
👋�“Hello world”
It’s a log line
We all started our programming journey with a log line…
Logging is the Hammer
Very versatile tool, you can get things done if you know how to use your hammer…
hammer will get it done but sometimes a screwdriver or nail gun works better…
It’s one tool, In our toolbox…
The observability toolbox
Traces
Metrics
Logs
Profiles
Utilizing the full Observability toolkit
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level with profiling
Utilizing the full Observability toolkit
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level with profiling
Observability challenges to tackle
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level
Open source time series database
High availability
Horizontal scalability
Native multi-tenancy
Durable storage
Fast query performance
Production-proven dashboards,�alerts, and runbooks
Prometheus and OpenTelemetry* metrics compatible
Observability challenges to tackle
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level
Loki
Announced!
0.0
2018
LogQL v2
2.0
2020
2021
2022
New TSDB index
Line-based deletes
Scale scale scale…
Custom retention
Write ahead log
Out-of-order support
Production ready
1.0
2019
The Grafana Loki team trying desperately to finish Loki 3.0 �(a giant electromechanical Norse god Loki). �Photorealistic digital art
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level
Observability challenges to tackle
{ resource.service = "A" } > { resource.service = "B" }
What traces go first through service A
through service B?
then
{ resource.service = "A" } >> { status = error } | select(resource.service)
What traces go first through service A
have errors?
then
Show me the services where those errors occur
{resource.service = "server" && span.target = "/account"} >> { status = error }
Observability challenges to tackle
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level
Grafana Pyroscope
Profiling
Profiling is the analysis of code execution by collecting stack traces samples, and a set of function calls that lead to resource consumption.
Stacktrace samples
Resource usage
Call stacks
Reading a flame graph
Execution path that leads to a function
github.com/grafana/loki/logql/syntax.ParseMatchers
Inner function calls made
Continuous profiling
Profiling applications continuously solves the problem of collecting a profile at the right time.
By collecting profiles all the time (continuously) and saving them in a database.
Resource usage over time
Query
From profiling to continuous profiling
Scalable, multi-tenant, and cloud native
Helm, Kubernetes service discovery.
Cloud native
Scalable and efficient
Horizontally scalable, object storage, and multi-tenant
Integrated
Built in the same spirit as Mimir, Loki, and Tempo and integrated with Grafana
Language support
Easy to get started
Silos are “okay” at start…
but connected tools are better
Error logs pinpoint user issue
Traces
Metrics
Logs
Unexpected cpu spike
Profiles
Anomalous span reveals error cluster
Code level root cause
Metrics/Logs: CPU spike and OOM root cause with profiling
Metrics/Tracing: CPU spike/latency root cause with profiling
Tracing: Span latency root cause with profiling
From metrics to traces
From metrics to traces
From logs to traces
From logs to traces
...and from traces to logs
Utilizing the full Observability toolkit
How do I:
Find a needle in a haystack with log queries?
How do I:
Find where latency and errors originate across many services with traces?
How do I:
Find anomalies in metrics trends?
How do I:
Find root causes of issues at the code level with profiling
Intro to mltp demo (sorry, no live demo today)
Scan for demo repo
Grafana + AWS
Grafana + AWS
Grafana + AWS
Grafana + AWS
That’s all folks 👋
Questions?
Slides: https://suraj.dev/talks