Adding Observability to Nomad Applications
Using the Grafana
Open Observability Platform
Copyright © 2020 HashiCorp
Not long ago...
Well that escalated quickly
Amazon architecture shift
Amazon retail website was a big monolithic architecture back in 2001, seven years later they run the death star.
*source: Werner Vogels on Twitter.
Operational Complexity
You need a mature operations team to manage lots of services, which are being redeployed regularly.
Microservice Trade-Offs by Martin Fowler
Cyril Tovena�(He/Him) 🇫🇷
Software Engineer at Grafana Labs
2. The Grafana Stack
3. Deployment with
Copyright © 2020 HashiCorp
What is Observability ?
Observability
The definition
Observability (o11y) is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
In practice, observability is achieved by using various tools and techniques to make sense of complex distributed systems.
Observability
The Three Pillars
Today’s reality
Disparate systems. Disparate data.
Copyright © 2020 HashiCorp
The Grafana Stack
Grafana
The Open Source Single Pane of Glass
Grafana
Our Opinionated Open Source Stack
Prometheus
Metrics
Loki
Logs
Tempo
Traces
Why
Prometheus
A pull-based monitoring system with dynamic service discovery, built for the cloud.
A powerful query language and multidimensional data model for rich, ad hoc analysis.
Open source, incredibly resource efficient and simple to operate.
github.com/prometheus/prometheus
Why
Loki
A multi-tenant log aggregation system inspired by Prometheus
Incredibly small index and highly compressed log data which makes it easy and affordable to operate.
Open source, support various cloud storage backend.
github.com/grafana/loki
Why
Tempo
An open source, easy-to-use and high-scale distributed tracing backend.
Tempo is cost-efficient, and is deeply integrated with Grafana, Prometheus, and Loki.
Support any of the open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.
github.com/grafana/tempo
Grafana
A better incident response workflow
Copyright © 2020 HashiCorp
Deployment with
The New Stack
A simple three-tier demo application, fully instrumented with Prometheus, Jaeger and Loki logging
github.com/grafana/tns
Deployment with Nomad
~/go/src/github.com/cyriltovena/observability-nomad > tree
├── Makefile
├── Vagrantfile
├── jobs
│ ├── grafana.nomad.hcl
│ ├── loki.nomad.hcl
│ ├── prometheus.nomad.hcl
│ ├── promtail.nomad.hcl
│ ├── tempo.nomad.hcl
│ └── tns.nomad.hcl
└── provisioning
├── dashboard.json
└── dashboard.yaml
TERMINAL
Copyright © 2020 HashiCorp
Demo time !
What’s Next ?
Monitoring more than just your applications.
23
Thank You
github.com/cyriltovena/observability-nomad
cyril.tovena@grafana.com @kuqd github.com/cyriltovena slack.grafana.com #loki