Better Living Through Statistics
Monitoring Doesn't (Have To) Suck
Jamie Wilkinson
Site Reliability Engineer, Google
jaq@{spacepants.org,google.com}
@jaqpants
#monitoringsucks
I love monitoring. This hashtag makes me sad.
But what to talk about?
I have no idea what has gone on in the real world since going into the black hole.
These #monitoringsucks people seem to be on the right path... do I have anything new to add?
Validation
http://blog.lusis.org/blog/2012/06/05/monitoring-sucking-just-a-little-bit-less/
"Instead of alerting on data and then storing it as an afterthought (perfdata anyone?) let’s start collecting the
data, storing it and then alerting based on it."
What is monitoring?
Measuring,
Recording,
Alerting,
Visualising
Monitoring systems
automate the boring parts:
Measuring,
Recording,
Alerting,
Visualisation
so you have more time to do fun things... and debug the occasional emergency.
The current state of monitoring
Blackbox vs Whitebox
Blackbox: treat the system as opaque: you can only use it as a user would.
a.k.a. "probing"
c.f. cucumber-nagios
fuel indicator light
Blackbox vs Whitebox
Whitebox: expose the internal state of the system for inspection
a.k.a. instrumentation, telemetry,
... ROCKET SCIENCE
c.f. ... graphite? new relic? metrics?
tacho, fuel gauge, water meter
What's wrong with blackbox?
Only boolean: no visibility into why
No predictive capability
All the stuff you do with historical data...
Pseudo-whitebox
Expose some internal state
Test the latest point in time against a threshold.
Return Y/N/maybe?
... ~same as blackbox
e.g. fuel indicator light, again
Problems with "check+alert" model
So, Why does #monitoringsuck?
TL;DR:
when the cost of maintenance is too high to improve the quality of alerts
An idea...
http://blog.lusis.org/blog/2012/06/05/monitoring-sucking-just-a-little-bit-less/
"Instead of alerting on data and then storing it as an afterthought (perfdata anyone?) let’s start collecting the
data, storing it and then alerting based on it."
Structure of timeseries
Alerting on thresholds
Alert when beer supply low
if cases - 1 - 1 <= 1:
alert BarneyWorriedAboutBeerSupply
Disk full alert
Alert when 90% full
Different filesystems have different sizes
10% of 2TB is 200GB
False positive!
Alert on absolute space, < 500MB
Arbitrary number
Different workloads with different needs: 500MB might not be enough warning
Disk full alert
More general alert:
How long before the disk is full?
and
How long will it take for a human to fix an (almost) full disk?
Alerting on rates of change
Dennis Hopper's Alert
if speed >= 50mph:
alert BombArmed
if BombArmed && speed < 50mph:
...
Keanu's Alert
if speed >= 50mph:
alert SaveTheBus!
Keanu's alert
v - a * t = 50
50 - v = - a * t
(v - 50)/a = t
if (v - 50)/a <= time to save bus:
alert StartSavingTheBus!
New tools at our disposal
Calculus!
the derivative of speed is acceleration
the derivative of acceleration is ... jerk
(impulse?)
Error spike
error count
Rate of errors vs nominal rate
rate of change increases greater than expected
errors per second
calculate rates of change of timeseries
Another new tool
Not just looking at the latest data point, or the derivative at the latest point, otherwise just reimplementing pseudo-whitebox
Look back 5 minutes, 1 hour, 7 days, back to the dawn of time.
Useful minimum: 2.5x the sampling interval.
Traffic spike with threshold
worth getting out of bed for?
Traffic spike with threshold
worth getting out of bed for?
Δt
NO
maybe?
Δt
observe timeseries history to gain context
Timeseries Have Types
Counter: monotonically nondecreasing
"preserves the order" i.e. UP
"nondecreasing" can be flat
Timeseries Have Types
Gauge: everything else... not monotonic
Counters FTW
Δt
Counters FTW
no loss of meaning after sampling
Δt
Gauges FTL
Δt
Gauges FTL
lose spike events shorter than sampling interval
Δt
prefer counters over gauges
Another new tool
Instances in a cluster don't work alone.
Discover the properties of the system by aggregating the parts.
How many queries per second is your cluster receiving?
Take the rate of the counters, and then sum them together!
Aggregation
cluster rate = rate(instance 1 + instance 2)
aggregate to each grouping in the system
high school maths recap
Timeseries Operations: Rates
δ(counter)/δt = gauge
δ(gauge)/δt = gauge
(beware of sampling errors)
Timeseries Operations: Aggregation
Σ0..n(counter) = counter
Σ0..n(gauge) = gauge
(beware of sampling error and quantization)
Timeseries Operations: Ratios
counter / counter = counter: instant means
gauge / gauge = gauge: rate comparisons
e.g. New deployment
δ(errors) / δ(queries) > threshold?
What to measure?
What to measure?
Whatever is important to the business!
Building blocks:
What not to measure
What to alert on?
Whatever is important to the business!
When to alert?
Alerts are not logs.
Make sure it's ACTIONABLE...
then DOCUMENT IT
3-month-in-the-future-you will thank you.
Blackbox testing still necessary
Blackbox tests are end-to-end tests.
End-to-end testing by definition covers everything you have missed.
You still have charts in the timeseries to inspect, right?
TL;DL
HOW?
Attach a statistical package to your timeseries
database, and experiment
Make smarter alerts!
Questions?