1 of 67

building twitter’s next-gen

ALERTING SYSTEM

2 of 67

dan sotolongo

@sortalongo

megan kanne

@megankanne

justin nguyen

@justanguyen

#talk-megan-justin-dan

M: But first, we’ll introduce ourselves.

M: I’m Megan. Along with Dan and Justin, I joined the Observability team about 3 years ago. When I first joined the team I was interested in visualization. Since then I’ve explored the user experience challenges of alerting as well as the distributed systems challenges of implementing that UX at scale.

D: I’m Dan. During my time on the team, I focused on alerting systems, old and new, but I’m also interested in distributed systems, computation theory, engineering systems that make other engineers’ lives easier. I recently joined Google, but Megan and Justin graciously let me join them in this presentation anyway.

J: Hi, I’m Justin. I’ve had a lot of fun working on lots of different parts of Twitter’s monitoring stack, but most of my time has been spent working on the alerting service and the visualization frontend. My favorite part about working on this team is empowering our users to better do their work.

3 of 67

OBSERVABILITY AT TWITTER

scaleable

robust

realtime

estwo on flickr

4 of 67

5 of 67

6 of 67

Collection Agent 1

Ingestion Service

Storage (Manhattan)

Timeseries DB Query Engine

Temporal Indexing Service

Alerting Service

Visualization Service

Alerting Cmd Line Tools

Service 1

Timeseries Cmd Line Tools

Collection Agent n

Service n

….

The Observability stack looks something like this. Twitter runs in multiple zones, which are collections of co-located servers that run within a network. A stack like the one in this diagram exists in each of those zones.

The lifecycle of a metric begins on the bottom of the screen, where users export metrics, which are all doubles, as JSON on http endpoints on their services. Collection agents sit alongside our users’ service instances and poll these endpoints every minute. They send the metrics to our timeseries database, which is backed by the Manhattan Key/Value store. From there, the metrics are served to our alerting and visualization services.

Again, most of this talk is centered around the alerting service, <click for transition> which is just one small part of this diagram. If you’re curious about the rest of it, we published a blog post a few months ago called “Observability at Twitter”. I think the first result on Google is the older 2013 version, but the newer post should be just under that.

7 of 67

OLD ALERTING SYSTEM

successes and challenges

8 of 67

300M

4.3B

14x

metrics written per minute

Nov 2013

June 2016

9 of 67

25K 3M

alerts per minute

alert monitors per minute

10 of 67

making alerts

11 of 67

alert >

rule >

monitor

12 of 67

the good:

the bad:

Simple config language
Large body of existing examples
Easy to write, commit, and upload new alerts

Simple config language
Large body of existing examples
Easy to write, commit, and upload new alerts

13 of 67

14 of 67

15 of 67

16 of 67

alerts

dashboards

17 of 67

alerts

dashboards

18 of 67

being on call

19 of 67

Zone1

Zone2

20 of 67

Zone1

Zone2

21 of 67

22 of 67

23 of 67

THE SOLUTION

improve configurations

reduce loss of visibility

taspicsvns on flickr

24 of 67

making alerts: simplicity

25 of 67

chart

alert

26 of 67

27 of 67

server-side validator

follows best practices

queries not expensive

...

28 of 67

on call: reliability

29 of 67

old

zone 1

zone 2

new

zone 1

zone 2

30 of 67

t0

node 1

state: ok

timestamp: t0

t1

node 1

t2

node 2

state: ok

last evaluation: t0

t1

t2

evaluate t1 & t2

31 of 67

30%

time to detect in minutes

old

new

2.5

1.75

32 of 67

alerting service

alert scheduler

alert runner

start

alert source

zookeeper

configs

timeseries db

alerting api

alert source

zone 1

zone 2

timeseries db

storage (Manhattan)

current state

eval

snooze

react

record

history recorder

notifier

stop?

alert evaluator

snoozed?

...

balancer

shards

balancer

So what does this architecture look like? Well, there’s a lot going on on this slide and I’m not going to dig into each box and line. If you’d like more information, we’d be happy to discuss this in more detail after the talk.

So these are the basic moving parts:

[CLICK] We have a balancer that assigns alerts to logical shards and re-balances shards when physical nodes go down, for instance during a deploy or zone failure.

[CLICK] Each minute, the system checkpoints the most recent successful evaluation and, on startup, resumes evaluation [CLICK] from this point.

This system is designed for graceful degradation, so failures have limited scope. For example, a failure in the sharding system just causes work allocation to freeze, and a failure in the historical database will cause history loss, but not impact anything else. Again, we’d be happy to discuss this architecture in more detail. Come and find us after the talk!

33 of 67

human reasoning

34 of 67

INTEGRATION

TESTING

bring together signals

postsumptio on flickr

35 of 67

CONTEXT

global context

(twitter)

peer context

(dependencies)

local context

(changes in my system)

36 of 67

runbook

contact

37 of 67

EMPOWER

HUMANS

elaine_macc on flickr

38 of 67

LESSONS LEARNED

seldonscott on flickr

39 of 67

distributed systems

Requirements

Availability
Scale

Challenges

Consistency
Structural complexity
Reasoning about time

40 of 67

alerting system distribution

Work is split into shards

Shard by ruleset
Reuse existing tools:

Zookeeper
Manhattan (similar to Riak, Cassandra)
Aurora/Mesos

“Well, as you could have predicted, distributing the alerting system was the cause of a number of challenges. I’ll focus on one in particular, but first, some context. <click> As you know, any system that wants to parallelize work needs some way to divide that work into smaller pieces and assign those pieces to workers. That’s called sharding, and how you actually do that is one of the key decisions when designing a distributed system.”

Here, the work is evaluating alerting rules every so often and taking action based upon that. <click> We chose to shard the alerts by ID and assign them to workers dynamically.
<click> Our implementation delegates as much of the hard work as is safe to do in a monitoring system. <click> It depends on Zookeeper for maintaining consensus, <click> on Manhattan (Twitter’s KV database) for keeping state, <click> and on Aurora for orchestrating the computing instances.”

41 of 67

Distributed Systems Design

42 of 67

engineering principles

Abstract cleanly
Minimize coordination
Parallelize work
Distribute load evenly
Do it fast!

“Some of this advice takes the form of principles. For example:

<click> We should come up with clean abstractions that capture the work that needs to happen.
<click> We should minimize the amount of coordination required between those abstractions to keep their interfaces simple.
<click> We should parallelize the work so we can scale the system.
<click> We should make sure that the load is distributed evenly across the machines so we can run efficiently.
<click> We should also make these decisions quickly and ship a product in a timely fashion.

<DON”T CLICK>
We take these principles seriously as engineers, like a code of honor that guides how we write code. If you think about it, we’re kind of like knights, with our own kind of chivalry.”
“Well, there’s one knight in particular that I want to talk about: ...”

43 of 67

44 of 67

(small)

45 of 67

46 of 67

47 of 67

48 of 67

END

49 of 67

sharding rulesets

Ruleset

Rule

Fanout

50 of 67

51 of 67

sharding rulesets

Ruleset

Rule

Fanout

Ruleset

Rule

Fanout

52 of 67

distributed systems design

?

53 of 67

So many vows…

No matter what you do,

you’re forsaking one vow or the other.

‘So many vows... No matter what you do, you're forsaking one vow or the other.’ We have tons of great advice, but in the end, our situation may force us to choose between contradictory tips. When that happens, we just have to do our research, buckle down, and take our best shot.
“As it turns out, our decision didn’t come out half bad. We chose to shard by ruleset to keep the system simple. That means that our load is not spread evenly across the cluster, and we’ve had to tune the deployment to handle the load. As it turns out, there are teams who really do need to have 20 rules with 5 thousand monitors in each. We could implement logic to spread the load more evenly, but right now, it’s easier to just overprovision the clusters.
“Still, things could have easily turned out differently. That’s why we didn’t want to come up here and tell you how to build your systems. There’s lots of people telling you how to do that, and a lot of it is good advice that’s worth looking into. Just make sure you also remember that it’s impossible to avoid making mistakes all the time. And the next time you’re spelunking through some horrible legacy project that’s riddled with bugs and impossible to read, think about Jaime Lannister before you call the author ‘Kingslayer’.”

54 of 67

users

Support

Collaboration

55 of 67

user support

Sisyphus, Marcell Jankovics

56 of 67

user support: front line

Interaction points

UI
Command line tools
Configuration libraries

Helping out

Comprehensive validation
Helpful usage messages
Extensive documentation of user-visible code

57 of 67

user support:� second line

User guides

Getting started
Core concepts
Best practices
FAQ
API Docs

Documentation lives with code� (in a monorepo)

58 of 67

User Support: Third Line

59 of 67

user collaboration

Migrations:

Some happy
Others…

60 of 67

61 of 67

user collaboration

NEW!

62 of 67

user collaboration

NEW!

63 of 67

user collaboration

NEW!

“What did help us was to rethink our goal. Our goal was not really to ‘get people to do what we want’. It was to make Twitter better as a whole. The problem was that, by thinking in terms of carrots <click> and sticks <click>, we implicitly cast our users as mules <click> to be lured and driven. But, as it turns out, people generally aren’t crazy about carrots and they really don’t appreciate being hit by sticks. Instead of looking at our users as goals we had to meet, we had to acknowledge them as the competent individuals they are and cooperate with them to reach a mutually-beneficial compromise. Of course, that’s just the start. Such a compromise usually meant committing to more development work and dealing with delays in our roadmap. We certainly didn’t want delays, but sometimes they’re necessary from the perspective of the organization as a whole.”

64 of 67

Peter Trevelyan, Shifting Lines

65 of 67

thanks to

66 of 67

Ian Brown

Jonathan Cao

Hao Huang

Aras Saulys

Ning Wang

Si Wang

Mike Moreno

Caitie McCaffrey

Anthony Asta

JC Martin

Ryan O’Neill

Steven Parkes

Jacob Reiff

Yann Ramin

Michael Suzuki

Franklin Hu

Cory Watson

67 of 67

QUESTIONS?

if this sounds cool, come talk to us:

Justin Nguyen� @jnguyen� @justanguyen

Megan Kanne� @megan� @megankanne

Dan Sotolongo� @sortalongo_� @sortalongo

#talk-megan-justin-dan