Hard failures are hard to diagnose
Even with lots of boxes “repro” is not easy
Quick but reliable diagnosis is key
Failure analysis in old Google days
- Field tech plugs in a monitor and - if it
- shows something - e-mails a photo to eng
Faster and more scalable approach
- Machines broadcast “crash shouts”
- Collectors listen for packets and record them
- Global system for collection and analysis of failures
Amazing how much you can glean from a stack trace …
- … when it’s the only thing you have

	embed