Monitoring and Analysis

Christopher Parnin

Overview

Monitoring

Flame Graphs

Simian Army

Activity

What would you monitor in an application?

Github status spike

Some metrics after recent DDOS attack at github.

​

https://status.github.com/

3/28/15

Monitoring Strategy

• Instrumentation of applications and your infrastructure
• Storing data retrievable for analysis
• Creating dashboards which aggregate and present data suitable for business/operations.
• Setting up notifications for the right people

Places to Collect Data

Hardware (via SNMP)

voltages, temperatures, fan speeds, peripheral health

Operating System

memory usage, swap usage, disk space, I/O bandwidth, CPU load

Middleware

memory, thread/db connection pools, connections, response time

Applications

business transactions, value, conversion rate, external third party status.

Green-Amber-Red Light

Monitoring Framework based on Nygard 2007:

Green:

• All expected events occurred
• No abnormal events
• All metrics nominal
• All states operational

Green-amber-red light

Amber:

• An expected event has not occurred
• At least one abnormal event, with medium severity
• One or more metric out of nominal range
• A noncritical state not fully operational

Green-amber-red light

Red:

• A required event has not occurred
• At least one abnormal event, with high severity
• One or more metric far out of nominal range
• A critical state not fully operational

Incident Management

Inspired by fire department emergency response.

​

Dashboard

Notifications

Behavior Metrics

UX Metrics

• Happiness
• Engagement
• Adoption
• Retention
• Task Success

http://www.dtelepathy.com/ux-metrics/?utm_content=buffer90f37

Tools supporting Monitoring

Nagios

OpenNMS

Flapjack

Zenoss

Operations Manager

Tivoli

Splunk

PagerDuty

serverdensity.com

​

Discussion

How would you use these concepts in your project?

Flame Graphs

Created by

Brendan Gregg,

author of DTrace book.

​

​

See for more details:

https://www.usenix.org/sites/default/files/conference/protected-files/gregg_lisa13_flamegraphs.pdf

​

# dtrace -x ustackframes=100 -n 'profile-997 /execname == "mysqld"/ { @[ustack()] = count(); } tick-60s { exit(0); }'

dtrace: description 'profile-997 ' matched 2 probes

CPU ID FUNCTION:NAME 1 75195 :tick-60s

libc.so.1`__priocntlset+0xa libc.so.1`getparam+0x83 libc.so.1`pthread_getschedparam+0x3c libc.so.1`pthread_setschedprio+0x1f mysqld`_Z16dispatch_command19enum_server_commandP3THDPcj+0x9ab mysqld`_Z10do_commandP3THD+0x198 mysqld`handle_one_connection+0x1a6 libc.so.1`_thrp_setup+0x8d libc.so.1`_lwp_start

​

mysqld`_Z13add_to_statusP17system_status_varS0_+0x47 mysqld`_Z22calc_sum_of_all_statusP17system_status_var+0x67 mysqld`_Z16dispatch_command19enum_server_commandP3THDPcj+0x1222 mysqld`_Z10do_commandP3THD+0x198 mysqld`handle_one_connection+0x1a6 libc.so.1`_thrp_setup+0x8d libc.so.1`_lwp_start

Construction

Map stack frame to rectangle, color is random

Merge adjacent functions, sort x-axis alphabetically

Reading a flame graph

Which function is running on CPU the most?

Reading a flame graph

f()!

Flames rising at Netflix

We noticed that request latencies to our Node.js application would increase progressively with time. Specifically, some of our endpoints’ latencies would start at 1ms and increase by 10ms every hour. We also saw a correlated increase in CPU usage

​

http://techblog.netflix.com/2014/11/nodejs-in-flames.html

​

What’s wrong?

Immediately, we see incredibly high stacks in the application (y-axis). We also see we’re spending quite a lot of time in those stacks (x-axis).

​

On closer inspection, it seems the stack frames are full of references to Express.js’s router.handle and router.handle.next functions.

The fix

Something was adding the same Express.js provided static route handler 10 times an hour.

​

This turned out be caused by a periodic (10/hour) function in our code... Unfortunately, it was also inadvertently adding a static route handler with the same path each time it ran.

Discussion

What are some other things you can use a flame graph for?

Simian Army

http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

http://techblog.netflix.com/2011/07/netflix-simian-army.html

​

“The name comes from the idea of unleashing a wild monkey with a weapon in your data center”

Chaos Monkey

Randomly,

disrupt any instance

in your infrastructure.

Lessons Learned

• State is bad
• Clusters are good
• Surviving single instance failure is not enough

Chaos Gorilla

Simulates an outage

of an entire Amazon

availability zone.

Lessons Learned

• Infrastructure control can be a bottleneck
• Hidden assumptions on deployment topology
• Large scale events hard to simulate
• Rapidly shifting traffic is error-prone

Latency Monkey

Artificial delays in our RESTful client-server communication layer to simulate service degradation and measures if upstream services respond appropriately

Lessons Learned

• Startup resiliency is often missed
• Unified approach to runtime dependency management important
• Fallbacks can fail too

​

Janitor Monkey

• Searches for unused resources and disposes of them.

​

Lessons Learned

• Label everything (do we need this?)
• Clutter builds up (if you don’t stop it soon, you will have to deal with bigger problems)

System entropy:

• Cruft
• Vulnerabilities
• Cost
• Complexity

Others

Doctor Monkey taps into health checks that run on each instance as well as monitors other external signs of health (e.g. CPU load) to detect unhealthy instances. Once unhealthy instances are detected, they are removed from service and after giving the service owners time to root-cause the problem, are eventually terminated.

Conformity Monkey finds instances that don’t adhere to best-practices and shuts them down. For example, we know that if we find instances that don’t belong to an auto-scaling group, that’s trouble waiting to happen. We shut them down to give the service owner the opportunity to re-launch them properly.

Security Monkey is an extension of Conformity Monkey. It finds security violations or vulnerabilities, such as improperly configured AWS security groups, and terminates the offending instances. It also ensures that all our SSL and DRM certificates are valid and are not coming up for renewal.

10-18 Monkey (short for Localization-Internationalization, or l10n-i18n) detects configuration and run time problems in instances serving customers in multiple geographic regions, using different languages and character sets.

​

Discussion

What is a new monkey you might create?