1 of 50

Philip Tellis

@bluesmoon

Nic Jansma

@nicj

https://github.com/SOASTA/boomerang

http://www.soasta.com/mpulse

https://github.com/SOASTA/measuring-continuity

2 of 50

Measuring Continuity

2016-06-22

#VelocityCONF 2016

3 of 50

boomerang

https://github.com/SOASTA/boomerang

WARNING: HUMANS WHO SUBMIT PULL REQUESTS ARE KNOWN TO HAVE BEEN HIRED

4 of 50

WHY

5 of 50

Delight

Or

Frustrate

6 of 50

Experience

Responsiveness

Cognitive Dissonance/Resonance

Smoothness

Delight

Frustration

7 of 50

RUM today

We measure everything up to navigation complete (page load or SPA nav)
We measure whether users bounce or convert

But

The bulk of user interaction and experience happens after navigation has completed

8 of 50

Which continuous variables can we measure and how?

9 of 50

Developer Tools

Let’s talk about how we can translate the data available in modern web browsers to the actual *experience* your visitors are getting after page load.

Everybody here probably has their favorite developer tools, right?

Here we see Chrome’s Developer Tools, which grow more and more powerful each year. As an application developer, we often want to try to collect “all the data” so we can get a sense of our visitor’s page load experiences. There is a treasure trove of information in dev tools. We use them to diagnose slow network activity, frame rates and JavaScript call stacks. The dev tools can be used for monitoring your app’s performance during development, but many of us would also like to collect similar data from our visitors in the wild too.

That’s what RUM is built around -- collecting performance metrics in aggregate to make decisions on how healthy your application is and alert when there are issues your customers are seeing.

10 of 50

Developer Tools

“The fact that something is possible to measure, and may even be highly desirable and useful to expose to developers, does not mean that it can be exposed as runtime JavaScript API in the browser, due to various privacy and security constraints”

— Performance APIs, Security and Privacy�

https://w3c.github.io/perf-security-privacy/

11 of 50

Continuity Metrics

12 of 50

FPS - Frames Per Second

requestAnimationFrame(callback)
Callback is run before the next paint�

1 // total frames seen this second

2 var frames = 0;

3

4 function measureFps() {

5 frames++;

6

7 // request a callback before the next frame

8 window.requestAnimationFrame(measureFps);

9 }

10

11 // start measuring

12 window.requestAnimationFrame(measureFps);

13

14 // report on frame rate (FPS) once a second

15 setInterval(function() {

16 console.log("FPS: " + frames);

17 frames = 0;

18 }, 1000);

13 of 50

FPS - Long Frames

Frames > 16.6 ms lead to < 60 FPS�

1 var lastFrame = performance.now();

2 var longFrames = 0;

3

4 function measureFps() {

5 var now = performance.now();

6

7 // calculate how long this frame took

8 if (now - lastFrame >= 18) { longFrames++; }

9

10 lastFrame = now;

11

12 window.requestAnimationFrame(measureFps);

13 }

14 window.requestAnimationFrame(measureFps);

15

16 // report on long frames once a second

17 setInterval(function() {

18 console.log("Long frames: " + longFrames);

19 longFrames = 0;

20 }, 1000);

14 of 50

FPS - Video

HTML5 VIDEO metrics (Chrome/FF)�

1 var latestFrame = 0;

2 var latestReportedFrame = 0;

3

4 setInterval(function() {

5 // find the first VIDEO element on the page

6 var vids = document.getElementsByTagName("video");

7 if (vids && vids.length) {

8 var vid = vids[0];

9 if (vid.webkitDecodedFrameCount || vid.mozPaintedFrames) {

10 latestFrame = vid.webkitDecodedFrameCount || vid.mozPaintedFrames;

11 }

12 }

13

14 console.log("Video FPS: "

15 + Math.max(latestFrame - latestReportedFrame, 0));

16

17 // reset count

18 latestReportedFrame = latestFrame;

19 }, 1000);

15 of 50

CPU - Page Busy

Browser doesn’t expose CPU metrics directly
Detect Busy by running a function at a regular interval
See if the callback runs at the time we expect
If the callback was delayed, the page was Busy
Busy can be caused by other JavaScript, layout, render, etc

So that gives us some good parity to Developer Tools for monitoring frame rate over time. What about the next item, CPU?

This is a case where the data available in the Developer Tools is far beyond what we can measure from JavaScript. Obviously we can’t get detailed trace or stack information from JavaScript, nor even overall browser CPU usage, primarily due to performance and privacy concerns.

However, we can still get a small sense of how busy the page is by doing our own “profiling”. For example, if we configure a callback to run on a regular interval (say every 25 milliseconds), we can see how often that callback hits its mark exactly. If it does not -- if it’s delayed -- we can presume that it was delayed because of other work the browser was doing. For example, other JavaScript was running, or the browser was working on layout, rendering or other tasks for the page.

Over time, over an interval, we can see how often these callbacks are delayed. If 20% of callbacks didn’t fire on time, we can make an educated guess that the page was busy around 20% of that interval.

We call this Page Busy.

16 of 50

CPU - Page Busy

11 setInterval(function() {

12 var now = performance.now();

13 var delta = now - last;

14 last = now;

15

16 // if we're more than 2x the poll

17 // + deviation, we missed one period completely

18 while (delta > ((POLLING_INTERVAL * 2)

19 + ALLOWED_DEVIATION_MS)) {

20 total++;

21 late++;

22 delta -= POLLING_INTERVAL; // adjust, try again

23 }

24

25 total++;

26

27 if (delta > (POLLING_INTERVAL + ALLOWED_DEVIATION_MS)) {

28 late++;

29 }

30 }, POLLING_INTERVAL);

17 of 50

NET - Resources

ResourceTiming
Bytes available in ResourceTiming2�

1 var resources =

2 window.performance.getEntriesByType("resource");

3

4 // number of resources fetched

5 var resourceCount = resources.length;

6

7 // number of bytes

8 var bytesOverWire = 0;

9 resources.forEach(function(res) {

10 bytesOverWire +=

11 res.transferSize ? res.transferSize : 0;

12 });

13

14 console.log("Resources: " + resourceCount

15 + " " + bytesOverWire + "b");

18 of 50

HEAP - Memory Usage

Non-standard
Reduced precision to avoid privacy concerns�

1 // report on JS object memory once a second

2 setInterval(function() {

3 var mem = window.performance

4 && window.performance.memory

5 && window.performance.memory.usedJSHeapSize;

6

7 console.log("Memory usage: " + mem);

8 }, 1000);

19 of 50

Battery

Monitor your visitor’s battery state
Reduce work on low battery

1 setInterval(function() {

2 navigator.getBattery().then(function(batt) {

3 console.log(batt.level);

4 });

5 }, 1000);

20 of 50

Interactions

21 of 50

Interactions - User Input

scroll
mousemove
click
keydown

22 of 50

Interactions - Visibility

Window’s visibility state�

1 document.addEventListener("visibilitychange", function() {

2 console.log(document.hidden ? "hidden" : "visible");

3 }, false);

Also look at the IntersectionObserver

23 of 50

Interactions - Orientation

How the device is being held

�

1 window.addEventListener("orientationchange", function() {

2 console.log("orientation: " + screen.orientation.angle);

3 });

24 of 50

Size Metrics

25 of 50

Size - Nodes

HTML size (bytes)
Overall Node count
IFRAME, IMG, SCRIPT, etc., node count

26 of 50

Size - DOM Changes

MutationObserver == change over time

1 var d = document;

2 var mutationCount = 0;

3 var domLength =

4 d.getElementsByTagName("*").length;

5

6 // create an observer instance

7 var observer = new MutationObserver(function(mutations) {

8 mutations.forEach(function(mutation) {

9 if (mutation.type !== "childList") { return; }

10 for (var i = 0; i < mutation.addedNodes.length; i++) {

11 var node = mutation.addedNodes[i];

12 mutationCount++;

13 mutationCount += node.getElementsByTagName ?

14 node.getElementsByTagName("*").length : 0;

15 }

16 });

17 });

18

19 // configure the observer

20 observer.observe(d, { childList: true, subtree: true });

27 of 50

Errors

1 var errorCount = 0;

2

3 window.onerror = function () {

4 errorCount++;

5 }

6

7 setInterval(function() {

8 console.log("Errors: " + errorCount);

9 errorCount = 0;

10 }, 1000);

28 of 50

Demo

https://github.com/SOASTA/measuring-continuity

29 of 50

So what?

Raw data != useful metrics

Let’s measure the user experience

Smoothness
Responsiveness
Reliability
Emotion

30 of 50

Smoothness - FPS during scroll

FPS during page load

Let’s start with framerate. On it’s own, it’s just a number. 60 FPS.

Over time, it tells a story. What was the framerate during the page load? What was the framerate while the user was scrolling? What was the framerate immediately after a user action like a button click? All of these are critical events in your visitor’s experience, and the framerate is one indicator to how smooth of an experience they’re having.

For any site that has content below the fold, the scrolling experience is where your site can shine, or, leave a bad impression with your visitors.

Simon Hearne built a wonderful bookmarklet that will use rAF to monitor the FPS while it forcefully scrolls your screen. It monitors the FPS until it hits the scroll bottom, and reports on the FPS over time.

�As one example, this site drops below 60 FPS several times during scrolling. You can feel it. The site fees sluggish and jittery. Towards the end of the page it starts to reach 60 FPS but there are still a few periods that drop it to ~10 FPS because of a periodic timer that modifies the DOM.

31 of 50

Smoothness - FPS after interaction

32 of 50

Responsiveness

How long it takes for the site to respond to input?

requestAnimationFrame to detect next paint
MutationObserver to detect DOM changes

UserTiming to monitor your own code
SPA instrumentation via boomerang

33 of 50

Responsiveness

1 document.addEventListener("click", function(e) {

2 var start = performance.now();

3 requestAnimationFrame(function() {

4 var delta = performance.now() - start;

5 console.log("Click responsiveness: " + delta);

6 });

7 }, false);

34 of 50

Reliability

JavaScript errors
Leaks:

JavaScript memory usage over time
DOM size increase over time

35 of 50

Tracking Emotion

36 of 50

Rage Clicks

Rage clicks are series of clicks in which your users are pummeling their mouse buttons in frustration. It’s like punching your site in the face, usually because it’s not doing what the user wants or expects it to.

— Caitlin Brett, FullStory

https://blog.fullstory.com/moar-magic-announcing-rage-error-and-dead-clicks-1f19e50a1421

37 of 50

Rage Clicks

1 var same = 0, x = 0, y = 0, targ = null;

2

3 document.addEventListener("click", function(e) {

4 var nX = e.clientX; var nY = e.clientY;

5

6 // calculate number of pixels moved

7 var pixels = Math.round(

8 Math.sqrt(Math.pow(y - nY, 2) +

9 Math.pow(x - nX, 2)));

10

11 if (targ == e.target || pixels <= 10) {

12 same++;

13 } else {

14 same = 0;

15 }

16

17 console.log("Same area clicked: " + same);

18

19 x = nX; y = nY; targ = e.target;

20 }, false);

38 of 50

Dead Clicks

Clicking without any meaningful visual (DOM) change
Might happen during (or right after) page load due to delayed JavaScript

39 of 50

Dead Clicks

40 of 50

Missed Clicks

User clicks near an element, but misses it

41 of 50

Mouse Movement

“People who are angry are more likely to use the mouse in a jerky and sudden, but surprisingly slow fashion.

People who feel frustrated, confused or sad are less precise in their mouse movements and move it at different speeds.”

— Inferring Negative Emotion from Mouse Cursor Movements

Martin Hibbeln, Jeffrey L. Jenkins, Christoph Schneider, Joseph S. Valacich, and Markus Weinmann

42 of 50

Ask Directly

43 of 50

Rage Clicking

Now in researching the company that makes these booths, Happy Or Not, I was presented with the popup in this screenshot.

Any time I see a modal popup or something like this, my immediate gut reaction is to hit the Escape key as fast as I can so I can get back to whatever I was doing. In this case, when I hit Escape, nothing happened. So I slammed the Escape key a few more times, hoping the site would get the hint that I wasn’t happy to be bothered by this popup. Only clicking on the small X got me out of the dialog.

This is another example of Dead and then Rage Clicking, in a way. I immediately hit Escape because I expected the popup to disappear, as many sites (and a long tradition of window interfaces) respect. When that didn’t work right away, I hit Escape a few more times for good measure.

Don’t make me find your silly little tiny X if I can quickly hit the Escape key like convention dictates.

44 of 50

Eye Tracking

https://webgazer.cs.brown.edu/

45 of 50

Eyebrow Tracking

https://webgazer.cs.brown.edu/

46 of 50

Emotion - α / β wave Tracking

Mind Reading Markup Language

47 of 50

48 of 50

Thank You

49 of 50

Photo Credits

Rum on Ice by wiserbailey�https://www.flickr.com/photos/25084516@N03/4317148060/

Angel Delight by Auntie P�https://www.flickr.com/photos/auntiep/360764980/

Frustrated by Kevin Lawver�https://www.flickr.com/photos/kplawver/1903240219/

50 of 50

Philip Tellis

@bluesmoon

Nic Jansma

@nicj

https://github.com/SOASTA/boomerang

http://www.soasta.com/mpulse

https://github.com/SOASTA/measuring-continuity