1 of 64

Automating your QA with Visual Regression Testing

Slides: bit.ly/ataylorme-visual-regression-workshop

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

2 of 64

First, We Need To �Get Things Set Up

LoopConf 3.0 • Slides: goo.gl/YB3v43

3 of 64

Make Sure NodeJS Is Up To Date

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

4 of 64

Copy The Example Repository Template

https://github.com/ataylorme/visual-regression-testing-workshop

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

5 of 64

Clon Your Copy Of The Example Repository

git clone https://github.com/ataylorme/visual-regression-testing-workshop.git

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

6 of 64

Download Dependencies

npm ci

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

7 of 64

Don't Skip Ahead

LoopConf 3.0 • Slides: goo.gl/YB3v43

8 of 64

Let's Dive In

LoopConf 3.0 • Slides: goo.gl/YB3v43

9 of 64

Hi, I’m Andrew

Developer Programs Engineer at Pantheon.

ataylorme on GitHub and Twitter.

andrew@pantheon.io

Some things I enjoy away from the computer are

Spending time with friends and family
Running, weight lifting, and mountain biking
Hiking, off roading, camping, and fishing

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

10 of 64

Maintainer of Websites

LoopConf 3.0 • Slides: goo.gl/YB3v43

11 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

12 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

13 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

14 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

15 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

16 of 64

Did You Notice �What Changed?

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

17 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

18 of 64

Anyone Else Do This?

@ataylorme • Slides: goo.gl/oXqt1a

19 of 64

Repeat

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

20 of 64

Make The Robots �Do The Work

LoopConf 3.0 • Slides: goo.gl/YB3v43

21 of 64

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

22 of 64

Visual Regression: The New Pixel Perfect

The Power of Machine Vision
Diligent Difference Detection
Easy to Create. Quick to run.
Powered by tools like BackstopJS

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

23 of 64

LoopConf 3.0 • Slides: goo.gl/YB3v43

24 of 64

LoopConf 3.0 • Slides: goo.gl/YB3v43

25 of 64

LoopConf 3.0 • Slides: goo.gl/YB3v43

26 of 64

LoopConf 3.0 • Slides: goo.gl/YB3v43

27 of 64

Visual Regression Benefits

"Reference vs Test" Comparison

Reference - either the production site or saved screenshots
Test - another environment to compare against

Support for Multiple �Viewports (Resizing)
Test as Many Templates as You Want
Helpful Error Reporting
Easy to Configure

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

28 of 64

Visual Regression Limits

Lots of Noise in the Testing Process
There Can Be False Positives
Difficulties with Dynamic Content

You can ignore by CSS selector

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

29 of 64

BackstopJS

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

30 of 64

We Can Have Robot Vision

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

31 of 64

Coding: A Node App�For Visual QA

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

32 of 64

Copy The Example Repository Template

https://github.com/ataylorme/visual-regression-testing-workshop

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

33 of 64

Clone Your Copy Of The Example Repository

git clone https://github.com/ataylorme/visual-regression-testing-workshop.git

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

34 of 64

Download Dependencies

npm ci

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

35 of 64

Run Visual Regression Tests

npm run start

And select Return to Pantheon Test

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

36 of 64

Wait For BackstopJS To Run

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

37 of 64

View The Resulting Report

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

38 of 64

Update The Site List

Edit inc/sitesToTest.js

This is where the list of sites to test is stored
Try adding one (or more) of your sites
label is the site name shown when selecting a site to test
nonProductionBaseUrl is your non-production environment (local, staging, etc.) URL
productionBaseUrl is your production site URL
Adjust pathsToTest, which is the array of URIs to test for each site

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

39 of 64

Run Visual Regression Tests Again

npm run start

Select your site from the list

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

40 of 64

Adjust The BackstopJS Config

Edit inc/backstopConfig.js

This JS file generates the BackstopJS config

See the BackstopJS configuration documentation for more info

Adjust viewports, delay, hidden selectors, etc.

Add a tablet viewport size of 768px by 1024px

Run npm run start to test again with the new configuration

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

41 of 64

Run Visual Regression Tests Again

npm run start

Select your site from the list

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

42 of 64

LoopConf 3.0 • Slides: goo.gl/YB3v43

43 of 64

Advanced: BackstopJS Custom Scripts

Make a new directory inc/site-name

Replace site-name with the site slug from inc/sitesToTest.js

Create an onReadyScript.js file in that directory
Export an async function accepting 3 arguments:� page, scenario and vp
vp will be one of the viewport slugs from inc/backstopConfig.js
page gives you full access to the Puppeteer page class
See the BackstopJS docs or inc/ataylorme/onReadyScript.js for examples

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

44 of 64

Example: Remove Element From Page

Change #wp-gdpr-cookie-notice to a CSS selector of something on your site

module.exports = async (page, scenario, vp) => {

// Remove an element from the page

await page.$eval('#wp-gdpr-cookie-notice', e => e.parentNode.removeChild(e));

}

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

45 of 64

Example: Expand Mobile Menu

module.exports = async (page, scenario, vp) => {

// If on mobile and the homepage, expand the menu

if ('phone' === vp.name && 'Homepage' === scenario.label) {

const menuToggleButtonSelector = 'button.menu-toggle';

try {

await page.waitForSelector(menuToggleButtonSelector, {timeout: 250,visible: true});

await page.click(menuToggleButtonSelector);

} catch (error) {

console.log("Could not find the menu toggle button.");

}

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

46 of 64

Example: Wait For All Network Requests

function waitForNetworkIdle(page, timeout, maxInflightRequests = 0) {

page.on('request', onRequestStarted);

page.on('requestfinished', onRequestFinished);

page.on('requestfailed', onRequestFinished);

let inflight = 0;

let fulfill;

let promise = new Promise(x => fulfill = x);

let timeoutId = setTimeout(onTimeoutDone, timeout);

return promise;

function onTimeoutDone() {

page.removeListener('request', onRequestStarted);

page.removeListener('requestfinished', onRequestFinished);

page.removeListener('requestfailed', onRequestFinished);

fulfill();

}

function onRequestStarted() {

++inflight;

if (inflight > maxInflightRequests)

clearTimeout(timeoutId);

}

function onRequestFinished() {

if (inflight === 0)

return;

--inflight;

if (inflight === maxInflightRequests)

timeoutId = setTimeout(onTimeoutDone, timeout);

}

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

47 of 64

Usage: Handling Lazy-Loaded Assets

module.exports = async (page, scenario, vp) => {

// Scroll to the bottom of the page

await page.evaluate(() => {

window.scrollBy(0, window.innerHeight);

});

// Wait for all network activity to stop

await waitForNetworkIdle(page, 500, 0);

}

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

48 of 64

Benefits of Adopting Automation

LoopConf 3.0 • Slides: goo.gl/YB3v43

49 of 64

Reduced Overhead

@ataylorme • Slides: goo.gl/oXqt1a

50 of 64

Consistency

@ataylorme • Slides: goo.gl/oXqt1a

51 of 64

Risk Mitigation

@ataylorme • Slides: goo.gl/oXqt1a

52 of 64

Confidence

@ataylorme • Slides: goo.gl/oXqt1a

53 of 64

Automation is �a State of Mind

LoopConf 3.0 • Slides: goo.gl/YB3v43

54 of 64

Visual Regression Automatically on PRs

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

55 of 64

Reports Saved In The Cloud

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

56 of 64

Adding More Tests

Visual Regression Testing

Unit Tests

Integration Tests

Browser Tests

Performance Tests

Load�Tests

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

57 of 64

Example: Audit With Lighthouse

Lighthouse is a tool developed by Google to audit sites on performance, accessibility, best practices and more.

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

58 of 64

Lighthouse Use Case

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

59 of 64

Automation is a Journey

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

60 of 64

Resources

@ataylorme • Slides: goo.gl/oXqt1a

61 of 64

Learning About Visual Regression

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

62 of 64

NodeJS App Repository

@ataylorme • Slides: bit.ly/ataylorme-visual-regression-workshop

63 of 64

Reference Project: GitHub + CircleCI Workflow

Composer Managed Dependencies
WP Rig Theme Build With Gulp
PHP Unit Testing
Testing WP Coding Standards (PHPCS)
Pantheon multidev for each new PR
Visual Regression Testing (Backstop JS)
Lighthouse Performance Audit
Behat Acceptance Testing (WordHat)

1 of 64

2 of 64

3 of 64

4 of 64

5 of 64

6 of 64

7 of 64

8 of 64

9 of 64

10 of 64

11 of 64

12 of 64

13 of 64

14 of 64

15 of 64

16 of 64

17 of 64

18 of 64

19 of 64

20 of 64

21 of 64

22 of 64

23 of 64

24 of 64

25 of 64

26 of 64

27 of 64

28 of 64

29 of 64

30 of 64

31 of 64

32 of 64

33 of 64

34 of 64

35 of 64

36 of 64

37 of 64

38 of 64

39 of 64

40 of 64

41 of 64

42 of 64

43 of 64

44 of 64

45 of 64

46 of 64

47 of 64

48 of 64

49 of 64

50 of 64

51 of 64

52 of 64

53 of 64

54 of 64

55 of 64

56 of 64

57 of 64

58 of 64

59 of 64

60 of 64

61 of 64

62 of 64

63 of 64

64 of 64