1 of 24

Optimizing Selenium for build performance

Tricks and tips for creating a fast and scalable test codebase

http://bit.ly/optimizing-selenium

2 of 24

Intro

Santiago Suarez Ordoñez

​

>5 years with Selenium and test automation

Selenium committer

Sauce Ninja at Sauce Labs

@santiycr

sso@saucelabs.com

3 of 24

Sauce Labs

4 of 24

Agenda

  • Ensure data independence
  • Don't DDOS your own QA server
  • Write atomic tests
  • Focus on the essential
  • Generate app state
  • Use guinea pigs
  • Remove third party
  • Get rid of static pauses
  • Split your build
  • Watch your build

5 of 24

The main secret to faster Selenium tests

6 of 24

Parallelization

Hardware == cheap

Engineers == $$$$

7 of 24

Paralellization is not easy

It takes some work to get there...

8 of 24

Don't DDOS Your Server

Serial tests == very low load

50 concurrent browsers == lots of requests

Pay attention and when needed, get a web server that better resembles production

9 of 24

Data Independence

Execution order => Data dependency

Parallel tests == no order

Never use data sets more than once!

Wipe your DB on every build

10 of 24

Data Dependency

11 of 24

Data Collision

12 of 24

Atomic Tests

TestSignupLoginChangePasswdLogout :,(

​

Small tests:

  • run in seconds
  • report failures precisely
  • have higher ROI in parallel

​

Small tests are good tests

​

13 of 24

Focus on the Essential

Don't test content during functional tests

Write integrity tests

Avoid Selenium for plain content when possible

​

Don't test performance during functional tests

Write performance tests

Test performance in a controlled environment

14 of 24

Generate Application State

Generate state out of the browser

The UI is inefficient

Causes interdependence (Y can't run unless X works)

Keep manual versions for special cases

​

​

15 of 24

Generate State via DB/Memcache

16 of 24

Generate State via Web Server

17 of 24

Use Guinea Pigs

Test widgets in isolation

Test extensively on an empty page

Run smoke tests on any real page

18 of 24

Guinea Pig Example

19 of 24

Remove Third Party Where Possible

Anything outside your control slows you down Adds non-determinism (brittleness)

Test third party with specific third party tests

Run functional tests including third party on a separate schedule

20 of 24

Get Rid of all Static Pauses

Every pause is there for a purpose

Find that purpose and wait smartly

Every second spent waiting for something that already happened, is a second wasted by your full development team

21 of 24

Break Your Build in Test Steps

Not all tests are worth parallelizing :,(

Doesn't mean all of them need to go serially

Have a serial tests and a parallel tests step

22 of 24

Watch your build

Sit at your build while it runs

Check logs

Find bottlenecks

23 of 24

QA

24 of 24

Thanks!

sso@saucelabs.com

@santiycr