Why Building

Record / Playback Tools

Is So Hard?

Oren Rubin

And why I think the Selenium API should evolve!

Oren

Rubin

Testim.io

CEO

Applitools

Director of R&D

Wix

Web Architect

Cadence

Compiler Engineer

IBM

Cloud Engineer

Google Developer Expert

Mentor

Google Launchpad

External Lecturer
Technion University

Meetup Co-organizer

Selenium-IL, GDG, Ember

...

Soon

  • Metrics
  • Record
  • Intro - CSS/XPath
  • Playback
  • Custom Action
  • Reuse
  • Debug
  • 3rd party Integration
  • Data driven
  • Reports

Good

Easy

Ramp up

30 seconds to record a user story.

25% code coverage.

75% integration-code coverage.

Bad Challenging

Features

  • Record
  • Playback
  • Reuse
  • Collaboration
  • Reports

Record

Record

Bug Reports

  • Document user actions
  • Text or pixels transcript

Playback

  • Relocate element in next runs - mimic
    • Reproduce bug - debug
    • Regression tests - prevent bugs

Record

Image Based Locators

  • Taken the element/window screenshot.

Object Based Locators

  • Important properties
    e. g. id=sucheBtn, text="Suche"

Record

Image Based Locators

Meetup Sikuli (EggPlant, etc..)

Record

Image Based Locators

  • Large/full screenshots are very descriptive


Challenges

  • Used to considered heavy
  • UI might change
  • Element alone sometime not enough

    e.g.

Record

Object Based

  • Input - use placeholder or label-for

    e.g.
  • Fallback - innerText / textContent


Challenges

  • Images e.g. - non descriptive.
    Accessibility (ARIA, alt) helps
  • Bad practice results in no input-label correlation

Record

Meet Usetrace

Intro: Query Languages

CSS Advantages

CSS

vs.

XPath

  • Web devs know CSS
  • Get shortcuts for web
  • CSS is a bit faster

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

CSS

CSS Selectors

By by tag name

css: my-gallery xpath: //my-gallery

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

CSS vs. XPath

By by property

Selector: [class=gallery]

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

CSS Selectors

CSS

Property named class and id have shortcuts: .class and #id

css: .im

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

CSS Selectors

CSS

Find element inside an element using space

css: .gallery .im

CSS Selectors

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

CSS

And op (&&) is by snapping selectors

Selector: img.gallery[title=1]

CSS Selectors

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

XPath has advantages

Parent: //img/../my-gallery

Text: xxx/text()='some-text'

In CSS4: $my-gallery img

CSS Selectors

<body>

<article>

<my-gallery class=gallery>

<img class="im f1" title=1>

<img class="im f2" title=2>

<xxx>

Record

Object Based

Choosing properties

Record

Which properties to save

  • Id
  • Class
  • Tag name
  • Text
  • Css selector
  • XPath
  • Link || partial link

Playback

Playback

Mimic User Interaction

on the same element

again and again.

Locating it based on its

Persistent Properties

Playback

Locate same Element

via its Persistent Properties

  • Id
  • Class
  • Tag name
  • Text
  • Css selector
  • XPath
  • Link || partial link

Id

Uniquely identifies an element in a document

Playback

by

ID Property

Id

Uniquely identifies an element in a document

Playback

by

ID Property

Id

Uniquely identifies an element in a document

Please name five different reasons:

Playback

by

ID Property

Failure #1

Tips!

  • Didn't see it in the DOM
  • No code change in this project
  • No code change in other projects
  • Just ran it again

Playback

by

ID Property

Failure #1

Random Ids

Reusable components can't have ids

Playback

by

ID Property

Failure #2

Tips!

  • Didn't see it in the DOM
  • HTML changes in this project
  • No code change in other projects

Playback

by

ID Property

Failure #2

Code changed

Testing is mostly AOP

Playback

by

ID Property

Failure #3

Tips!

  • Didn't find it using jQuery
    i.e. $("#my-id") === []
  • No code change in this project
  • New version in parent project

Playback

by

ID Property

Failure #3

Iframes

src has cache killers
...?version=1.4
xxx.com/v1.4/…

Finding the right iframe..
is as hard as finding the element!

Playback

by

ID Property

Failure #4

Tips!

  • Found other element using jQuery
    i.e. $("#my-id") === [<other-element>]
  • No code change in this project
  • No code change in other projects

Playback

by

ID Property

Failure #4

Same Id twice!

WTF?!

Playback

by

ID Property

Find element return first element

Failure #5

Tips!

  • Works 50% of test runs
  • See it, but $("#my-id") finds nothing
  • No code change in this project
  • No code change in other projects
  • No timing issues

Playback

by

ID Property

Failure #5

Two Body-s!

WTF?!

Playback

by

ID Property

Some old frameworks suck

Id

Uniquely identifies an element in a document

Tips!

  • Random Id
  • Devs changed element's id
  • Inside an iframe
  • Two elements with same id
  • Two Bodys! WTF?!

NOT!

Playback

by

ID Property

Class

Existing properties used for styling

Playback

by

Class Property

Challenge #1

Refactor Style => Fail!

I like

Blonde!

Playback

by

Object Properties

Challenge #2

Non unique > which one?

Playback

by

Object Properties

Challenge #2

Non unique /div/span/div/div/ul/li/div[0]

Playback

by

Object Properties

Text, Link, Partial Links

All considered too fragile

Playback

by

Object Properties

Best Practices

  • Non reusable e.g. headers/footers
    • Use Id
  • Components
    • Use classes or self debugging symbols
    • Split between
      • Locating the component
      • Locating components’ parts

Page Object Design Pattern

=>

Playback

by

Object Properties

Meet Testim.io

Locate via Statistical Analysis & Crowd Wisdom

Playback

by

Object Properties

Custom Actions

& Validations

  • Parameters
  • DOM
  • Window
  • DB
  • Debug
  • Return value
    • Function based => single object
    • Module based => export

Meet Selenium IDE

Playback

by

Object Properties

  • One click Install
  • Debuggable

  • Numerous plugins
  • Export to code

  • Supports variables

Meet Selenium IDE

Playback

by

Object Properties

Saves one locator

Meet Selenium IDE

Variables are Global

No isolated scope

Playback

by

Object Properties

Custom Actions

& Validations

Meet Selenium IDE

Code isn't a 1st class citizen

Custom Actions

& Validations

Meet Usetrace

Custom Actions

& Validations

Meet Usetrace

  • No click Install
  • JS actions

  • JS validations
  • Integration to CI

  • Supports variables

Custom Actions

& Validations

Meet Usetrace

Can't debug

Custom Actions

& Validations

Meet Usetrace

Code isn't a 1st class citizen

Can't reuse

Custom Actions

& Validations

Meet Usetrace

No grouping / reuse / params

Custom Actions

& Validations

Meet Usetrace

Did I mention you can't debug :(

Reuse

  • Group actions
    • Good even without reuse e.g. c# regions
  • Reuse existing group
    • In same test - validation
    • In other tests
  • Parameters
    • In param e.g. user/pass
    • Out e.g. create user dynamically
  • JS should be 1st class citizen

Debug

  • Between steps
  • Within Custom Code
  • Console
  • XHRs

3rd Party

Integration

  • CI Integration
    • CI/CD
    • Browser Grids
  • Image Validations
  • Monitoring
  • Reports

Data Driven

Data Driven

=

Reuse + Parameters

Reports

Next Steps

New API comprising Customs (JS) Locator

New API for Object Repository

Metrics

How do you

pick software?

How devs

think they

pick tools?

  • Features
  • Ramp up
  • Integration
  • Debugging
  • Performance
  • UI/UX

How I

pick tools

  • Debugging
  • Ramp up
  • Integration
  • UI/UX
  • Features
  • Performance

How people

really

pick tools?

  • Ramp up
  • UI/UX
  • Features
  • Integration to environment
  • Debugging
  • Performance

Thank you :)

Oren Rubin

oren@testim.io | @shexman | linkedin

Why Building Record/Playback Tools Is So Hard? - Google Slides