1 of 67

Not Only Cars:�"AI, Please Test My App"

Gil Tayar (@giltayar)

January, 2018

@giltayar

2 of 67

@giltayar

3 of 67

@giltayar

4 of 67

@giltayar

5 of 67

@giltayar

6 of 67

@giltayar

7 of 67

@giltayar

8 of 67

@giltayar

9 of 67

@giltayar

10 of 67

@giltayar

11 of 67

@giltayar

12 of 67

@giltayar

13 of 67

@giltayar

14 of 67

@giltayar

15 of 67

@giltayar

16 of 67

@giltayar

17 of 67

About Me

  • I’m Gil Tayar
  • My developer experience goes all the way back to the ‘80s.
  • Am, was, and always will be a developer
  • Testing the code I write is my passion
  • Currently evangelist and architect @ Applitools
  • We deliver Visual Testing tools: �If you’re serious about testing, checkout Applitools Eyes�
  • Sometimes my arms bend back
  • But the gum I like is coming back in style

@giltayar

@giltayar

@giltayar

18 of 67

Driverless Cars Are Here!

1st May 2012�Google passes Nevada’s self-driving car test

@giltayar

@giltayar

@giltayar

19 of 67

And It’s War!

  • Between Ford, GM, Mercedes, Volvo, BMW, Tesla, …

​

  • But also between Google, Apple, Intel, Uber, …

@giltayar

@giltayar

20 of 67

Why? Because it’s boring (and stressful)

@giltayar

@giltayar

21 of 67

...and dangerous

@giltayar

@giltayar

@giltayar

22 of 67

We can start concentrating on the what

@giltayar

@giltayar

23 of 67

Just Like With Testing

?

@giltayar

@giltayar

24 of 67

Autonomous Driving

@giltayar

25 of 67

Driverless Cars Are Here!

@giltayar

26 of 67

Driverless Cars Are Here!

Who’s Winning? Google or Tesla?

@giltayar

27 of 67

5 Levels of Autonomous Driving

  • Classification system of self-driving car
  • Developed by the Society of Automotive Engineers (SAE)
  • Adopted by the National Highway Traffic Safety Administration (NHTSA)

@giltayar

28 of 67

Level 0

​

@giltayar

29 of 67

Level 1: “Driving Assistance”

@giltayar

@giltayar

30 of 67

Technological Advances for Level 1 Driving Assistance Cars

  • It’s all about vision:
    • LIDAR: Light Detection And Ranging
      • Pulses of laser light that bounce off objects and help measure ranges
      • Similar to radar
    • Stereoscopic cameras
    • Radar
  • And the algorithms to understand �what is being seen

@giltayar

31 of 67

Level 2: Partial Automation

@giltayar

@giltayar

32 of 67

Technological Advances for Level 2 Cars

  • It’s all about AI:
    • Machine Learning
    • Neural Networks
    • Deep Learning
  • It’s all about the algorithms �to understand what is being seen�and to act upon them
  • Actually, not algorithms…
    • It’s all about NN

@giltayar

33 of 67

How Do Neural Networks Work?

@giltayar

34 of 67

Convolutional Neural Networks

@giltayar

35 of 67

How Do Neural Networks Work?

@giltayar

36 of 67

Each Individual Neuron

@giltayar

37 of 67

And back to NN...

@giltayar

38 of 67

Each Individual Neuron

@giltayar

39 of 67

Each Individual Neuron

But who decides the weights�of each individual neuron?

@giltayar

40 of 67

Learning and Backpropagation

  • Start with random weights
  • Feed network with inputs that have known output
  • If the network’s output is correct, good
    • But try again! You can do better.
  • If not, fix the weights, and loop
  • This is the “learning” part of�Machine Learning

Inputs to outputs

Errors to weight adjustment

@giltayar

41 of 67

Learning and Backpropagation

  • Start with random weights
  • Feed network with inputs that have known output
  • If the network’s output is correct, good
    • But try again! You can do better.
  • If not, fix the weights, and loop

Inputs to outputs

Errors to weight adjustment

This is why mileage numbers are important.

Deep learning needs lots of data!

@giltayar

42 of 67

Machine vs Deep Learning

  • Deep Learning is a subset of Machine Learning
  • Machine Learning is not just with Neural Networks
    • There are other methods
  • Machine Learning started with simple Neural Networks
    • Few inputs, few neurons
  • Deep Learning uses high computational powers to take Neural Networks one step forward
    • Lots of inputs (full images)
    • Tons of neurons and layers

​

@giltayar

43 of 67

So how does this work in cars?

In: image, out: traffic signals

@giltayar

44 of 67

So how does this work in cars?

In: image, out: steering actions

@giltayar

45 of 67

Level 3: Conditional Automation

  • Much more autonomy
  • Human need not monitor
  • Car will notify when it can’t handle situation, and let the driver take over

@giltayar

46 of 67

Level 4: High Automation

  • No steering wheel!
  • Driver never needs to intervene
  • Some conditions do not work
    • Snow storm
    • Fog
  • Google cars

@giltayar

@giltayar

47 of 67

Level 5: Full Automation

  • The holy grail of autonomous cars
  • Works driverless under any condition!

@giltayar

@giltayar

48 of 67

Autonomous Testing

@giltayar

49 of 67

Level 0: “No Autonomy”

@giltayar

@giltayar

50 of 67

Level 1: Drive Assistance

What was the differentiator�for Level 1 driving?

@giltayar

51 of 67

Level 1: Drive Assistance

What was the differentiator �for Level 1 driving?

@giltayar

52 of 67

How Can Visuals Help?

  • Action
  • Action
  • Validation
  • Action
  • Validation
  • Action
  • Action
  • Action
  • Validation

If the AI can see the page, can it assist in validation?

@giltayar

53 of 67

Looking at the page can check everything at once

@giltayar

54 of 67

It’s Not Just Pixels

  • Comparing pages visually is not just pixel comparison
  • Complex algorithms are involved
  • But they sometimes get things wrong
  • We need better understanding of what we see

@giltayar

55 of 67

Level 2: Partial Automation

​

What was the differentiator �for Level 2 driving?

@giltayar

56 of 67

Machine Learning Enhancements

  • Better comparison of pages
  • Better validations - less mistakes
  • Grouping of similar changes
  • Comparing pages with�different content

@giltayar

57 of 67

Level 3: Conditional Automation

Not only checking against a baseline

Checking any page for design and data problems

​

@giltayar

@giltayar

58 of 67

How?

  • Machine Learning
  • Looking at hundred of test results and seeing how things change over time
  • If there is an anomaly, send it over to a human to check

@giltayar

59 of 67

But a Human Still Drives the Tests

@giltayar

60 of 67

Level 4: Partial Automation

  • The AI drives the tests
  • Observes users using the application,�observes interactions over time
  • From those observations, generates �tests
    • Using reinforcement learning techniques

@giltayar

61 of 67

Level 5: Full Automation

@giltayar

62 of 67

Fly on the wings of fantasy

  • The AI will be able to converse with the product manager, understand the application, and fully drive the tests itself!

​

  • Unfortunately, given that no human has been able to intelligently understand a product manager’s description of an application, AI-s will need to be much smarter than humans to reach this level!

@giltayar

63 of 67

Level 6: Skynet

@giltayar

@giltayar

64 of 67

Are we there yet?�Should we go looking for another job?

@giltayar

@giltayar

65 of 67

Answer These Questions

  • Does your job require thinking, or is it rote and repetitive?
    • If thinking, you’re good. AI won’t be replacing you. It will be replacing you.
  • Otherwise
  • If a tool took away the repetitive and boring parts, would you be doing thinking work?
    • If you will, you’re good. AI will be that tool. It will be helping you!
  • Otherwise
  • Can you starting thinking?
    • If you can, you’re good. AI won’t be replacing you. It will be helping you!
  • Otherwise:
    • Sorry, AI has replaced all the boring and repetitive jobs you will apply to.

@giltayar

66 of 67

AI Is A Tool

  • Use it
  • Benefit from it
  • Don’t worry about it

@giltayar

@giltayar

67 of 67

Thank You

Gil Tayar (@giltayar)

@giltayar