​

​

​

​

​

​

​

​

​

Welcome & Opening Remarks

Matt Lowrie, Google

​

​

Evolution of Business and Engineering Productivity

Google

Manasi Joshi, Sr. Staff Engineer

Multiple Axes of Growth

Scale

Complexity

Connectivity

...

1.0 The Small Beginnings

Google Test in 2005

​

Culture

Infrastructure

Talent

The Test Pyramid

Unit

Component

Regression

Integration

Log

Privacy

Canary

Achieve fine balance

VELOCITY

​

​

​

DURABILITY

​

​

​

QUALITY

UTILITY

© Disney

Test Strategy 1.0

The Model

• Dev/Prod defect ratios
• 99% defect free
• Testing features matrix

Cons

• Lack of tests beyond unit/smoke
• Poor release velocity - 1 or 2 /month
• Lack of granularity of metrics

Challenges in Scaling!

Release Strategy 1.0

The Model

• Low frequency: 1 or 2 per month
• Pre-submit tests: Unit + some Smoke tests
• Lots of testing during releases

Cons

• Longer release iterations
• Releases with patches
• Long cycles - features to production
• Many late aborts in the release cycle

Challenges in Scaling!

2.0 Transition to Mid-size...Google Test in 2012

- shift in mindset

Talent

Infrastructure

Brand

Metrics

Culture

Display Ads:Evolution of Organization

​

• Talent - 10x (Hiring and Acquisitions)
• Query - 1000x
• Revenue - 1000x
• Brand

Infrastructure

Platform: Google & acquisitions

Features: 20+ ad formats

Devices: Desktop, mobile phone, tablet, xbox ...

User Data: 1st party vs 3rd party

Evolution of Test Strategy

​

The Model

• Product Aligned Test Infra
• Cross-functional Tools and Dashboards
• Handling of user data privacy

Pros

• Product excellence
• Bad pattern detection, code coverage
• Customer trust and business compliance : BRAND

Tool Proliferation and Redundancy : Con … or a Pro

​

Survival of the fittest!

​

Metrics and Measurements

​

Key Indicators

• Releases
• Feature code to Prod time
• … and many more

Call to Action

• Instrument the projects
• Metric improvement influences quarterly goals
• Shift in test culture

You cannot improve what you cannot measure.

Test and Release Strategy 2.0

​

Achievements

• Continuous Deployment
• 5x Increase in release cycle speed
• Avg line coverage ~70%

Metrics

Canary Testing

Monitoring

Frequent

Releases

Continuous Builds/Tests

Developer Testing

Feedback

3.0 Road to the Future … 2017 and beyond

Next Gen!

Talent

Infrastructure

Brand

Privacy

Test Strategy: New Challenges

​

​

The Model

• Integrated vertical experience
• Data driven eng productivity
• Supporting multiple platforms (mobile, VR, ….)

Pros

• Awesome user experiences
• Makes engineering friction free
• Follow change in user needs and computing platforms

Android Device Diversity (2015)

~25K distinct Android devices

​

~680K devices surveyed

​

~1200 device brands

More on Mobile

​

​

Key Needs

• App stability and increased quality
• Faster Releases

… and many more

​

Call to Action

• Improve developer productivity
• 1 week cycle with best in class tests and monitoring
• Solve challenge with test coverage and giant components.

​

THANK YOU

Automating Telepresence Robot Driving

By Tanya Jenkins�Principal Automation Engineer

Cantilever Consulting, LLC

Event: Google Test Automation Conference, November 2016� ©Cantilever Consulting, LLC, 2016

What does it mean to be ‘Present’ in the 21^st Century?

Conferencing �- You can hear, but it is hard to visualize the remote environment.

Teleconferencing�- You can hear and see, but you cannot always react to the environment in a meaningful way because you are stationary.

�

Is this guy really present?

Telepresence is ‘Action-at-a-distance’

​

​

With telepresence, a ‘pilot’ drives a robot in a remote environment. Seeing, hearing, and now actions, are completely within the pilot’s control. The pilot is present in that remote environment.

A person can now be ‘present’ anywhere, but how do you test that?

What’s different in testing Telepresence?

Validation Goes Beyond Testing the UI

Manual vs Automated Approach

Automated Test Control Challenges

Dedicated Test Automation Environment

Tracking Position and Orientation of Beam Device

​

The Beam UI gives visual confirmation of position to the pilot, but Automation can’t use the same mechanism.

​

​

Telepresence testing is more than UI testing

Beam Telepresence System

Manual vs Automated Approach

Validation of Telepresence Driving

Manual Validation of about 100 tests...

2 Desktop Platforms (i.e. Windows, MacOS)

X 1 Beam Device

X 1 Driving Input device (keyboard)

​

...Grew to 1000s of tests

12 Desktop Platforms

X 3 Beam Devices

X 6 Driving Input Device permutations�

​

Definitely time to automate!

​

Automated Test Control Challenges

What framework to use for Client automation?�

How to handle various Beam Driving Controls (Keyboard, Mouse, Gamepad) on the Client?�

How to track the Beam’s movements/position in a remote location (the lab)?�

How to retrieve this tracking information after it’s recorded?�

​

​

​

​

​

Options for Tracking the Beam’s position

​

Option 1: Dream Big – Beacons… And Drones!

Option 2: A Little Too Simple – Text on Wall

Option 3: Just Right – Lidar

​

Image courtesy of Lino Schmid

Automated Test Control Challenges

(answered in a nutshell)

What framework to use for Client automation?� FrogLogic Squish, Python most flexible for our application

How to handle various Beam Driving Controls (Keyboard, Mouse, Gamepad)?� Abstract the controls and control commands by developing a Driving API

How to track the Beam’s movements/position in a remote location (the lab)?� Install Lidar in lab, take readings at specific spots during Automated Driving Course

How to retrieve this tracking information after it’s recorded?� Remote log into the NUC to start Lidar and retrieve data. Also, retrieve same info from Beam’s motor board via the client

​

​

​

​

Okay, so what is Lidar?

​

Lidar, LiDAR or LIDAR (from Wikipedia), “is a surveying method that measures distance to a target by illuminating that target with a laser light.”

​

Animation of Lidar

Lidar Scanning Sample Visualization

Dedicated Test Automation Equipment

Beam (with modifications for Lidar scanning)

​

Beam Charging Dock

​

Hokuyo Lidar

​

NUC small form computer

​

​

Dedicated Test Automation Equipment

Beam with modifications for Lidar scanning (Details)

​

​

​

Non-Traditional Testing Lab� Not typical hardware or software

Safety Concerns� Beam weighs about 100 lbs and is expensive

Room Considerations� Isolation (from noise & interference)� Lighting (natural & man-made light)� Flooring (density)

Dedicated Test Automation Environment

Automated Driving Course

Automated Driving Course Animation

Automation Test causes Beam to Drive. Position info is taken at each labeled point.

​

Starts on Dock at A, drives forward to B.

​

Turns left toward C, then opposite to the right.

​

Turns around toward Dock.

​

Drives in reverse when near Dock.

​

Initiates Auto-Docking until Beam is parked successfully at A.

Tracking Position and Orientation of Beam Device

Data Collection � Position and orientation information corresponding to each labeled point of Automated Driving Course is collected from the Lidar output.� Expected distance traveled is calculated using data collected from the Motor Board of the Beam.

Data Verification

Position information from Lidar and from Beam’s Motor Board are compared. The test passes if the comparison is within the margin of error.

Automated Driving Course Demo

Thank you!

Special Thanks

Models:�Karl Boucher�Oswaldo Torres�Brad Sandman

Tech Consultants:�Brad Sandman�Harsha Kikkeri�Francois Marie-Lefevere

​

Other Credits

Animation:�Tanya Jenkins

​

​

​

​

Beam device:�Suitable Technologies, Inc

​

Also:

Q & A

FAQ

Q. Are there other Beams at Google?

1. If you are a Google employee you can access one of the many Beams on Google’s campus.

​

​

​

​

​

​

​

​

​

​

Break

​

​

​

​

​

what’s in your wallet?

About Me

Hima Mandali

What is CapitalOne?

CapitalOne at a Glance

A leading diversified bank with $339.1 billion in assets, $235.8 billion in loans and $221.1 billion in deposits¹

​

4^th largest credit card issuer in the U.S.⁴

The 3^rd largest issuer of small business credit cards in the U.S.⁵

​

Largest U.S. direct bank⁷

8^th largest bank based on U.S. deposits²

​

More than 65 million customer accounts �and 45,000 associates

​

A FORTUNE 500 Company - #112

1) Source: Company reported data as of Q2’16

2) Source: FDIC, Domestic deposits ranking as of Q2’16

3) Source: FDIC, June 2015, deposits capped at $1B per branch

4) Source: Company-reported domestic credit card outstandings, Q2’16,

5) Source: The Nilson Report, Issue 1089, June 2016

6) Note: Financial institutions includes banks & specialty finance lenders,

Source: AutoCount, FY 2015

7) Source: FDIC, company reports as of Q2’16

Source: ABA 2015 Survey

http://www.aba.com/Press/Pages/081115MobileBankingSurvey.aspx

​

Digital customer touch points

What’s really in your wallet ?

Mobile First

​

Continuous automated test runs on DevOps pipeline

Opportunities

Why is the solution needed ?

Observed minor discrepancies between real devices and simulator/emulator tests

​

​

Need for increased automated test coverage

Immediate feedback

Quality delivery to production on an hourly basis

Faster test executions and faster integrated automated deployment pipeline

Appium

​

Cucumber, Galen, Maven,

JVM, Java/Selenium

​

Layout Testing

Functional Testing

Layout Test

Html Test Reports

Functional Test

Functional and Layout Testing

• Functional Testing (Cucumber/Java)
• Validate the application behavior
• Schedule One Time Payment
• Verify the error messages text
• Verifying the confirmation message

​

• Layout Testing (Galen/Java)
• Validating visual and display aspects
• Verifying the location of Button or Menus
• Responsive Design
• Appearance of Pay Bill button or Card image

​

​

​

Demo

60

Test results in Pipeline

Test results in Pipeline

Test results in Pipeline

63

Test results in Pipeline

64

Test results in Pipeline

65

Test results in Pipeline

66

"Product" or "Program" Level Dashboard View

Our DevOps dashboard is open sourced

QUESTIONS ?

Smart Test Execution:

​

Using Automated Test Run Statistics to Optimize Their Execution

What is Unity?

Unity Editor

• 2D & 3D
• AR & VR
• Physics

​

Asset Store

Unity Services

INDUSTRY-LEADING MULTIPLATFORM SUPPORT

5.5M Users

Registered with Unity

238K Unique Games

Downloaded in Q2 2016

770M people

Playing Unity-made games

1.7B Mobile Devices

Running Unity-made games

Unity Code Base

• 2M Lines of Code split across:
• Platforms
• Editor
• Runtime

​

Unity Test Automation

31,261 Tests

• 10,603 unit tests
• 5,167 integration tests
• 3,460 runtime tests
• 519 graphics tests

Test runs per

• Day 1,690,668
• Week 9,854,880
• Month 39,896,746
• Year 311,043,693

​

Build farm

• 180+ Agents
• Slow UX

Bottleneck

Why?

Branches

• 14,238 branches in total
• Median age 3.5 days
• Average age 37 days

​

​

​

Getting to the main branch ‘trunk’

​

Pull Request to go thru:

• Code Review with peers
• Green ABV (A Build Verification) on a build farm
• Added to Trunk Merge Queue
• Batch of PRs merged to draft branch
• Pretty much all tests are launched
• Goes to trunk if successful

ABV is slow :(

3-6 hours

Can we speed it up?

Zoo of how to execute tests

Unified Test Runner (UTR) & Hoarder

UTR

Hoarder

Unified way to run tests

​

• Locally
• Build farm
• Sends all information on a test execution to Hoarder

​

All data on test run

​

• MS SQL Server
• ASP.Net Web Service
• AngularJS UI

​

Hoarder

Evergreen tests

Smart Test Execution (Hoarder++)

Let’s not run tests:

• 100% Successful
• During last month
• >100 test runs
• On all branches
• Enabled for all branches except trunk, draft/trunk and draft/trunk-queue

​

​

​

Integration

Tests

Windows 64

Editor

Test

Suite

Run

​

​

Saved Time on Average :)

​

• Integration Tests Windows 64 Editor Very Slow Test Suite
• 1 hour 41 minutes => 38 minutes - 63 minutes saved! - 62% saved!
• Integration Tests Windows 64 Editor Test Suite
• 1 hour 15 minutes => 37 minutes - 39 minutes saved :) - 51% saved!
• Integration Tests Mac 64 Editor Test Suite
• 1 hour 7 minutes => 23 minutes - 44 minutes saved! :) - 66% saved!

​

Integration Tests Mac 64 Editor

Integration Tests Mac 64 Editor

Next Steps

• Find which tests we have excluded but then they failed at trunk merge time and learn from it
• Use statistics for the current branch and trunk, draft/trunk and draft/trunk-queue (Boris’ mode :)
• Come up with some aggregate threshold number with formula including:
• Date of last failure
• Statistical methods (standard deviation etc)?
• Add test coverage data so that we know what tests to run on a particular changeset

Results

Questions? Ideas? Comments...

Thank you

​

boris@unity3d.com

​

Selenium-based test automation for

Windows and Windows Phone

Who am I?

99

Nikolai Abalov

Software Development Engineer at 2GIS

Selenium

“Free and open protocol for testing that has become a defacto standard”

Appium

101

• Selenium 3.0
• W3C working draft
• WebDriver spec

WebDriver protocol

Over HTTP

WebDriver

Implementation

System

under test

Magic

Get element text (JSON Wire Example)

102

GET http://127.0.0.1:9999/session/AwesomeSession/element/123/text HTTP/1.1

...

Content-Type: application/json;charset=UTF-8

Accept: application/json

HTTP/1.1 200 OK

Content-Type: application/json;charset=UTF-8

...

​

{"sessionId":"AwesomeSession", "status":0, "value":"GTAC"}

Selenium + Appium

103

* using UISpy I was able to locate elements in Internet Explorer, but not in Chrome

104

/Microsoft/WinAppDriver

Winium

Windows Phone Automation

107

/2gis/Winium.Mobile

Winium.Mobile

108

109

Tests

Winium.Mobile.Driver

Emulator

WebDriver Protocol

Over HTTP

XDEVirtualMachine

API

Internal API

Over HTTP

Automation Server

App under

test

How to start

110

• Prepare the app
• Write tests
• Run

Prepare the app

111

1. Add NuGet package Winium.StoreApps.InnerServer
2. In MainPageOnLoaded (or anywhere on UI thread) initialize and start server

#if DEBUG

AutomationServer.Instance.InitializeAndStart();

#endif // DEBUG

• Make your app testable (assign automation names, ids, etc.)
• Build and package your appx

Write tests

112

import unittest

from selenium.webdriver import Remote

​

class TestMainPage(unittest.TestCase):

desired_capabilities = {"app": "C:\\YorAppUnderTest.appx"} # files, ...

​

def setUp(self):

self.driver = Remote(command_executor="http://localhost:9999",

desired_capabilities=self.desired_capabilities)

​

def test_button_tap_should_set_textbox_content(self):

self.driver.find_element_by_id('SetButton').click()

assert 'CARAMBA' == self.driver.find_element_by_id('MyTextBox').text

​

def tearDown(self):

self.driver.quit()

/2gis/Winium.Mobile/wiki/Supported-Commands

Write tests

113

(WIP) You can switch between native and web contexts

import unittest

from appium.webdriver import Remote

​

# switch to WebView and act on web elements to login, etc.

self.driver.switch_to.context('WEBVIEW_1')

self.driver.find_element_by_tag_name('h1').click()

​

# switch back to Native app and act on native elements

self.driver.switch_to.context('NATIVE_APP')

​

Extended commands

114

# direct use of Windows Phone automation APIs

app_bar_button = driver.find_element_by_id('GoAppBarButton')

driver.execute_script('automation: InvokePattern.Invoke', app_bar_button)

​

list_box = driver.find_element_by_id('MyListBox')

si = {'v': 'smallIncrement', 'count': 10}

driver.execute_script('automation: ScrollPattern.Scroll', list_box, si)

​

# setting value of public property

text_box = driver.find_element_by_id('MyTextBox')

driver.execute_script('attribute: set', text_box, 'Width', 10)

driver.execute_script('attribute: set', text_box, 'Background.Opacity', 0.3)

Run

115

1. Start Winium.Mobile.Driver.exe (download release from )
2. Run tests as usual

Demo

118

/2gis/Winium.StoreApps.CodedUi

Winium.StoreApps.CodedUi

119

120

Tests

Winium.StoreApps.Driver

Emulator

WebDriver Protocol

Over HTTP

XDEVirtualMachine

API

App under

test

vs.test.console

CodedUI

Internal API

Windows Desktop Automation

122

/2gis/Winium.Desktop

Winium.Desktop

123

* using UISpy I was able to locate elements in Internet Explorer, but not Chrome

** solvable by using Windows Emulator or RDP child session directly

124

Tests

WebDriver Protocol

Over HTTP

Winium.Desktop.Driver

Winium.Cruciatus

App under

test

Desktop

UI Automation

Framework

Demo

Automating calc.exe (python bindings)

127

# setup

dc = {'app': 'C:/windows/system32/calc.exe'}

driver = Remote(command_executor='http://localhost:9999', desired_capabilities=dc)

# test

window = driver.find_element_by_class_name('CalcFrame')

menu = window.find_element_by_id('MenuBar')

view_menu_item = menu.find_element_by_name('View')

view_menu_item.click()

view_menu_item.find_element_by_name('Scientific').click()

​

window.find_element_by_id('132').click()  # 2

window.find_element_by_id('97').click()   # ^

window.find_element_by_id('138').click()  # 8

window.find_element_by_id('121').click()  # =

​

result = window.find_element_by_id('150').get_attribute('Name')

assert '256' == result

# teardown

driver.close()

Automating calc.exe (C# bindings)

128

// setup

var dc = new DesiredCapabilities();

dc.SetCapability("app", @"C:/windows/system32/calc.exe");

var driver = new RemoteWebDriver(new Uri("http://localhost:9999"), dc);

// test

var window = driver.FindElementByClassName("CalcFrame");

var menu = driver.FindElement(By.Id("MenuBar"));

var viewMenuItem = menu.FindElement(By.Name("View"));

viewMenuItem.Click();

viewMenuItem.FindElement(By.Name("Scientific")).Click();

​

window.FindElement(By.Id("132")).Click(); // 2

window.FindElement(By.Id("97")).Click(); // ^

window.FindElement(By.Id("138")).Click(); // 8

window.FindElement(By.Id("121")).Click(); // =

​

var result = windows.FindElement(By.Id("150")).GetAttribute("Name");

Assert.AreEqual("256", result);

// teardown

driver.Close();

More open source automation tools

129

VMMaster (Browsers Cloud) https://github.com/2gis/vmmaster

• Request clean environment for selenium tests on demand

​

STF Utils https://github.com/2gis/stf-utils

• Connect Android devices to local machine from your STF instance with one simple command.
• Record videos of your automated tests.
• Write your own python apps using STF API client implementation.

Thank you!

Nikolai Abalov

n.abalov@2gis.ru

https://github.com/2gis/Winium

https://github.com/2gis/Winium.Mobile

https://github.com/2gis/Winium.Desktop

https://github.com/2gis/Winium.StoreApps.CodedUi

​

​

​

​

​

​

​

​

​

Lunch!

​

​

The Quirkier Side of Testing

Brian Vanpee, Senior Test Engineer Extraordinaire

November 15 2016

​

An Odd Trait or Behaviour

Programming Languages Have Quirks

A few examples...

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Objective-C

HMCharacteristicValueLockMechanismLastKnownActionUnsecuredUsingPhysicalMovementExterior

HMCharacteristics.h

“

”

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

JavaScript

> null == undefined;

< TRUE

> isFinite(null);

< TRUE

> isFinite(undefined);

< FALSE

> null === undefined;

< FALSE

​

​

Try it in your Chrome browser right now! View -> Developer -> JavaScript Console

JavaScript

> Math.max(3, 0);

< 3

> Math.max(3, true);

< 3

> Math.max(-1, [1]);

< 1

​

Try it in your Chrome browser right now! View -> Developer -> JavaScript Console

JavaScript

> Math.max(-1, []);

< 0

> Math.max(-1, undefined);

< NaN

> Math.max(-1, null);

< 0

> var a = null + 1

< 1

​

Try it in your Chrome browser right now! View -> Developer -> JavaScript Console

Some quirks are found in many languages...

> String me = new String("\u00DFrian Vanpee"); // ßrian Vanpee

> System.out.println(me + " is now " + me.toUpperCase());

< ßrian Vanpee is now SSRIAN VANPEE

​

For a Fascinating Read go to https://goo.gl/G7QAI

Some quirks are specific to a language...

> int a = 0; System.out.println(a++ + " is the new " + a++);

< 0 is the new 1

​

> int a = 0; cout << a++ << " is the new " << a++ << endl;

< 1 is the new 0

​

// In C:

> int a = 0; printf ("%d is the new %d", a++,a++);

< 1 is the new 0

Sequence Points

A sequence point is a point in time at which side effects which have been seen so far are guaranteed to be complete.

Wikipedia

“

”

Source: https://en.wikipedia.org/wiki/Sequence_point

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Sequence Points

Between the previous and next sequence point an object shall have its stored value modified at most once by the evaluation of an expression.

​

Section 6.5 - “Expressions” of the C99 Standard

“

”

Source: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Sequence Points

> int a = 0; cout << a++ << " is the new " << a++ << endl;

< 1 is the new 0

​

> int b = 0; b = b++; b = b++; cout << "b is " << b << endl;

< b is 0 b = b++; is equivalent to b = b; 0 + 1;

​

> int c = 0; c = ++c; c = ++c; cout << "c is " << c << endl;

< c is 2 c = ++c; is equivalent to c = c + 1;

​

​

​

Finding the Quirk is the real fun!

​

Proprietary + Confidential

What does this do?

> return 1, 2, 3;

The comma operator (in C/C++, JS, Perl)

LHS , RHS Always evaluate LHS, Always evaluate RHS

LHS && RHS Always evaluate LHS, If LHS is true evaluate RHS

LHS || RHS Always evaluate LHS, If LHS is false evaluate RHS

Note: Not Every comma is an operator!

​

Eg. for (i = 0, j = 10; i <= 10; i++, j-- ) { ... }

Comma Operator in C/C++, JS, Perl

> return 1, 2, 3;

< 3

​

> return(1), 2, 3;

< 3

​

> return a=1, b=2, c=3;

< 3

​

> return 1, return 2, return 3;

< Syntax Error!

​

What value for i prints ‘Well Done’?

if (i == -i) {

System.out.print("Well ");

}

System.out.println("Done!");

C/C++ : int i = 0;

OBJ-C : int i = 0;

JAVA : int i = 0;

JS : int i = 0;

PHP : $i = 0;

What value for i prints ‘Well Done’?

if (i == -i && i != 0) {

System.out.print("Well ");

}

System.out.println("Done!");

C/C++ : int i = INT_MIN;

OBJ-C : int i = INT_MIN;

JAVA : int i = Integer.MIN_VALUE;

JS :

PHP :

Why this works

Recall that the range of 32-bit integer is -2³¹ to 2³¹-1

​

-2^31 = 0x80000000 -> 1 sign bit, all 0’s

​

Negating Integer.MIN_VALUE gives:

Integer.MAX_VALUE + 1 = 0x7FFFFFFF + 1 = 0x80000000

​

In other words, Integer.MIN_VALUE is its own negation!

What value for i prints ‘Well Done’?

if ($i == $i + 1) {

echo ("Well ");

}

echo ("Done!");

C/C++ : float i = INFINITY;

OBJ-C : double i = INFINITY;

JAVA : double i = Double.POSITIVE_INFINITY;

JS : var i = Number.POSITIVE_INFINITY;

PHP : $i = INF;

What value for i prints ‘Well Done’?

if (i != i) {

System.out.print("Well ");

}

System.out.println("Done!");

C/C++ : double i = 0.0 / 0.0; OR double i = NAN; OR #define i = a++

OBJ-C : double i = NAN;

JAVA : double i = Double.NaN;

JS : double i = Number.NaN;

PHP : $i = NAN;

What does this do?

int i = 5 - - - - - - - - 6;�while (i --> 1) { System.out.println(i + “...”); }

> 10...9...8...7...6...5...4...3...2...1...

​

Equivalent to:

int i = 5 - (- (- (- (- (- (- (- 6)))))));�while (i-- > 1) { System.out.println(i + “...”); }�

​

​

Next Presentation!

(Thank you!)

​

Proprietary + Confidential

ML Algorithm for Setting up� Mobile Test Environment

Rajkumar J. Bhojan

Principal Consultant,

Wipro Technologies.

GTAC - 2016

Agenda

• About Me
• Mobile Test Environment
• Device Management - Challenges
• ML Algorithm for Selecting Devices
• Q&A

​

Rajkumar J Bhojan (Raj)

• Principal Consultant, Wipro Technologies, Boston, MA

​

• Two Decades of professional experience in both Academics and IT

​

• Research Scholar, published many papers and filed couple of Patents

​

• Worked as a Test Architect, QA Manager, Corporate Trainer, Scrum Master and Test Consultant

Mobile Test Automation Environment

OS Platform

Network Environment

Testing Types

Mobile Devices ?

Mobile Testing Tools

Mobile Device Management – Challenges

Devices Released Every Year

Refresh Every 12-24 months

New Capabilities (Camera, GPS, Orientation, Voice, etc.)

Multiple Carriers, N/w switching, Disconnected Use, Throughput

Expensive to Buy

Mobile Device Management – Challenges (Contd.,)

Regional N/w - 400 N/w Operators

Processing Speed, Memory, Commn. Protocols

Non-Availability of Devices

Tight Deadline – Faster time to Market

Selecting Real Devices

​

​

​

​

​

​

• What are the key criteria for selecting the devices ?

• Does my device compatible with all types of mobile testing ?

• What is the permutation and combination of devices for my regression testing?

• What is the Device Market share ?

• What about the User Ratings / Reviews ?

​

Selecting Optimal Devices

Selecting Devices from a pool of Hundreds and thousands of devices

How to classify these devices?

24,000 Android Devices

Decision Tree Algorithm

• Cleaned the Dataset 120/15000 Samples
• Separated Trained Data and Test Data
• Computed the Entropy
• Classified data, based on Mobile OS and other Attributes
• Constructed Decision Tree for available Devices
• Predicted devices for next testing
• Ensemble Algorithm - Repeated the classification step for each attributes.

Sample Data

• Data Set - 200/15000 Devices

​

​

Table 1– Trained Data

Classification Graphs

Demo Screen Shot

#Import all required Library

from pandas import Series, DataFrame

import pandas as pd

import numpy as np

import os

import matplotlib.pylab as plt

from sklearn.cross_validation import train_test_split

from sklearn.tree import DecisionTreeClassifier

from sklearn.metrics import classification_report

import sklearn.metrics

from sklearn import tree

import warnings

warnings.filterwarnings('ignore')

#Load the data from CSV files

df = pd.read_csv("D:/DataSet/Device_demo.csv")

# Clean the Data

data_clean = df.dropna()

data_clean.dtypes

data_clean.describe()

Demo Screen Shot

features = list(clf.columns[:7])

y = clf["SELECTION"]

X = clf[features]

clf = tree.DecisionTreeClassifier()

clf = clf.fit(X,y)

Demo Screen Shot

from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=10)

clf = clf.fit(X, y)

# Existing Data with Selection

print(clf.predict([1,64,3,5.1,1,515,4])) ---- > 1

# Existing Data with Not Selected

print(clf.predict([2,128,2,5.5,3,401,5])) ------→ 0

# New data with different memory size

print(clf.predict([1,128,3,5.1,1,515,4])) ----→ 1

Take Aways:

Predicted value = A set of Recommended Devices for the regression testing

1. Leverage ML for identifying devices

2. Time & Cost

3. Device Based Issues

​

Thank You

Rajkumar J. Bhojan

Wipro Technologies

​

Q & A

“Can you hear me now?” :

Surviving Audio Quality Testing

Alexander Brauckmann

Dan Hislop

Surviving Audio Quality Testing

Presenters

Introduction

Scale of the Challenge

Audio Test Architecture Vision

AQA as a Service

AQA Demo

Q&A

Dan Hislop

Alexander Brauckmann

​

Scale of the Challenge

Citrix SaaS Family

Scale of Citrix GoTo Audio

Testing Challenges to Solve

• Manual audio testing is time consuming and subjective
• “Unrelated” changes can affect audio delivery
• Re-usable test anyone can launch to verify audio

​

​

No audio is a dealbreaker for online meetings

​

• Limited number of audio experts for testing
• Many teams incorporating audio platform into product
• Make testing easy for non-experts

Audio Test Architecture Vision

Audio Test Architecture Vision

Improve overall audio quality for Citrix customers

with unified internal teams

using best-in-class audio testing

��

Audio Quality Testing: Categories

Automating the top of the pyramid is the hardest to solve, requiring audio experts and product teams to work together.

Audio Quality Testing: Design Goals

Create a universal test solution that ALL teams can use to validate audio quality before release

Use audio expertise to create common components

Automated audio quality measurement pass/fail in CI after every audio-related check-in.

Provide tools & libraries for client teams to extend their existing automation.

​

Audio Quality Testing: Key Components

Standardized input files:

4 unique voices

Mock Mic: to inject audio file into client as if a person using a microphone

Mock Speaker: capture audio from client as if a person listening

Scoring Server: to compare the input to output and determine quality

Clean Audio

Reference Files

Client 1

Injects Audio

Client 2

Records Audio

AQA Service

Analyzes Audio

File Transfer

Media Flow

File Transfer

Audio Quality Testing: Overview

Clean Audio

Reference Files

Client 1

Injects Audio

Client 2

Records Audio

AQA Service

Analyzes Audio

Final Score and Pass/Fail

File Transfer

Media Flow

File Transfer

AQA as a Service

AQA Architecture and Implementation

Client

1: Upload files

2: Create job

3: Start job

4: Fetch result

AQA Service

Audio Quality Testing

POLQA (“Perceptual Objective Listening Quality Assessment”)

Reference/Played-out Signal

Degraded/Recorded Signal

• Active Speech Ratios
• MOS (Mean Opinion Score)
• Delay

MOS Score:

• Predict speech quality based as obtained in subjective listening tests
• Ranges between 5 (good) to 1 (bad)

Audio Presence Testing

Frequency Analysis

​

Speech Presence

​

Amplitude Analysis

1 kHz

Misc.

Misc.

0.5 kHz

1 kHz

Speech

Speech

Speech

0.9

0.5

0.1

Demo

Demo

Demo

Demo

Q & A

IATF: An Automated Cross-platform and Multi-device API Test Framework

Yanbin Zhang

Intel software and services group

What is IATF

IATF - Interactive API Test Framework

An Automated Cross-platform and Multi-device API Test Framework

Why we need IATF

Project background

​

Intel® Collaboration Suite for WebRTC community at http://webrtc.intel.com

​

• Intel CS for WebRTC Conference Server
• Intel CS for WebRTC Client SDK
• Intel CS for WebRTC Gateway

​

​

Project background http://webrtc.intel.com�

A Growing Ecosystem of Intel® Collaboration Suite for WebRTC

Many other Usages

Why we need IATF

P2P Servers

stream 1

stream 1

stream 1

Signaling

Signaling

Signaling

Peer-to-Peer Video Communication with Peer Server

Project background

http://webrtc.intel.com

Why we need IATF

RTC Servers

stream 2

stream 3

stream 1

streams 1,2,3

mixed streams 1+2+3 (SD)

mixed streams 1+2+3 (HD)

stream 1

Project background

MCU-based multi-party video conference communication modes

http://webrtc.intel.com

Why we need IATF

How to automatically test the interoperability across those various SDKs on different platform becomes a big problem

JavaScript

C++

Challenges of our WebRTC SDK Testing

Test steps in a single test case run at different devices and with different programming language

Test steps are tied and depended to other steps

Interaction and Dependence

Challenges of our WebRTC SDK Testing

​

​

​

Can not dependence on UI automation framework.

​

​

​

​

​

No automation test framework supports design and runs such kind of interactive API test cases

​

​

No UI application

No existing API test framework

Not only for WebRTC

Cross-platform and multi-device gaming SDK testing

Cross-platform and multi-device messaging SDK testing

……

​

Any SDK supporting multi-user communication!

IATF Design Philosophy

​

​

​

​

Deploy test cases automatically on Android, iOS, Windows, Linux, etc.

​

​

​

- Real-time status sharing and control.

​

​

​

- Easy to integrate with third-party API test framework

Cross-platform support

Cross-device communication

Extensible

Test case workflow

Peer Server

Intel Reference

Signaling Server

Implementation

Bob

2. connect

4 Receive Invitation

5. Accept Invitation

Intel CS for WebRTC Android SDK

Intel CS for WebRTC JS SDK

1 connect

3 .Invite Bob

6. Receive accepted message

RTP

7. Publish local stream to Bob

8. Publish local stream to Alice

Socket.io

Socket.io

Alice

Typical WebRTC P2P test scenario

Demo – WebRTC P2P test scenario

​

Application using Intel WebRTC JS SDK<-> Intel WebRTC Android SDK

Demo

WebRTC P2P test scenario with IATF

IATF Logical and Physical Structures

IATF General Execution Steps

Test Case Building and Deployment

Run Test

Get test result and generate test reports

Clean test environment

Read test environment configuration

Check the test devices status

Prepare test devices

Read , build and Deploy test cases

Test steps sequence controller

Devices status monitor

Test Case Building and Deployment

Deploy test cases

Deploy test cases

IATF Controller Server

Alice

Bob

3. Read configuration and clean test environment

​

port=10086

adbPath=/usr/bin/adb

antPath=/usr/bin/ant

shellPath=/bin/sh

karmaPath=/usr/lib/node_modules/karma/bin/karma

………

1. Start lock server

2. Check the test devices status and prepare test devices

karma start ….

adb install

Peer Server

Intel reference implementation

Sequence control message

Test Sequence Control with IATF

​

​

Key problem we should solve

​

​

​

​

​

​

​

​

​

​

​

​

​

.

Communication

Test Sequence Control with IATF

Platform dependent APIs are provided below

​

​

​

​

​

​

​

​

​

​

lockEvent can be defined as specified per your test scenario steps.

​

Connect(“IATF Server Address”)

​

​

WaitLock(“lockEvent”)

​

​

NotifyLock( “lockEvent”)

​

Test Sequence Control with IATFq

Join waiting list

Remove from waiting list

Search waiting list

IATF Lock Server Module

​

WaitRemote()

OnNotify()

WaitLock(“Invite to TestClient1”)

NotifyLock( “Invite to TestClient1”)

TestClient1

TestClient2

Status: unLocked

Status: Locked

Test Steps Sequence Control Example

IATF Test Server

Peer Server

Status: Locked

WaitLock(invite, From:Alice)

API: Invite(Bob)

WaitLock(Accept,From:Bob)

API: Invite(Bob)

Status: Unocked

Status: Locked

NotifyLock(invite,From: Alice)

Alice

Bob

NotifyLock(invite,From: Alice)

Status: Unlocked

1 Status: Connect

1 Status: Connect

API: onInvited event is trigged

Status: Locked

WaitLock(startTest)

Status: Locked

WaitLock(startTest)

NotifyLock(startTest)

​

​

Demo – Intel WebRTC P2P test scenario with IATF

Android SDK<->Android SDK

217

Demo - Intel CS for WebRTC Conference test scenario with IATF

MCU Servers

stream 2

stream 3

Mixed stream 1+2+3+4+5

Mixed stream 1+2+3+4+5

mixed streams 1+2+3+4+5

stream 1

Mixed stream 1+2+3+4+5

mixed streams 1+2+3+4+5

stream 4

stream 5

Questions?

Thank You!

Legal Notices and Disclaimers

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance�

This document contains information on products, services and/or processes in development.  All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. �

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. �

Intel and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. © 2016 Intel Corporation.

​

Using Formal Concept Analysis in software testing

Fedor Strok

Model-Based

224

Requirements�

• Model
• Test sequence
• Oracle

Implementation

Test output: fail or pass

​

225

Models

Model Driven Engineering

• Extract from existing systems
• Model before implementation
• Build both system and model

​

​

226

Alternatives

Online

Uses the SUT

Offline

Generates test cases, which can be executed in future

​

​

227

Possible implementations

State Machine

Theorem proving

constraint programming

Markov chain

Input parameter modelling

​

228

Formal Concept Analysis

G – set of objects

M – set of attrivutes

I – binary relation between G and M, interpretation: object has attribute

​

​

229

Formal Concepts and Implications

If an object has all attributes from A, the it has all attributes from B:

​

Armstrong Rules:

​

230

Attribute Exploration

Interactive procedure of exploring the domain

Can be started with arbitrary set of data

Works with 2 types of data:

set of implications

set of counter-examples

demo available: https://github.com/orivej/fca

231

Example

Simple program

Input: one number

Output: features of given number (even, factorial, divided_by_three, odd, prime)

​

232

Example

Does the implication holds?

→ even, factorial, divided by three, odd, prime

​

233

Example

Does the implication holds?

→ even, factorial, prime�

234

Example

Does the implication holds?

→ prime�

235

Example

Does the implication holds?

factorial → even

�

236

Example

Does the implication holds?

even → factorial

�

237

Example

Does the implication holds?

divided_by_three → even,factorial

�

238

Example

Does the implication holds?

factorial, divided by three → even

Yes, first factorial dividable by 3, is dividable by 2

​

239

Example

Does the implication holds?

prime, divided by three → even, factorial, odd

​

240

Example

Does the implication holds?

prime, divided by three → odd

Yes, only 3

​

241

Example

Does the implication holds?

even, odd → factorial, prime, divided by three

Left part is always false

242

Example

Does the implication holds?

even, divided by three → factorial

243

Example

Does the implication holds?

even, prime → factorial

Yes, 2

244

Example

Does the implication holds?

factorial, prime → even

Yes, 2

End

245

Example

factorial, prime → even

factorial, divided by three → even

prime, divided by three → odd

even, odd → factorial, prime, divided by three

even, prime → factorial

246