1 of 5

1

UI

E2E

Integration

Unit

E2E

Integration

UI

Unit

10K cases

5K cases

6K cases

40K+ cases

OBJECTIVE

Identify instances in Realize automated test suite that could be redundant
Optimize suite to reduce testing time while maintaining coverage

vs.

Mark Bernardo, 2019

Machine Learning for Test Optimization

2 of 5

2

Input regression test folders from Zephyr, use unsupervised machine learning in Python to cluster and categorize test cases

Use Python packages to compare similarities between test steps and results within the associated cluster

Filter the test cases so that only those with highly similar steps and results remain - these are the “potentially redundant” cases

1

2

3

4

Compare with current regression test runs to see if any pairs of redundant tests are being run in the same job

Mark Bernardo, 2019

Machine Learning for Test Optimization

Steps

Reporting program is centric around 4 main python scripts
The first script requires test exports from Zephyr: the database where all of the Realize regression tests are stored and categorized into folders. This script will intake a folder that contains anywhere from 10 to thousands of tests, and inside the folder will cluster tests containing similar terms. Clustering is a form of unsupervised machine learning that relies on term frequency to assign distance values of terms for each test. Python will take a given distance threshold, then group tests in the folder together based on how similar they are when observed at the given distance. Repository files have been grouped with cluster numbers in each file ranging from 2 clusters in the smallest test set to 43 for the largest test set. ** Figure 1 ‘neighborhoods’ of tests that contain similar terms **
The second script takes the clusters identified from the first script and compares the steps of all tests as well as the results of all tests. For each test within the given cluster, the strings for the test’s steps and results are compared with the steps and results of every other test inside the cluster. The script will return pairs of tests where the steps are at least 90% similar, and also tests where the results are at least 90% similar. Another feature of this script is returning the terms that are not common to both tests, just for visualization purposes.
The third script acts as a filter for the output of the second script. There are tests where the steps may be high similarity, but the results are low similarity, and vice versa. This third script will iterate through all of the comparisons, and return only where both the steps and the results are 90% similar or greater.
The fourth and final script takes the filtered similarity cases as well as an automated test report from Jenkins TestNG. Python extracts all of the Test IDs and time costs of the tests being run and compares it with a master list of high-similarity test pairs. Any overlap between the TestNG run and the list is displayed.

3 of 5

RESULTS

Found 27 pairs of high-similarity cases in the first run
Time cost of running all high-similarity cases ~ 4 hr 28 min

3

Mark Bernardo, 2019

Machine Learning for Test Optimization

4 of 5

NEXT STEPS

SQE TEAM HANDOFF

Manual analysis for indicated tests
Goal is to optimize as many cases as possible

APPLICATIONS

Potential for additional savings in other test suites
Cross-checking between test hierarchy

4

Mark Bernardo, 2019

Machine Learning for Test Optimization

Outcomes and Applications

The resulting data was shared with members of the SQE team to see which of these test pairs could be optimized. In an idealworld, we could cut out the ‘redundant’ half of these tests and this section would only take 2 hours while maintaining the same coverage. However, it is ultimately up to the SQE team to determine which tests- if any, can be adjusted or optimized to improve the testing cycle. For example, two tests can be 98% similar in steps and results, but the difference could be something important like student account vs. teacher account. This program right now would hopefully serve as a tool for an engineer with domain knowledge to use for indicating these high-similarity cases.
Another potential area of application is with the manual UI test suite. The report from TestNG only covers the automated test suite, which although automated still takes 16 hours. The manual test suite requires engineers to navigate the realize environment by hand, making it the most expensive test to run. This could be an area of use for making sure the engineers are only working on tests that are coverage-efficient.
My last area of application is with tests outside of the UI repository- I had started working with comparisons between UI, end-to-end and rest tests. This would serve to balance out the pyramidal distribution that the ideally efficient test hierarchy would have.

5 of 5

SPECIAL THANKS TO:

John Bunker

Jeffrey Tai

Alex Barberio

Ali Mohamad

Yves Sabato

Jeffrey Huffman

Alfonso Buono

Mahika Kudlugi

Haifeng Wang

Michael Chai

Adeleke McMillan

Randy Zhou

5

SCHOOL EMAIL: mgbernardo@wpi.edu

Jaya Padmanabhan