MLR - When and what - ManAutoTest
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

View only
Demographic infoType of Paper - Contribution FacetType of Paper - Research FacetDecision supportFactors considered (in when/what to automate)Measurement (quantitative)Attributes of the software SUT(s) or project
#ResourcesLinkYearC / J / W / M / S / B / TAuthor Affiliation (A, I, C)Heuristics / guidelineMethod / techniqueToolModelMetricProcessEmpirical results onlyOther1-Solution Proposal (example)2-Validation Research (weak empirical study)3-Evaluation Research (strong empirical study)ExperiencePhilosophicalOpinionOtherWhen to automateWhat to automateROI (indirectly help what/when)SumStability of SUTOther SUT aspectsNeed for regression testingTest TypeTest reuse/ repeatabilityTest ImportanceTest oracleTest stabilityAutomation (test) toolSkills level of testersOther human and organizational factorsEconomic factorsReducing manual test effortAutomatability of testingEconomic factorsDevelopment processOtherCostBenefitROIOther metrics usedFormula#SU/ / project namesDomainsTest cost metricsTest benefit metricsROI of test automationExperimental / simple (toy) exampleReal open-sourceCommercialCommercial and quantitative evidenceNoteworthy phrases
Technical and scientific sources
1A Search-based Approach for Cost-Effective Software Test Automation-Decision Support and an Industrial Case Study Tour Sheet (ETS)Energy control software367% (267% return)111
2A way of Improving Test Automation Cost-Effectiveness reuserelevance (severity of faults targetted by test cases)automation tool qualityresource (money, hw), test portabilitymanual complexity, manual exec effortAutomation effort111Nice idea. But no real validation or case
3Automated Unit Testing of a SCADA Control Software- An Industrial Case Study Based on Action Research (after the fact, Post-hoc) analysis11111RocketSCADA control software11138 hours11
4Comparative study of test automaton ROI
5Cost Benefits Analysis of Test Automation of GUI tests, Automation of build tests111.552 (55% return)11
6Cost-benefit analysis of using test automation in the development of embedded software elevator softwareembedded1140%11
7Economic perspectives in test automation- balancing automated and manual testing with opportunity cost
8Establishment of automated regression testing at ABB industrial experience report on 'avoiding the pitfalls and understanding of Concept of regression testingOrganizational maturity11
9Observations and lessons learned from automated testing related issues, e.g., expectations21115project a...eSystem to Manage Distribution of Assets, Java-Based Application Platform, Point of Sale System for Life Insurances, Sales Support for Tailored Industrial Facilities, System Test Automation for Control System11
10Pragmatic approach to software test automation Much to Automate1Automation fails when the current application has unstable designWe need to think about automation in case of reruns where it means to reduce cost.low test risk and complexity stable enviroment and datatest engineer's skilldesign the test case first, first manual then automatte, SUT known, low test risk and complexity
11Software Test Automation in Practice: Empirical Observations features (Generic and independent products facilitate and customized and complex products hinder testing automation)High reusability facilitates and low reusability hinders testing automation1consider automation costs11212 software development organizationselectronics, Internet servics, Logistics software, Safety and logistics system, Financial software11
12Software test automation–Developing an infrastructure designed for success tests for stable applicationsAutomate regression tests:Automate tests that have been written (manual test must exits first), Automate tests with no timing issuesboringness of the testautomatability18Building Technologies, HVAC (heating, ventilating, and air conditioning)1111
13Surviving the Top 10 Challenges of Software Test Automation criticality / riskTest repeatabilityautomatability1
14The Return on Investment (ROI) of Test Automation ApplicationPharmaceutical Engineering1111
15The When & How of Test Automation of the product/application is ensureddeciding the right toolInterface to be tested is determined, Scope of automation defined, deciding the right test mode (script recording/script development)1
16Visual GUI Testing in Practice Challenges, Problems and Limitations (after the fact, Post-hoc) analysis1This shows that the SUT must have reached a certain level of maturity for VGT to be applicable[automate] more stable test cases first112two safety-critical systemssafety-critical and military systems113 and 14 executions111
17When to Automate Software Testing -Decision Support based on System Dynamics Tour Sheet (ETS)Energy control software111
Books and book chapters
18Advanced Software Testing - Vol. 2: Guide to the ISTQB Advanced Certification as an Advanced Test Manager error prone [manual] tests1automatability1
19Automated software testing: introduction, management, and performance costtraining cost, tool expertise
20How Google Tests Software tests, no human cleverness and intuition->it should be automated, 11automate 20% of test that find 80% of bugs
21Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality2009BI111112Adequate skills1automatability1111
22Just Enough Software Test Automation of SUT, feature set require a significant number of data combinations to validate1number of buildstest tool capable of interacting with all of the necessary attributes of the featuretight delivery schedule1
23Project 1: Failure, Project 2: Success, Positive tests, Easy to continue to work with12Project 1, Project 2?11
24Software for Medical Devices and Our Need for Good Software Test Automation, miniweb, doityourself, hamletmedical11
25Software Test Automation: Effective Use of Test Execution Tools, test diversity (to get a broad view of quality)often runtest importance, funtion importanceeasy to automate1
26Software Testing Principles and Practices Tests that Pertain to Standards
Internet articles and white papers
27Achieving business benefits through automated software testing repeatabilityease of deveoping automated test oracles111Complexity of the test environment11an embedded softwareembedded11
28Agile Testers, Should You Automate? context for the article
29Automated Testing Advantages, Disadvantages and Guidelines and well-understood use cases or scenarios firstredundant tasks or scenarios
30Automating tests vs. test-automation systementerprise11
31Automation Selection Criteria – Picking the “Right” Candidates
32Automation Testing versus Manual Testing Guidelines of test case11
33Avoid Throwaway Test Automation
34Choosing What To Automate
length of transactions (test cases)
Highest priority features1113POS Credit Card Application, Retail web site, Software Installation1
35Determining What to Automate test cases first11Test Case: Generic Application Home Page Login/Navigate Testweb apps11
36Does Test Automation Save Time and Money?
37For those of you dreaming the 100% automation dream...please wake up! functional testing1
38Functional Test Automation - from SAP testing book of test inputs predictability of expected outputs (oracle)1111
39If you are going to run a test more than once, it should be automated. build verification tests111Unnnamed$1M+ investment of tests on a rack of 25 dedicated machines shortening their regression testing cycle by 75%11
40Learn When to Automate and When Not To- 99 Ways Workshop #5
41Manual vs. Automated Testing - explored - The Borland Blog and performance testingTesting across multiple OS platforms and multi lingual sitesautomate High Risk casesresource limitation1
42Regression Testing – What to Automate and how tests firsteasy tests first
43ROI on Test Automation - A Simple Yet Powerful Approach in support of when/ what11111111an example1
44Sources of Return on Investment (ROI) in Test Automation - QA InfoTech Corporate Blog in support of when/ what111automation makes the work more interesting and challenging, extra coverage from automation
45Test Automation: The Whens, Hows, and Whys number of tests
46Test Automation – Knowing When to Automate
47Test Automation and Best Practices management software$16,50022 iterations111
48Test Automation for Effective Post Deployment testing, Long lifecycle application1Proof of concept is needed111Telecom application for Public health careRegression test 48h -> 1.5h Over next 3 years 150,000$92.5% productivity improvement11
49Test Automation Patterns tests that bring most ROI21111
50Test Automation Process Customized applications with short life span. Life span of the product is long.1Check the functionality with huge set of data.
Verify database which contains additional data.
Time and resources are a constraint.1Spending a lot of time by testing manuallyComplexity of Application. Main Application has lot of interdependency with other Applications which in turn cannot be automated. 1WHEN:
Test application with in less time. Code coverage is vast.
51Test Automation ROI ROI on high level, but level of each test111quality of oracles11Relialbility of tests
52Test Automation ROI Calculator with various input factors11Number of configurations to be tested
Infrastructure cost
1Cost of the testing resources1Releases planned for a yearRequired regression cycle per year,
Number of existing regression test cases
53Test Automation- Delivering Business Value
54Test automation: When, how and how much technologies development velocity, Any test automation program should start with an honest assessment of the people, process and available technologies.1Task suitablity for automation (good test data creation, smoke and regression test1process, development paradigm,Test automation target, quality objective, Manual testing is not complemented time frame1
55Testing in the Fast Lane: Automating Acceptance Testing in an Extreme Programming Environment automated tests for critical business functionality first, and expanded the tests as we had time.We chose or created lightweight test tools which feature both power and flexibility.process, XP agileWe tried to avoid creating a suite of tests which takes more time to maintain than we had available.Agile context for the article
We evaluated each test to see if it makes sense to automate.
1a telephone directory
lookup website
56The Business Argument for Investing in Test Automation
GUI testing, Unit testing
1111Not givenNot given1
57The ROI of Test Automation of tests. E.g. performance there is no other practical way to do it.1 to automate the tests I couldn’t possibly perform manually. 1
What are automation goals?
1) automate all things that offer immediate returns and
2) if you have a hard time convincing someone that it needs to be automated and if you feel you need an equation to see the long term advantages of automation, don’t bother automating it.
58Too much automation or not enough? When to automate test (yes), build verification test (yes), Usability test (NO),-Reasons repeating same test changes in input/software/enviroment, Test that finds bugs, validate important scenarios,-Oracles (nondeterministic test cannot be repeated)
-Opportunity cost between manual and automated,

- Partial automation of any of the Test Step, the so called SEARCH (Setup, Execution, Analysis, Reporting, Cleanup, and Help)
59Too Much Test Automation? automation is verfication of basic qualityAutomation Cost, easiness to automateautomation scope, bugs in tests1
60What are good guidelines to determine when to automate a test that are: repetitive, simplistic, free up time for more complex tasks, are stable enough, provide a good measure of confidence in stability and qualityrepetitive tests for buggiest areasdeterministic, will yield pass/fail without false results1
61What's my Testing ROI? - High level estimation111n of releases, scope growth, cost of field defects,market value for product, working hours1manual test duration1cost of capital, , test effectivness improvement, field defects1111Jobman 4.0Mobile$392,5003,344,336CMDs’ estimated test automation ROI is 1,468% over two years.111
62What's Your Automation Index? the test require many data combinations using the same test steps

(i.e., multiple data inputs for the same feature)?
Does the test need to be verified on multiple software and hardware
Does the test cover most critical feature paths? Does the test cover high-risk areas?Are the expected results constant,Is the test impossible or prohibitively expensive to perform manually...?1Checklist of 12 questions on when to automate a test case1
63When and How Much to Automate UI testing that validates field types, Test complication, . code that are used by other parts of the application, . Key functionalities ... a rogue defect could potentially result in a lost customer.1
64When automated tests make sense, and when they don't code. Input checks. API code Important corner cases, Multiple people are commiting code to a piece of code. Human judgement needed (look and feel, video quality),Hard to manually setup tests.1
Favourig manual Localized changes, simple to set up,
65When Should a Test Be Automated? Load tests need to be automated. Lifetime of automation11Code changes break automation. Humans (manual tests) find different defects than automation, . product-specific test libraries may help test automation be resilient to change in SUT Code1
66When To Automate of breakageLikelihood breakage1Microsoft VisioOffice software security testing, render testing11
67When to Automate Testing
Script death (lifetime),
Core features, fragile comparison of results, , easy to find oracle, failure report accuracy,1 tedious to test manually, is it possibly to test this feature automatically11
68When to Automate Your Testing (and When Not To) driven, load performance,large data, GUI intensive, requires human intervention, requires 3rd party system, Complex Functionality static tests (not chaning) Subjective Validation,Skillset,Job Security (of manual testing, , Regression test, Smoke test,, coverage, productivity, accuracy, , undpredicatable results, New/Changing Functionality, Strategic Development,
69When to do Manual Testing and when to Automated Testing order of exec, simultanious exec, multiple data set for a single function, Regression or Smoke, Test report needed, Number of tests, , , random negative testing needed,High priority test team expertisefeasability of automation1
70When to Test Manually and When to Automate test cases1
71Why and When to Perform Automation Testing test,Script death (lifetime) Percentage of tests feasable for automation, , ease of automation
72Why How and When to Automate Testing do automation testing you should have the skilled resources Manual better when no time to automate
73What to Automate First test phase (maintenance, unit, system, acceptance (bad))easily repeatable needed precision in comparison, is human judgement oracle requiredwho creates and uses test automation,who controls changes, prior knowledge of changes (change management)1 effort to perform manuall, number of conditions (combinatorial explosion) in a testEasy / feasable to automate, easily performed, effort to analyze failures, effort to maintain1new defects / regressions, test coverage by automation, ,1finance3200%111
74Intelligent Mistakes in Test Automation things people do not like to do or do badly, test not possible or not practical to do manually (checking internal states of objects)111
75Return on Investment Calculator for Test Automation per year
76ROI Calculator by GlobalNowIT n of browsers, n of mobile platforms , test case change1 effort to executeeffort to automate1number of releases,
77IBM Rational Quality Management ROI Calculator, Type of App (Java, html, Sap, ...),1cost to automate1releases per year n of test cases, , n of manu testers, n of auto testers, , days testing, hour rate, licences
78Return on Investment (ROI) Calculator by AutomatedTestingInstitute test efforttest development, test exec, test suite analysis time, test suite maintenance time1n of manual tests, n of automated tests, n of executions per cycle, n of cycles, hourly rate, overnight execution, failure cost1
Main menu