MLR - When and what - ManAutoTest
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
ABCKOPQRSTUVWXYZAAABACADAEAFAGAHAIAJAKALAMANAOAPAQARASATAUAVAWAXAYAZBBBCBDBEBFBUBVBWBXBYBZCACBCCCECF
1
78Done:78785811761455116135034133532539643301715107881038141843725141219162824219911110171142
2
Demographic infoType of Paper - Contribution FacetType of Paper - Research FacetDecision supportFactors considered (in when/what to automate)Measurement (quantitative)Attributes of the software SUT(s) or project
3
#ResourcesLinkYearC / J / W / M / S / B / TAuthor Affiliation (A, I, C)Heuristics / guidelineMethod / techniqueToolModelMetricProcessEmpirical results onlyOther1-Solution Proposal (example)2-Validation Research (weak empirical study)3-Evaluation Research (strong empirical study)ExperiencePhilosophicalOpinionOtherWhen to automateWhat to automateROI (indirectly help what/when)SumStability of SUTOther SUT aspectsNeed for regression testingTest TypeTest reuse/ repeatabilityTest ImportanceTest oracleTest stabilityAutomation (test) toolSkills level of testersOther human and organizational factorsEconomic factorsReducing manual test effortAutomatability of testingEconomic factorsDevelopment processOtherCostBenefitROIOther metrics usedFormula#SU/ / project namesDomainsTest cost metricsTest benefit metricsROI of test automationExperimental / simple (toy) exampleReal open-sourceCommercialCommercial and quantitative evidenceNoteworthy phrases
4
Technical and scientific sources
5
1A Search-based Approach for Cost-Effective Software Test Automation-Decision Support and an Industrial Case Studyhttps://docs.google.com/file/d/0B6dKdxaNjBENMUJuMTU2R1UwbUE/edit?usp=drive_web2014WC111111111111Electronic Tour Sheet (ETS)Energy control software367% (267% return)111
6
2A way of Improving Test Automation Cost-Effectivenesshttps://drive.google.com/open?id=0B6dKdxaNjBENa2NqcEV1Znc0dE0&authuser=02006CA1111Test reuserelevance (severity of faults targetted by test cases)automation tool qualityresource (money, hw), test portabilitymanual complexity, manual exec effortAutomation effort111Nice idea. But no real validation or case
7
3Automated Unit Testing of a SCADA Control Software- An Industrial Case Study Based on Action Researchhttps://docs.google.com/file/d/0B6dKdxaNjBENOWFGeFRKUDJObXM/edit2012CC11a-posteriori (after the fact, Post-hoc) analysis11111RocketSCADA control software11138 hours11
8
4Comparative study of test automaton ROIhttps://docs.google.com/file/d/0B6dKdxaNjBENODVyT0R6STJNM2c/edit2011JA1111111111example1111
9
5Cost Benefits Analysis of Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENLUpscnM2SUFmUm8/edit1999CI111101111112Automation of GUI tests, Automation of build tests111.552 (55% return)11
10
6Cost-benefit analysis of using test automation in the development of embedded softwarehttps://docs.google.com/file/d/0B6dKdxaNjBENcHZ6VWhZY2VMSXc/edit2014TA111111211111413KONE elevator softwareembedded1140%11
11
7Economic perspectives in test automation- balancing automated and manual testing with opportunity costhttps://docs.google.com/file/d/0B6dKdxaNjBENeTVDWG4wam85c1E/edit?usp=drive_web2006WA1111101111111example11
12
8Establishment of automated regression testing at ABB industrial experience report on 'avoiding the pitfallshttps://drive.google.com/open?id=0B6dKdxaNjBENTVNFZ19ILUFiVkE&authuser=02004CI1110awareness and understanding of Concept of regression testingOrganizational maturity11
13
9Observations and lessons learned from automated testinghttps://docs.google.com/file/d/0B6dKdxaNjBENN1lXUTRVNzNCdUk/edit2005CI1111ROI related issues, e.g., expectations21115project a...eSystem to Manage Distribution of Assets, Java-Based Application Platform, Point of Sale System for Life Insurances, Sales Support for Tailored Industrial Facilities, System Test Automation for Control System11
14
10Pragmatic approach to software test automationhttps://docs.google.com/file/d/0B6dKdxaNjBENSHBWSEVqUlYxUFk/edit2013CI111How Much to Automate1Automation fails when the current application has unstable designWe need to think about automation in case of reruns where it means to reduce cost.low test risk and complexity stable enviroment and datatest engineer's skilldesign the test case first, first manual then automatte, SUT known, low test risk and complexity
15
11Software Test Automation in Practice: Empirical Observationshttps://docs.google.com/file/d/0B6dKdxaNjBENWVFLTWR1aFBpdmc/edit2010JA11survey10SUT features (Generic and independent products facilitate and customized and complex products hinder testing automation)High reusability facilitates and low reusability hinders testing automation1consider automation costs11212 software development organizationselectronics, Internet servics, Logistics software, Safety and logistics system, Financial software11
16
12Software test automation–Developing an infrastructure designed for successhttps://drive.google.com/open?id=0B6dKdxaNjBENTzBTNThtM2ZrT28&authuser=02001CI11111Automate tests for stable applicationsAutomate regression tests:Automate tests that have been written (manual test must exits first), Automate tests with no timing issuesboringness of the testautomatability18Building Technologies, HVAC (heating, ventilating, and air conditioning)1111
17
13Surviving the Top 10 Challenges of Software Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENLXNzWG5XMHpxTVE/edit2003JI1111SUT criticality / riskTest repeatabilityautomatability1
18
14The Return on Investment (ROI) of Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENZklNeTNycURLTlE/edit2012JI111121111111Dispense ApplicationPharmaceutical Engineering1111
19
15The When & How of Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENLS1zWm5zU1RNTjA/edit2005CI111110Stability of the product/application is ensureddeciding the right toolInterface to be tested is determined, Scope of automation defined, deciding the right test mode (script recording/script development)1
20
16Visual GUI Testing in Practice Challenges, Problems and Limitationshttps://docs.google.com/file/d/0B6dKdxaNjBENTEcyaTlkMHJrUVU/edit2014JA11a-posteriori (after the fact, Post-hoc) analysis1This shows that the SUT must have reached a certain level of maturity for VGT to be applicable[automate] more stable test cases first112two safety-critical systemssafety-critical and military systems113 and 14 executions111
21
17When to Automate Software Testing -Decision Support based on System Dynamicshttps://docs.google.com/file/d/0B6dKdxaNjBENRGZDWmN4SWJPMEE/edit?usp=drive_web2014CC1111111111111Electronic Tour Sheet (ETS)Energy control software111
22
Books and book chapters
23
18Advanced Software Testing - Vol. 2: Guide to the ISTQB Advanced Certification as an Advanced Test Managerhttps://drive.google.com/open?id=0B6dKdxaNjBENQ1plN2lsU2ZYU1k&authuser=02008BI11112Automate error prone [manual] tests1automatability1
24
19Automated software testing: introduction, management, and performancehttp://www.google.fi/books?hl=fi&lr=&id=kl2H0G6EFf0C&oi=fnd&pg=PR5&dq=decision+automate+software+testing&ots=wOTRxZqHvX&sig=QdMmUYOnq0H7Zus6gdeMhMe9FQ0&redir_esc=y#v=onepage&q=decision%20automate%20software%20testing&f=false2008BI11111Tool costtraining cost, tool expertise
25
20How Google Tests Softwarehttps://drive.google.com/file/d/0B3t8fTgRROXwR0V5am1UWEJuMlk/view?usp=sharing2012BI11111smoke tests, no human cleverness and intuition->it should be automated, 11automate 20% of test that find 80% of bugs
26
21Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality2009BI111112Adequate skills1automatability1111
27
22Just Enough Software Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENTk9mMFhiWE9UVUU/edit2002BI1111111complexity of SUT, feature set require a significant number of data combinations to validate1number of buildstest tool capable of interacting with all of the necessary attributes of the featuretight delivery schedule1
28
23Project 1: Failure, Project 2: Successhttps://docs.google.com/file/d/0B6dKdxaNjBENVld2OVk3eWVDYWc/edit2012BI11111Simple, Positive tests, Easy to continue to work with12Project 1, Project 2?11
29
24Software for Medical Devices and Our Need for Good Software Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENVld2OVk3eWVDYWc/edit2012BI11111114Phoenix, miniweb, doityourself, hamletmedical11
30
25Software Test Automation: Effective Use of Test Execution Toolshttps://docs.google.com/file/d/0B6dKdxaNjBENX1I2aW5TSmV3Z0U/edit1999BI11111, test diversity (to get a broad view of quality)often runtest importance, funtion importanceeasy to automate1
31
26Software Testing Principles and Practiceshttps://drive.google.com/open?id=0B6dKdxaNjBENWm52QmlDTDVCems&authuser=02006BI11111Automate Tests that Pertain to Standards
32
Internet articles and white papers
33
27Achieving business benefits through automated software testinghttps://docs.google.com/file/d/0B6dKdxaNjBENX1FBSHR1RFZfcms/edit2012IAWPI11101Test repeatabilityease of deveoping automated test oracles111Complexity of the test environment11an embedded softwareembedded11
34
28Agile Testers, Should You Automate?http://professionalservices.matrixresources.com/blog/agile-testers-should-you-automate2013IAWPI111011Agile context for the article
1
35
29Automated Testing Advantages, Disadvantages and Guidelineshttp://www.exforsys.com/tutorials/testing/automated-testing-advantages-disadvantages-and-guidelines.html2005IAWPI111111well-developed and well-understood use cases or scenarios firstredundant tasks or scenarios
36
30Automating tests vs. test-automationhttp://googletesting.blogspot.com.tr/2007/10/automating-tests-vs-test-automation.html2007IAWPI1111111testability1store-administration systementerprise11
37
31Automation Selection Criteria – Picking the “Right” Candidateshttps://docs.google.com/file/d/0B6dKdxaNjBENYlVwMWV6RmczUHM/edit2008IAWPI1111111111
38
32Automation Testing versus Manual Testing Guidelineshttp://blogs.msdn.com/b/saraford/archive/2005/02/07/368833.aspx2005IAWPI111121importance of test case11
39
33Avoid Throwaway Test Automation
https://docs.google.com/file/d/0B6dKdxaNjBENblNzcGk4RHFhdDQ/edit
2008IAWPI1111111
40
34Choosing What To Automatehttps://docs.google.com/file/d/0B6dKdxaNjBENSkNGdGNiZklaamc/edit2010IAWPI111111
length of transactions (test cases)
Highest priority features1113POS Credit Card Application, Retail web site, Software Installation1
41
35Determining What to Automatehttps://docs.google.com/file/d/0B6dKdxaNjBENREhfUVhrMXhnUjQ/edit2013IAWPI1111111Stable test cases first11Test Case: Generic Application Home Page Login/Navigate Testweb apps11
42
36Does Test Automation Save Time and Money?http://www.stickyminds.com/article/does-test-automation-save-time-and-money2001IAWPI11111
43
37For those of you dreaming the 100% automation dream...please wake up!http://blogs.msdn.com/b/imtesty/archive/2006/08/02/686010.aspx2006IAWPI1111111non functional testing1
44
38Functional Test Automation - from SAP testing bookhttps://docs.google.com/file/d/0B6dKdxaNjBENLXMwc0trZWlYNUk/edit2006IAWPI111111predictability of test inputs predictability of expected outputs (oracle)1111
45
39If you are going to run a test more than once, it should be automated.http://www.teknologika.com/blog/tenet-if-you-are-going-to-run-a-test-more-than-once-it-should-be-automated/2005IAWPI11111automated build verification tests111Unnnamed$1M+ investment of tests on a rack of 25 dedicated machines shortening their regression testing cycle by 75%11
46
40Learn When to Automate and When Not To- 99 Ways Workshop #5http://www.mkltesthead.com/2013/07/99-ways-workshop-5-learn-when-to.html2013IAWPI111111111
47
41Manual vs. Automated Testing - explored - The Borland Bloghttp://blog.borland.com/manualvsautomatedtesting/192/2013IAWPI111011Load and performance testingTesting across multiple OS platforms and multi lingual sitesautomate High Risk casesresource limitation1
48
42Regression Testing – What to Automate and howhttps://docs.google.com/file/d/0B6dKdxaNjBENamdOVlhJeTN0MXM/edit1997IAWPI11111easy tests firsteasy tests first
49
43ROI on Test Automation - A Simple Yet Powerful Approachhttps://docs.google.com/file/d/0B6dKdxaNjBENajFLcmpCdUpWOHM/edit2008IAWPI11ROI in support of when/ what11111111an example1
50
44Sources of Return on Investment (ROI) in Test Automation - QA InfoTech Corporate Bloghttp://www.qainfotech.com/blog/2012/07/sources-of-return-on-investment-roi-in-test-automation/2012IAWPI11ROI in support of when/ what111automation makes the work more interesting and challenging, extra coverage from automation
51
45Test Automation: The Whens, Hows, and Whyshttp://www.deriskit.com/news-article_test-automation-the-whens-hows-and-whys.html2014IAWPI1111111high number of tests
52
46Test Automation – Knowing When to Automatehttp://oshyn.com/software-development/test-automation-when-automate2012IAWPI111111
53
47Test Automation and Best Practiceshttps://www.atlassian.com/test-automation2014IAWPI11111111JIRAdefect management software$16,50022 iterations111
54
48Test Automation for Effective Post Deployment testinghttps://docs.google.com/file/d/0B6dKdxaNjBENSHdWYkFxN2JZLXc/edit2007IAWPI111101, Long lifecycle application1Proof of concept is needed111Telecom application for Public health careRegression test 48h -> 1.5h Over next 3 years 150,000$92.5% productivity improvement11
55
49Test Automation Patternshttp://www.dorothygraham.co.uk/patterns/desPatterns/index.html2014IAWPI1111Automate tests that bring most ROI21111
56
50Test Automation Process https://docs.google.com/file/d/0B6dKdxaNjBENQzJOZFFXVVVkbzg/edit2007IAWPI11101 Customized applications with short life span. Life span of the product is long.1Check the functionality with huge set of data.
Verify database which contains additional data.
Time and resources are a constraint.1Spending a lot of time by testing manuallyComplexity of Application. Main Application has lot of interdependency with other Applications which in turn cannot be automated. 1WHEN:
Test application with in less time. Code coverage is vast.
57
51Test Automation ROIhttp://blogs.msdn.com/b/imtesty/archive/2009/09/02/test-automation-roi-part-ii.aspx2009IAWPI1111No ROI on high level, but level of each test111quality of oracles11Relialbility of tests
58
52Test Automation ROI Calculatorhttps://docs.google.com/file/d/0B6dKdxaNjBENQ0xyQ09CWVEtX3M/edit2010IAWPI111ROI with various input factors11Number of configurations to be tested
Infrastructure cost
1Cost of the testing resources1Releases planned for a yearRequired regression cycle per year,
Number of existing regression test cases
59
53Test Automation- Delivering Business Valuehttps://docs.google.com/file/d/0B6dKdxaNjBENMElMalRvOEhZbXc/edit2008IAWPI1111111
60
54Test automation: When, how and how muchhttps://docs.google.com/file/d/0B6dKdxaNjBENa2tqc2k4UEEtZm8/edit2011IAWPI11111available technologies development velocity, Any test automation program should start with an honest assessment of the people, process and available technologies.1Task suitablity for automation (good test data creation, smoke and regression test1process, development paradigm,Test automation target, quality objective, Manual testing is not complemented time frame1
61
55Testing in the Fast Lane: Automating Acceptance Testing in an Extreme Programming Environmenthttps://docs.google.com/file/d/0B6dKdxaNjBENWUxsSHdBWlVMdXc/edit2002IAWPI11111We automated tests for critical business functionality first, and expanded the tests as we had time.We chose or created lightweight test tools which feature both power and flexibility.process, XP agileWe tried to avoid creating a suite of tests which takes more time to maintain than we had available.Agile context for the article
We evaluated each test to see if it makes sense to automate.
1a telephone directory
lookup website
Web1
62
56The Business Argument for Investing in Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENa2NwMlJqRkZlOG8/edit2012IAWPI11111
GUI testing, Unit testing
1111Not givenNot given1
63
57The ROI of Test Automationhttps://docs.google.com/file/d/0B6dKdxaNjBENSUtCNTZod0MwN2s/edit2007IAWPI11112Type of tests. E.g. performance there is no other practical way to do it.1 to automate the tests I couldn’t possibly perform manually. 1
What are automation goals?
1) automate all things that offer immediate returns and
2) if you have a hard time convincing someone that it needs to be automated and if you feel you need an equation to see the long term advantages of automation, don’t bother automating it.
11
64
58Too much automation or not enough? When to automatehttps://docs.google.com/file/d/0B6dKdxaNjBENdHd1OFExQ2NsaDQ/edit2009IAWPI1111111unit test (yes), build verification test (yes), Usability test (NO),-Reasons repeating same test changes in input/software/enviroment, Test that finds bugs, validate important scenarios,-Oracles (nondeterministic test cannot be repeated)
-Opportunity cost between manual and automated,


- Partial automation of any of the Test Step, the so called SEARCH (Setup, Execution, Analysis, Reporting, Cleanup, and Help)
1
65
59Too Much Test Automation?http://blogs.msdn.com/b/steverowe/archive/2005/01/28/362668.aspx2005IAWPI11111 automation is verfication of basic qualityAutomation Cost, easiness to automateautomation scope, bugs in tests1
66
60What are good guidelines to determine when to automate a testhttp://sqa.stackexchange.com/questions/822/what-are-good-guidelines-to-determine-when-to-automate-a-test2012IAWPI111111Test that are: repetitive, simplistic, free up time for more complex tasks, are stable enough, provide a good measure of confidence in stability and qualityrepetitive tests for buggiest areasdeterministic, will yield pass/fail without false results1
67
61What's my Testing ROI?https://docs.google.com/file/d/0B6dKdxaNjBENOUI4UVlBaWoyaHc/edit2011IAWPI1111ROI - High level estimation111n of releases, scope growth, cost of field defects,market value for product, working hours1manual test duration1cost of capital, , test effectivness improvement, field defects1111Jobman 4.0Mobile$392,5003,344,336CMDs’ estimated test automation ROI is 1,468% over two years.111
68
62What's Your Automation Index?https://docs.google.com/file/d/0B6dKdxaNjBENd1diUEs4SC1lR1E/edit2007IAWPI1111111Does the test require many data combinations using the same test steps

(i.e., multiple data inputs for the same feature)?
Does the test need to be verified on multiple software and hardware
configurations?
Does the test cover most critical feature paths? Does the test cover high-risk areas?Are the expected results constant,Is the test impossible or prohibitively expensive to perform manually...?1Checklist of 12 questions on when to automate a test case1
69
63When and How Much to Automatehttp://www.bqurious.com/when-and-how-much-to-automate/2014IAWPI1111111Mundane UI testing that validates field types, Test complication, . code that are used by other parts of the application, . Key functionalities ... a rogue defect could potentially result in a lost customer.1
70
64When automated tests make sense, and when they don'thttp://ole-laursen.blogspot.com.tr/2012/03/when-automated-tests-make-sense-and.html2012IAWPI111111Complex code. Input checks. API code Important corner cases, Multiple people are commiting code to a piece of code. Human judgement needed (look and feel, video quality),Hard to manually setup tests.1
Favourig manual Localized changes, simple to set up,
1
71
65When Should a Test Be Automated?https://docs.google.com/file/d/0B6dKdxaNjBENZUI2azBwMEd0b2s/edit1998IAWPI11111 Load tests need to be automated. Lifetime of automation11Code changes break automation. Humans (manual tests) find different defects than automation, . product-specific test libraries may help test automation be resilient to change in SUT Code1
72
66When To Automatehttp://blogs.msdn.com/b/micahel/archive/2005/12/14/whentoautomate.aspx2005IAWPI11111111Likelihood of breakageLikelihood breakage1Microsoft VisioOffice software security testing, render testing11
73
67When to Automate Testinghttp://davidweiss.blogspot.com.tr/2006/08/when-to-automate-testing.html2006IAWPI11111211
Script death (lifetime),
Core features, fragile comparison of results, , easy to find oracle, failure report accuracy,1 tedious to test manually, is it possibly to test this feature automatically11
74
68When to Automate Your Testing (and When Not To)https://docs.google.com/file/d/0B6dKdxaNjBENSjN5cHpWQVdXVXc/edit2013IAWPI1111211Data driven, load performance,large data, GUI intensive, requires human intervention, requires 3rd party system, Complex Functionality static tests (not chaning) Subjective Validation,Skillset,Job Security (of manual testing, , Regression test, Smoke test,, coverage, productivity, accuracy, , undpredicatable results, New/Changing Functionality, Strategic Development,
75
69When to do Manual Testing and when to Automated Testinghttp://blog.testing-whiz.com/2011/12/when-to-do-manual-testing-and-when-to.html2011IAWPI1111111predetermined order of exec, simultanious exec, multiple data set for a single function, Regression or Smoke, Test report needed, Number of tests, , , random negative testing needed,High priority test team expertisefeasability of automation1
76
70When to Test Manually and When to Automatehttp://blogs.msdn.com/b/steverowe/archive/2008/02/26/when-to-test-manually-and-when-to-automate.aspx?Redirected=true2008IAWPI1111111major test cases1
77
71Why and When to Perform Automation Testinghttp://intensetesting.wordpress.com/2013/11/07/why-and-when-to-perform-automation-testing/2013IAWPI111111Load test,Script death (lifetime) Percentage of tests feasable for automation, , ease of automation
78
72Why How and When to Automate Testinghttp://www.softwaretestingclass.com/why-how-and-when-to-automate-software-testing/2014IAWPI1111111To do automation testing you should have the skilled resources Manual better when no time to automate
79
YouTube
80
73What to Automate Firsthttp://www.youtube.com/watch?v=eo66ouKGyVk2014YouTubeI111111what test phase (maintenance, unit, system, acceptance (bad))easily repeatable needed precision in comparison, is human judgement oracle requiredwho creates and uses test automation,who controls changes, prior knowledge of changes (change management)1 effort to perform manuall, number of conditions (combinatorial explosion) in a testEasy / feasable to automate, easily performed, effort to analyze failures, effort to maintain1new defects / regressions, test coverage by automation, ,1finance3200%111
81
74Intelligent Mistakes in Test Automationhttps://www.youtube.com/watch?v=Dbim6ZkSHBg2013YouTubeI111111automate things people do not like to do or do badly, test not possible or not practical to do manually (checking internal states of objects)111
82
tools
83
75Return on Investment Calculator for Test Automationhttp://www.elbrus.com/services/test_automation_roi_calc/TOOLI111111releases per year
84
76ROI Calculator by GlobalNowIT http://globalnowit.com/calculator/TOOLI1111 n of browsers, n of mobile platforms , test case change1 effort to executeeffort to automate1number of releases,
85
77IBM Rational Quality Management ROI Calculatorhttp://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ROI_Rational.htmlTOOLI1111, Type of App (Java, html, Sap, ...),1cost to automate1releases per year n of test cases, , n of manu testers, n of auto testers, , days testing, hour rate, licences
86
78Return on Investment (ROI) Calculator by AutomatedTestingInstitutehttp://www.automatedtestinginstitute.com/home/index.php?Itemid=65TOOLI11111training1Manual test efforttest development, test exec, test suite analysis time, test suite maintenance time1n of manual tests, n of automated tests, n of executions per cycle, n of cycles, hourly rate, overnight execution, failure cost1
Loading...
Main menu