| A | B | C | K | O | P | Q | R | S | T | U | V | W | X | Y | Z | AA | AB | AC | AD | AE | AF | AG | AH | AI | AJ | AK | AL | AM | AN | AO | AP | AQ | AR | AS | AT | AU | AV | AW | AX | AY | AZ | BB | BC | BD | BE | BF | BU | BV | BW | BX | BY | BZ | CA | CB | CC | CE | CF | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 78 | Done: | 78 | 78 | 58 | 11 | 7 | 6 | 1 | 4 | 5 | 5 | 11 | 6 | 1 | 35 | 0 | 34 | 1 | 33 | 53 | 25 | 39 | 6 | 43 | 30 | 17 | 15 | 10 | 7 | 8 | 8 | 10 | 38 | 14 | 18 | 43 | 7 | 25 | 14 | 12 | 19 | 1 | 6 | 28 | 24 | 21 | 9 | 9 | 11 | 11 | 0 | 17 | 11 | 42 | ||||
2 | Demographic info | Type of Paper - Contribution Facet | Type of Paper - Research Facet | Decision support | Factors considered (in when/what to automate) | Measurement (quantitative) | Attributes of the software SUT(s) or project | ||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | # | Resources | Link | Year | C / J / W / M / S / B / T | Author Affiliation (A, I, C) | Heuristics / guideline | Method / technique | Tool | Model | Metric | Process | Empirical results only | Other | 1-Solution Proposal (example) | 2-Validation Research (weak empirical study) | 3-Evaluation Research (strong empirical study) | Experience | Philosophical | Opinion | Other | When to automate | What to automate | ROI (indirectly help what/when) | Sum | Stability of SUT | Other SUT aspects | Need for regression testing | Test Type | Test reuse/ repeatability | Test Importance | Test oracle | Test stability | Automation (test) tool | Skills level of testers | Other human and organizational factors | Economic factors | Reducing manual test effort | Automatability of testing | Economic factors | Development process | Other | Cost | Benefit | ROI | Other metrics used | Formula | # | SU/ / project names | Domains | Test cost metrics | Test benefit metrics | ROI of test automation | Experimental / simple (toy) example | Real open-source | Commercial | Commercial and quantitative evidence | Noteworthy phrases | |
4 | Technical and scientific sources | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
5 | 1 | A Search-based Approach for Cost-Effective Software Test Automation-Decision Support and an Industrial Case Study | https://docs.google.com/file/d/0B6dKdxaNjBENMUJuMTU2R1UwbUE/edit?usp=drive_web | 2014 | W | C | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Electronic Tour Sheet (ETS) | Energy control software | 367% (267% return) | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||
6 | 2 | A way of Improving Test Automation Cost-Effectiveness | https://drive.google.com/open?id=0B6dKdxaNjBENa2NqcEV1Znc0dE0&authuser=0 | 2006 | C | A | 1 | 1 | 1 | 1 | Test reuse | relevance (severity of faults targetted by test cases) | automation tool quality | resource (money, hw), test portability | manual complexity, manual exec effort | Automation effort | 1 | 1 | 1 | Nice idea. But no real validation or case | |||||||||||||||||||||||||||||||||||||||
7 | 3 | Automated Unit Testing of a SCADA Control Software- An Industrial Case Study Based on Action Research | https://docs.google.com/file/d/0B6dKdxaNjBENOWFGeFRKUDJObXM/edit | 2012 | C | C | 1 | 1 | a-posteriori (after the fact, Post-hoc) analysis | 1 | 1 | 1 | 1 | 1 | Rocket | SCADA control software | 1 | 1 | 138 hours | 1 | 1 | ||||||||||||||||||||||||||||||||||||||
8 | 4 | Comparative study of test automaton ROI | https://docs.google.com/file/d/0B6dKdxaNjBENODVyT0R6STJNM2c/edit | 2011 | J | A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | example | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||
9 | 5 | Cost Benefits Analysis of Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENLUpscnM2SUFmUm8/edit | 1999 | C | I | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | Automation of GUI tests, Automation of build tests | 1 | 1 | 1.552 (55% return) | 1 | 1 | |||||||||||||||||||||||||||||||||||
10 | 6 | Cost-benefit analysis of using test automation in the development of embedded software | https://docs.google.com/file/d/0B6dKdxaNjBENcHZ6VWhZY2VMSXc/edit | 2014 | T | A | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 4 | 1 | 3 | KONE elevator software | embedded | 1 | 1 | 40% | 1 | 1 | |||||||||||||||||||||||||||||||
11 | 7 | Economic perspectives in test automation- balancing automated and manual testing with opportunity cost | https://docs.google.com/file/d/0B6dKdxaNjBENeTVDWG4wam85c1E/edit?usp=drive_web | 2006 | W | A | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | example | 1 | 1 | |||||||||||||||||||||||||||||||||||||
12 | 8 | Establishment of automated regression testing at ABB industrial experience report on 'avoiding the pitfalls | https://drive.google.com/open?id=0B6dKdxaNjBENTVNFZ19ILUFiVkE&authuser=0 | 2004 | C | I | 1 | 1 | 1 | 0 | awareness and understanding of Concept of regression testing | Organizational maturity | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||||
13 | 9 | Observations and lessons learned from automated testing | https://docs.google.com/file/d/0B6dKdxaNjBENN1lXUTRVNzNCdUk/edit | 2005 | C | I | 1 | 1 | 1 | 1 | ROI related issues, e.g., expectations | 2 | 1 | 1 | 1 | 5 | project a...e | System to Manage Distribution of Assets, Java-Based Application Platform, Point of Sale System for Life Insurances, Sales Support for Tailored Industrial Facilities, System Test Automation for Control System | 1 | 1 | |||||||||||||||||||||||||||||||||||||||
14 | 10 | Pragmatic approach to software test automation | https://docs.google.com/file/d/0B6dKdxaNjBENSHBWSEVqUlYxUFk/edit | 2013 | C | I | 1 | 1 | 1 | How Much to Automate | 1 | Automation fails when the current application has unstable design | We need to think about automation in case of reruns where it means to reduce cost. | low test risk and complexity | stable enviroment and data | test engineer's skill | design the test case first, first manual then automatte, SUT known, low test risk and complexity | ||||||||||||||||||||||||||||||||||||||||||
15 | 11 | Software Test Automation in Practice: Empirical Observations | https://docs.google.com/file/d/0B6dKdxaNjBENWVFLTWR1aFBpdmc/edit | 2010 | J | A | 1 | 1 | survey | 1 | 0 | SUT features (Generic and independent products facilitate and customized and complex products hinder testing automation) | High reusability facilitates and low reusability hinders testing automation | 1 | consider automation costs | 1 | 12 | 12 software development organizations | electronics, Internet servics, Logistics software, Safety and logistics system, Financial software | 1 | 1 | ||||||||||||||||||||||||||||||||||||||
16 | 12 | Software test automation–Developing an infrastructure designed for success | https://drive.google.com/open?id=0B6dKdxaNjBENTzBTNThtM2ZrT28&authuser=0 | 2001 | C | I | 1 | 1 | 1 | 1 | 1 | Automate tests for stable applications | Automate regression tests: | Automate tests that have been written (manual test must exits first), Automate tests with no timing issues | boringness of the test | automatability | 1 | 8 | Building Technologies, HVAC (heating, ventilating, and air conditioning) | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||
17 | 13 | Surviving the Top 10 Challenges of Software Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENLXNzWG5XMHpxTVE/edit | 2003 | J | I | 1 | 1 | 1 | 1 | SUT criticality / risk | Test repeatability | automatability | 1 | |||||||||||||||||||||||||||||||||||||||||||||
18 | 14 | The Return on Investment (ROI) of Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENZklNeTNycURLTlE/edit | 2012 | J | I | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Dispense Application | Pharmaceutical Engineering | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||
19 | 15 | The When & How of Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENLS1zWm5zU1RNTjA/edit | 2005 | C | I | 1 | 1 | 1 | 1 | 1 | 0 | Stability of the product/application is ensured | deciding the right tool | Interface to be tested is determined, Scope of automation defined, deciding the right test mode (script recording/script development) | 1 | |||||||||||||||||||||||||||||||||||||||||||
20 | 16 | Visual GUI Testing in Practice Challenges, Problems and Limitations | https://docs.google.com/file/d/0B6dKdxaNjBENTEcyaTlkMHJrUVU/edit | 2014 | J | A | 1 | 1 | a-posteriori (after the fact, Post-hoc) analysis | 1 | This shows that the SUT must have reached a certain level of maturity for VGT to be applicable | [automate] more stable test cases first | 1 | 1 | 2 | two safety-critical systems | safety-critical and military systems | 1 | 13 and 14 executions | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||
21 | 17 | When to Automate Software Testing -Decision Support based on System Dynamics | https://docs.google.com/file/d/0B6dKdxaNjBENRGZDWmN4SWJPMEE/edit?usp=drive_web | 2014 | C | C | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Electronic Tour Sheet (ETS) | Energy control software | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||
22 | Books and book chapters | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
23 | 18 | Advanced Software Testing - Vol. 2: Guide to the ISTQB Advanced Certification as an Advanced Test Manager | https://drive.google.com/open?id=0B6dKdxaNjBENQ1plN2lsU2ZYU1k&authuser=0 | 2008 | B | I | 1 | 1 | 1 | 1 | 2 | Automate error prone [manual] tests | 1 | automatability | 1 | ||||||||||||||||||||||||||||||||||||||||||||
24 | 19 | Automated software testing: introduction, management, and performance | http://www.google.fi/books?hl=fi&lr=&id=kl2H0G6EFf0C&oi=fnd&pg=PR5&dq=decision+automate+software+testing&ots=wOTRxZqHvX&sig=QdMmUYOnq0H7Zus6gdeMhMe9FQ0&redir_esc=y#v=onepage&q=decision%20automate%20software%20testing&f=false | 2008 | B | I | 1 | 1 | 1 | 1 | 1 | Tool cost | training cost, tool expertise | ||||||||||||||||||||||||||||||||||||||||||||||
25 | 20 | How Google Tests Software | https://drive.google.com/file/d/0B3t8fTgRROXwR0V5am1UWEJuMlk/view?usp=sharing | 2012 | B | I | 1 | 1 | 1 | 1 | 1 | smoke tests, no human cleverness and intuition->it should be automated, | 1 | 1 | automate 20% of test that find 80% of bugs | ||||||||||||||||||||||||||||||||||||||||||||
26 | 21 | Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality | 2009 | B | I | 1 | 1 | 1 | 1 | 1 | 2 | Adequate skills | 1 | automatability | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||
27 | 22 | Just Enough Software Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENTk9mMFhiWE9UVUU/edit | 2002 | B | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | complexity of SUT, feature set require a significant number of data combinations to validate | 1 | number of builds | test tool capable of interacting with all of the necessary attributes of the feature | tight delivery schedule | 1 | ||||||||||||||||||||||||||||||||||||||||
28 | 23 | Project 1: Failure, Project 2: Success | https://docs.google.com/file/d/0B6dKdxaNjBENVld2OVk3eWVDYWc/edit | 2012 | B | I | 1 | 1 | 1 | 1 | 1 | Simple, Positive tests, Easy to continue to work with | 1 | 2 | Project 1, Project 2 | ? | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||
29 | 24 | Software for Medical Devices and Our Need for Good Software Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENVld2OVk3eWVDYWc/edit | 2012 | B | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 4 | Phoenix, miniweb, doityourself, hamlet | medical | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||
30 | 25 | Software Test Automation: Effective Use of Test Execution Tools | https://docs.google.com/file/d/0B6dKdxaNjBENX1I2aW5TSmV3Z0U/edit | 1999 | B | I | 1 | 1 | 1 | 1 | 1 | , test diversity (to get a broad view of quality) | often run | test importance, funtion importance | easy to automate | 1 | |||||||||||||||||||||||||||||||||||||||||||
31 | 26 | Software Testing Principles and Practices | https://drive.google.com/open?id=0B6dKdxaNjBENWm52QmlDTDVCems&authuser=0 | 2006 | B | I | 1 | 1 | 1 | 1 | 1 | Automate Tests that Pertain to Standards | |||||||||||||||||||||||||||||||||||||||||||||||
32 | Internet articles and white papers | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
33 | 27 | Achieving business benefits through automated software testing | https://docs.google.com/file/d/0B6dKdxaNjBENX1FBSHR1RFZfcms/edit | 2012 | IAWP | I | 1 | 1 | 1 | 0 | 1 | Test repeatability | ease of deveoping automated test oracles | 1 | 1 | 1 | Complexity of the test environment | 1 | 1 | an embedded software | embedded | 1 | 1 | ||||||||||||||||||||||||||||||||||||
34 | 28 | Agile Testers, Should You Automate? | http://professionalservices.matrixresources.com/blog/agile-testers-should-you-automate | 2013 | IAWP | I | 1 | 1 | 1 | 0 | 1 | 1 | Agile context for the article | 1 | |||||||||||||||||||||||||||||||||||||||||||||
35 | 29 | Automated Testing Advantages, Disadvantages and Guidelines | http://www.exforsys.com/tutorials/testing/automated-testing-advantages-disadvantages-and-guidelines.html | 2005 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | well-developed and well-understood use cases or scenarios first | redundant tasks or scenarios | |||||||||||||||||||||||||||||||||||||||||||||
36 | 30 | Automating tests vs. test-automation | http://googletesting.blogspot.com.tr/2007/10/automating-tests-vs-test-automation.html | 2007 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | testability | 1 | store-administration system | enterprise | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||
37 | 31 | Automation Selection Criteria – Picking the “Right” Candidates | https://docs.google.com/file/d/0B6dKdxaNjBENYlVwMWV6RmczUHM/edit | 2008 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||
38 | 32 | Automation Testing versus Manual Testing Guidelines | http://blogs.msdn.com/b/saraford/archive/2005/02/07/368833.aspx | 2005 | IAWP | I | 1 | 1 | 1 | 1 | 2 | 1 | importance of test case | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||
39 | 33 | Avoid Throwaway Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENblNzcGk4RHFhdDQ/edit | 2008 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||||
40 | 34 | Choosing What To Automate | https://docs.google.com/file/d/0B6dKdxaNjBENSkNGdGNiZklaamc/edit | 2010 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | length of transactions (test cases) | Highest priority features | 1 | 1 | 1 | 3 | POS Credit Card Application, Retail web site, Software Installation | 1 | |||||||||||||||||||||||||||||||||||||||
41 | 35 | Determining What to Automate | https://docs.google.com/file/d/0B6dKdxaNjBENREhfUVhrMXhnUjQ/edit | 2013 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Stable test cases first | 1 | 1 | Test Case: Generic Application Home Page Login/Navigate Test | web apps | 1 | 1 | |||||||||||||||||||||||||||||||||||||||
42 | 36 | Does Test Automation Save Time and Money? | http://www.stickyminds.com/article/does-test-automation-save-time-and-money | 2001 | IAWP | I | 1 | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||||||
43 | 37 | For those of you dreaming the 100% automation dream...please wake up! | http://blogs.msdn.com/b/imtesty/archive/2006/08/02/686010.aspx | 2006 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | non functional testing | 1 | ||||||||||||||||||||||||||||||||||||||||||||
44 | 38 | Functional Test Automation - from SAP testing book | https://docs.google.com/file/d/0B6dKdxaNjBENLXMwc0trZWlYNUk/edit | 2006 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | predictability of test inputs | predictability of expected outputs (oracle) | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||
45 | 39 | If you are going to run a test more than once, it should be automated. | http://www.teknologika.com/blog/tenet-if-you-are-going-to-run-a-test-more-than-once-it-should-be-automated/ | 2005 | IAWP | I | 1 | 1 | 1 | 1 | 1 | automated build verification tests | 1 | 1 | 1 | Unnnamed | $1M+ investment of tests on a rack of 25 dedicated machines | shortening their regression testing cycle by 75% | 1 | 1 | |||||||||||||||||||||||||||||||||||||||
46 | 40 | Learn When to Automate and When Not To- 99 Ways Workshop #5 | http://www.mkltesthead.com/2013/07/99-ways-workshop-5-learn-when-to.html | 2013 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||
47 | 41 | Manual vs. Automated Testing - explored - The Borland Blog | http://blog.borland.com/manualvsautomatedtesting/192/ | 2013 | IAWP | I | 1 | 1 | 1 | 0 | 1 | 1 | Load and performance testing | Testing across multiple OS platforms and multi lingual sites | automate High Risk cases | resource limitation | 1 | ||||||||||||||||||||||||||||||||||||||||||
48 | 42 | Regression Testing – What to Automate and how | https://docs.google.com/file/d/0B6dKdxaNjBENamdOVlhJeTN0MXM/edit | 1997 | IAWP | I | 1 | 1 | 1 | 1 | 1 | easy tests first | easy tests first | ||||||||||||||||||||||||||||||||||||||||||||||
49 | 43 | ROI on Test Automation - A Simple Yet Powerful Approach | https://docs.google.com/file/d/0B6dKdxaNjBENajFLcmpCdUpWOHM/edit | 2008 | IAWP | I | 1 | 1 | ROI in support of when/ what | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | an example | 1 | ||||||||||||||||||||||||||||||||||||||||
50 | 44 | Sources of Return on Investment (ROI) in Test Automation - QA InfoTech Corporate Blog | http://www.qainfotech.com/blog/2012/07/sources-of-return-on-investment-roi-in-test-automation/ | 2012 | IAWP | I | 1 | 1 | ROI in support of when/ what | 1 | 1 | 1 | automation makes the work more interesting and challenging, extra coverage from automation | ||||||||||||||||||||||||||||||||||||||||||||||
51 | 45 | Test Automation: The Whens, Hows, and Whys | http://www.deriskit.com/news-article_test-automation-the-whens-hows-and-whys.html | 2014 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | high number of tests | |||||||||||||||||||||||||||||||||||||||||||||
52 | 46 | Test Automation – Knowing When to Automate | http://oshyn.com/software-development/test-automation-when-automate | 2012 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||||||
53 | 47 | Test Automation and Best Practices | https://www.atlassian.com/test-automation | 2014 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | JIRA | defect management software | $16,500 | 22 iterations | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||
54 | 48 | Test Automation for Effective Post Deployment testing | https://docs.google.com/file/d/0B6dKdxaNjBENSHdWYkFxN2JZLXc/edit | 2007 | IAWP | I | 1 | 1 | 1 | 1 | 0 | 1 | , Long lifecycle application | 1 | Proof of concept is needed | 1 | 1 | 1 | Telecom application for Public health care | Regression test 48h -> 1.5h Over next 3 years 150,000$ | 92.5% productivity improvement | 1 | 1 | ||||||||||||||||||||||||||||||||||||
55 | 49 | Test Automation Patterns | http://www.dorothygraham.co.uk/patterns/desPatterns/index.html | 2014 | IAWP | I | 1 | 1 | 1 | 1 | Automate tests that bring most ROI | 2 | 1 | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||
56 | 50 | Test Automation Process | https://docs.google.com/file/d/0B6dKdxaNjBENQzJOZFFXVVVkbzg/edit | 2007 | IAWP | I | 1 | 1 | 1 | 0 | 1 | Customized applications with short life span. Life span of the product is long. | 1 | Check the functionality with huge set of data. Verify database which contains additional data. | Time and resources are a constraint. | 1 | Spending a lot of time by testing manually | Complexity of Application. Main Application has lot of interdependency with other Applications which in turn cannot be automated. | 1 | WHEN: Test application with in less time. Code coverage is vast. | |||||||||||||||||||||||||||||||||||||||
57 | 51 | Test Automation ROI | http://blogs.msdn.com/b/imtesty/archive/2009/09/02/test-automation-roi-part-ii.aspx | 2009 | IAWP | I | 1 | 1 | 1 | 1 | No ROI on high level, but level of each test | 1 | 1 | 1 | quality of oracles | 1 | 1 | Relialbility of tests | |||||||||||||||||||||||||||||||||||||||||
58 | 52 | Test Automation ROI Calculator | https://docs.google.com/file/d/0B6dKdxaNjBENQ0xyQ09CWVEtX3M/edit | 2010 | IAWP | I | 1 | 1 | 1 | ROI with various input factors | 1 | 1 | Number of configurations to be tested | Infrastructure cost | 1 | Cost of the testing resources | 1 | Releases planned for a year | Required regression cycle per year, Number of existing regression test cases | ||||||||||||||||||||||||||||||||||||||||
59 | 53 | Test Automation- Delivering Business Value | https://docs.google.com/file/d/0B6dKdxaNjBENMElMalRvOEhZbXc/edit | 2008 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||||
60 | 54 | Test automation: When, how and how much | https://docs.google.com/file/d/0B6dKdxaNjBENa2tqc2k4UEEtZm8/edit | 2011 | IAWP | I | 1 | 1 | 1 | 1 | 1 | available technologies | development velocity, Any test automation program should start with an honest assessment of the people, process and available technologies. | 1 | Task suitablity for automation (good test data creation, smoke and regression test | 1 | process, development paradigm, | Test automation target, quality objective, Manual testing is not complemented time frame | 1 | ||||||||||||||||||||||||||||||||||||||||
61 | 55 | Testing in the Fast Lane: Automating Acceptance Testing in an Extreme Programming Environment | https://docs.google.com/file/d/0B6dKdxaNjBENWUxsSHdBWlVMdXc/edit | 2002 | IAWP | I | 1 | 1 | 1 | 1 | 1 | We automated tests for critical business functionality first, and expanded the tests as we had time. | We chose or created lightweight test tools which feature both power and flexibility. | process, XP agile | We tried to avoid creating a suite of tests which takes more time to maintain than we had available. | Agile context for the article | We evaluated each test to see if it makes sense to automate. | 1 | a telephone directory lookup website | Web | 1 | ||||||||||||||||||||||||||||||||||||||
62 | 56 | The Business Argument for Investing in Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENa2NwMlJqRkZlOG8/edit | 2012 | IAWP | I | 1 | 1 | 1 | 1 | 1 | GUI testing, Unit testing | 1 | 1 | 1 | 1 | Not given | Not given | 1 | ||||||||||||||||||||||||||||||||||||||||
63 | 57 | The ROI of Test Automation | https://docs.google.com/file/d/0B6dKdxaNjBENSUtCNTZod0MwN2s/edit | 2007 | IAWP | I | 1 | 1 | 1 | 1 | 2 | Type of tests. E.g. performance there is no other practical way to do it. | 1 | to automate the tests I couldn’t possibly perform manually. | 1 | What are automation goals? 1) automate all things that offer immediate returns and 2) if you have a hard time convincing someone that it needs to be automated and if you feel you need an equation to see the long term advantages of automation, don’t bother automating it. | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||
64 | 58 | Too much automation or not enough? When to automate | https://docs.google.com/file/d/0B6dKdxaNjBENdHd1OFExQ2NsaDQ/edit | 2009 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | unit test (yes), build verification test (yes), Usability test (NO), | -Reasons repeating same test changes in input/software/enviroment, | Test that finds bugs, validate important scenarios, | -Oracles (nondeterministic test cannot be repeated) | -Opportunity cost between manual and automated, - Partial automation of any of the Test Step, the so called SEARCH (Setup, Execution, Analysis, Reporting, Cleanup, and Help) | 1 | ||||||||||||||||||||||||||||||||||||||||
65 | 59 | Too Much Test Automation? | http://blogs.msdn.com/b/steverowe/archive/2005/01/28/362668.aspx | 2005 | IAWP | I | 1 | 1 | 1 | 1 | 1 | automation is verfication of basic quality | Automation Cost, easiness to automate | automation scope, bugs in tests | 1 | ||||||||||||||||||||||||||||||||||||||||||||
66 | 60 | What are good guidelines to determine when to automate a test | http://sqa.stackexchange.com/questions/822/what-are-good-guidelines-to-determine-when-to-automate-a-test | 2012 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | Test that are: repetitive, simplistic, free up time for more complex tasks, are stable enough, provide a good measure of confidence in stability and quality | repetitive | tests for buggiest areas | deterministic, will yield pass/fail without false results | 1 | ||||||||||||||||||||||||||||||||||||||||||
67 | 61 | What's my Testing ROI? | https://docs.google.com/file/d/0B6dKdxaNjBENOUI4UVlBaWoyaHc/edit | 2011 | IAWP | I | 1 | 1 | 1 | 1 | ROI - High level estimation | 1 | 1 | 1 | n of releases, scope growth, cost of field defects,market value for product, working hours | 1 | manual test duration | 1 | cost of capital, , test effectivness improvement, field defects | 1 | 1 | 1 | 1 | Jobman 4.0 | Mobile | $392,500 | 3,344,336 | CMDs’ estimated test automation ROI is 1,468% over two years. | 1 | 1 | 1 | ||||||||||||||||||||||||||||
68 | 62 | What's Your Automation Index? | https://docs.google.com/file/d/0B6dKdxaNjBENd1diUEs4SC1lR1E/edit | 2007 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Does the test require many data combinations using the same test steps (i.e., multiple data inputs for the same feature)? | Does the test need to be verified on multiple software and hardware configurations? | Does the test cover most critical feature paths? Does the test cover high-risk areas? | Are the expected results constant, | Is the test impossible or prohibitively expensive to perform manually...? | 1 | Checklist of 12 questions on when to automate a test case | 1 | ||||||||||||||||||||||||||||||||||||||
69 | 63 | When and How Much to Automate | http://www.bqurious.com/when-and-how-much-to-automate/ | 2014 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Mundane UI testing that validates field types, Test complication, . | code that are used by other parts of the application, . Key functionalities ... a rogue defect could potentially result in a lost customer. | 1 | |||||||||||||||||||||||||||||||||||||||||||
70 | 64 | When automated tests make sense, and when they don't | http://ole-laursen.blogspot.com.tr/2012/03/when-automated-tests-make-sense-and.html | 2012 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | Complex code. Input checks. API code | Important corner cases, Multiple people are commiting code to a piece of code. | Human judgement needed (look and feel, video quality), | Hard to manually setup tests. | 1 | Favourig manual Localized changes, simple to set up, | 1 | ||||||||||||||||||||||||||||||||||||||||
71 | 65 | When Should a Test Be Automated? | https://docs.google.com/file/d/0B6dKdxaNjBENZUI2azBwMEd0b2s/edit | 1998 | IAWP | I | 1 | 1 | 1 | 1 | 1 | Load tests need to be automated. | Lifetime of automation | 1 | 1 | Code changes break automation. Humans (manual tests) find different defects than automation, . product-specific test libraries may help test automation be resilient to change in SUT Code | 1 | ||||||||||||||||||||||||||||||||||||||||||
72 | 66 | When To Automate | http://blogs.msdn.com/b/micahel/archive/2005/12/14/whentoautomate.aspx | 2005 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | Likelihood of breakage | Likelihood breakage | 1 | Microsoft Visio | Office software security testing, render testing | 1 | 1 | ||||||||||||||||||||||||||||||||||||||
73 | 67 | When to Automate Testing | http://davidweiss.blogspot.com.tr/2006/08/when-to-automate-testing.html | 2006 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | Script death (lifetime), | Core features, | fragile comparison of results, , easy to find oracle, failure report accuracy, | 1 | tedious to test manually, | is it possibly to test this feature automatically | 1 | 1 | |||||||||||||||||||||||||||||||||||||
74 | 68 | When to Automate Your Testing (and When Not To) | https://docs.google.com/file/d/0B6dKdxaNjBENSjN5cHpWQVdXVXc/edit | 2013 | IAWP | I | 1 | 1 | 1 | 1 | 2 | 1 | 1 | Data driven, load performance,large data, GUI intensive, requires human intervention, requires 3rd party system, Complex Functionality | static tests (not chaning) | Subjective Validation, | Skillset, | Job Security (of manual testing | , , Regression test, Smoke test,, coverage, productivity, accuracy, , undpredicatable results, New/Changing Functionality, Strategic Development, | ||||||||||||||||||||||||||||||||||||||||
75 | 69 | When to do Manual Testing and when to Automated Testing | http://blog.testing-whiz.com/2011/12/when-to-do-manual-testing-and-when-to.html | 2011 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | predetermined order of exec, simultanious exec, multiple data set for a single function, Regression or Smoke, Test report needed, Number of tests, , , random negative testing needed, | High priority | test team expertise | feasability of automation | 1 | |||||||||||||||||||||||||||||||||||||||||
76 | 70 | When to Test Manually and When to Automate | http://blogs.msdn.com/b/steverowe/archive/2008/02/26/when-to-test-manually-and-when-to-automate.aspx?Redirected=true | 2008 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | major test cases | 1 | ||||||||||||||||||||||||||||||||||||||||||||
77 | 71 | Why and When to Perform Automation Testing | http://intensetesting.wordpress.com/2013/11/07/why-and-when-to-perform-automation-testing/ | 2013 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | Load test, | Script death (lifetime) | Percentage of tests feasable for automation, , ease of automation | ||||||||||||||||||||||||||||||||||||||||||||
78 | 72 | Why How and When to Automate Testing | http://www.softwaretestingclass.com/why-how-and-when-to-automate-software-testing/ | 2014 | IAWP | I | 1 | 1 | 1 | 1 | 1 | 1 | 1 | To do automation testing you should have the skilled resources | Manual better when no time to automate | ||||||||||||||||||||||||||||||||||||||||||||
79 | YouTube | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
80 | 73 | What to Automate First | http://www.youtube.com/watch?v=eo66ouKGyVk | 2014 | YouTube | I | 1 | 1 | 1 | 1 | 1 | 1 | what test phase (maintenance, unit, system, acceptance (bad)) | easily repeatable | needed precision in comparison, is human judgement oracle required | who creates and uses test automation, | who controls changes, prior knowledge of changes (change management) | 1 | effort to perform manuall, number of conditions (combinatorial explosion) in a test | Easy / feasable to automate, easily performed, effort to analyze failures, effort to maintain | 1 | new defects / regressions, test coverage by automation, , | 1 | finance | 3200% | 1 | 1 | 1 | |||||||||||||||||||||||||||||||
81 | 74 | Intelligent Mistakes in Test Automation | https://www.youtube.com/watch?v=Dbim6ZkSHBg | 2013 | YouTube | I | 1 | 1 | 1 | 1 | 1 | 1 | automate things people do not like to do or do badly, test not possible or not practical to do manually (checking internal states of objects) | 1 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||
82 | tools | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
83 | 75 | Return on Investment Calculator for Test Automation | http://www.elbrus.com/services/test_automation_roi_calc/ | TOOL | I | 1 | 1 | 1 | 1 | 1 | 1 | releases per year | |||||||||||||||||||||||||||||||||||||||||||||||
84 | 76 | ROI Calculator by GlobalNowIT | http://globalnowit.com/calculator/ | TOOL | I | 1 | 1 | 1 | 1 | n of browsers, n of mobile platforms , test case change | 1 | effort to execute | effort to automate | 1 | number of releases, | ||||||||||||||||||||||||||||||||||||||||||||
85 | 77 | IBM Rational Quality Management ROI Calculator | http://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ROI_Rational.html | TOOL | I | 1 | 1 | 1 | 1 | , Type of App (Java, html, Sap, ...), | 1 | cost to automate | 1 | releases per year | n of test cases, , n of manu testers, n of auto testers, , days testing, hour rate, licences | ||||||||||||||||||||||||||||||||||||||||||||
86 | 78 | Return on Investment (ROI) Calculator by AutomatedTestingInstitute | http://www.automatedtestinginstitute.com/home/index.php?Itemid=65 | TOOL | I | 1 | 1 | 1 | 1 | 1 | training | 1 | Manual test effort | test development, test exec, test suite analysis time, test suite maintenance time | 1 | n of manual tests, n of automated tests, n of executions per cycle, n of cycles, hourly rate, overnight execution, failure cost | 1 | ||||||||||||||||||||||||||||||||||||||||||