| A | B | C | D | E | F | G | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 1-low/2-medium/3-high | 1(bad) - 5(good) | 1(bad) - 5(good) | 1(bad) - 5(good) | ||||||||||||||||||||||
2 | Category | Factor | Question | Importance | TestComplete | Playwright | Cypress | Comments | ||||||||||||||||||
3 | Authoring speed | Authoring speed, dev experience | How fast is the process of creating test cases? | 3 | 5 | 3 | 4 | Playwright has trace viewer and inspector, good but not as convenient as all-in-one dev app from Cypress. | ||||||||||||||||||
4 | Availability | open source or license-based library? | Is it open-source? | 2 | 3 | 5 | 5 | Playwright, Cypress: open source; Test complete | ||||||||||||||||||
5 | Collaboration | Test suite repo – team collaboration | Does it support team collaboration? | 3 | 5 | 5 | 5 | |||||||||||||||||||
6 | Design | Reusable Functions | Does it reusable functions that can be used in multiple tests? | 3 | 5 | 5 | 5 | |||||||||||||||||||
7 | Design | Data Parameterization | Does it data parametrization? | 3 | 5 | 5 | 5 | |||||||||||||||||||
8 | Features | Navigate and verify our courses (iframes) | Does it support iFrames? | 3 | 5 | 5 | 3 | |||||||||||||||||||
9 | Features | Multi Tabs | Does it support multiple tabs? | 3 | 5 | 5 | 3 | |||||||||||||||||||
10 | Features | Multi Domains | Does it support multiple domains? | 3 | 5 | 3 | ||||||||||||||||||||
11 | Features | Database Connection | Does it support connecting testes to database, executing queries? | 2 | 5 | 5 | 5 | Cypress: https://testersdock.com/cypress-database-testing/; Playwright, Cypress: can use any Node.js/NPM library | ||||||||||||||||||
12 | Features | Ability to API mock to test edge cases | Does it support mocks? | 2 | 5 | 5 | 5 | |||||||||||||||||||
13 | Features | Reporting | Does it support test results reports? Can be exported to HTML? | 3 | 5 | 5 | 5 | Reporting is typically used on the test runner level | ||||||||||||||||||
14 | Features | File uploads/downloads | Does it provide access to windows displayed by the browser itself, not by web pages? (E.g. File Download/Upload windows) | 2 | 5 | 5 | 5 | |||||||||||||||||||
15 | Features | Headless mode | Does it support headless mode? | 3 | 5 | 5 | 5 | |||||||||||||||||||
16 | Features | Parallel test run | Can the tests run in parallel? | 3 | 4 | 5 | 4 | Cypress restricts running tests parallelly on a local machine. Cypress provides a Parallel Testing option by setting up multiple machines. Playwright: By default, test files are run in parallel. In order to achieve that, it runs several worker processes that run at the same time. Tests in a single file are run in order, in TestComplete, you can create tests that will run in test environments controlled by an external framework rather than on your local workstation. TestComplete has support of parallel tests but it requires setting up remote environments and some features are not supported | ||||||||||||||||||
17 | Features | Assertions | Does it have extensive set of assertions? | 3 | 5 | 5 | 4 | |||||||||||||||||||
18 | Features | Data driven tests | How easy it is to run data driven tests? | 2 | 5 | 4 | 5 | |||||||||||||||||||
19 | Features | Debugging and and logs | How easy it is to debug? Do we get verbose logs saying what tool was trying to do before any test failure? | 3 | 5 | 4 | 5 | Playwright has trace viewer and inspector, good but not as convenient as all-in-one dev app from Cypress | ||||||||||||||||||
20 | Features | Auto-waiting | Does it support auto-waiting? | 3 | 5 | 5 | 5 | |||||||||||||||||||
21 | Features | Execution speed | How fast it executes tests compared to other tools | 2 | 3 | 5 | 4 | |||||||||||||||||||
22 | Features | Locators | Does it provide multiple ways to use locators? Support XPATH, locate by text? | 3 | 5 | 5 | 5 | |||||||||||||||||||
23 | Features | Node.js and JavaScript support | Does it support Node.js and JavaScript support? | 2 | 5 | 5 | 5 | |||||||||||||||||||
24 | Features | Visual regression | Does it support visual regression testing? | 2 | 5 | 4 | Cypress: has external open-source plugins. Playwright: built-in feature. | |||||||||||||||||||
25 | Features | Plugins and extensions | Does it have rich plugin ecosystem to support and extend the built-in capabilities of the tool? | 5 | 5 | Cypress has a lot of plugins. This in some cases is due to limitations of built-in features. Playwright does not have so many plugins but most of the important features are supported natively. | ||||||||||||||||||||
26 | Future-proof | Future-proof | Who “owns” the tool? Will it be supported for a long time? | 2 | 5 | 5 | 5 | Cypress: Cypress.io, Inc., founded in 2015; Playwright: Microsoft. TestComplete: Smart Bear | ||||||||||||||||||
27 | Integration | Connection with devops CI/CD | Is it possible to integrate with CI/CD tools (Azure DevOps) | 3 | 5 | 5 | 5 | |||||||||||||||||||
28 | Platform | Cross browser | Does it support 3 major browsers: Chrome, Edge, Firefox? | 3 | 5 | 5 | 5 | |||||||||||||||||||
29 | Platform | Mobile app testing | Does it support mobile app testing? | 2 | 5 | 4 | 4 | |||||||||||||||||||
30 | Pricing | Pricing | How expensive it is to use this tool for team of 3 testers | 3 | 1 | 5 | 4 | Playwright: free; Cypress: free, paid plans available; TestComplete: $3,260 Base plan | ||||||||||||||||||
31 | Stability | Stability | Is it stable? | 2 | 5 | 5 | 5 | |||||||||||||||||||
32 | Support | Docs and resources | Detailed documentation, tutorials and cources? | 3 | 5 | 5 | 5 | |||||||||||||||||||
33 | Support | Regular updates | Does the tool release on a schedule that coincides with third party dependencies such as WebKit, Chrome upgrades, WebDrivers etc | 2 | 5 | 5 | 5 | Cypress, Playwright: miltiple releases each month; TestComplete: regular updates | ||||||||||||||||||
34 | Support | Community | Has community support? Slack, dedicated forum, anything else? | 3 | 4 | 5 | 5 | TestComplete: negative feedback about their support; Playwright has Slack channel, good feedback; Cypress has Gitter community ~10K members | ||||||||||||||||||
35 | Adoption | Adoption | Was it adopted by multiple teams. How popular this tool? | 2 | 5 | 5 | Cypress: 40K+ GitHub stars / 2.7K issues. Playwright: 41K+ GitHubb stars/0.6K issues (2022-08-25) | |||||||||||||||||||
36 | User satisfaction | User satisfaction | What is the feedback from teams who used this tool? Good or bad? | 2 | 5 | 5 | Cypress: 40K+ GitHub stars / 2.7K issues. Playwright: 41K+ GitHubb stars/0.6K issues (2022-08-25) | |||||||||||||||||||
37 | ||||||||||||||||||||||||||
38 | Importance | |||||||||||||||||||||||||
39 | 1 | importance 3-high | 19 | |||||||||||||||||||||||
40 | 2 | importance 2-medium | 14 | |||||||||||||||||||||||
41 | 3 | importance 1-low | 0 | |||||||||||||||||||||||
42 | ||||||||||||||||||||||||||
43 | Low code, No code - Questions to ask | |||||||||||||||||||||||||
44 | 1 | Can you re-use code? | ||||||||||||||||||||||||
45 | 2 | How do you pass paramaters? | ||||||||||||||||||||||||
46 | 3 | Is "Round Tripping" supported? | ||||||||||||||||||||||||
47 | 4 | Can you step-through and debug? | ||||||||||||||||||||||||
48 | 5 | Version control | ||||||||||||||||||||||||
49 | 6 | Maintainability | ||||||||||||||||||||||||
50 | ||||||||||||||||||||||||||
51 | ||||||||||||||||||||||||||
52 | ||||||||||||||||||||||||||
53 | ||||||||||||||||||||||||||
54 | ||||||||||||||||||||||||||
55 | ||||||||||||||||||||||||||
56 | ||||||||||||||||||||||||||
57 | ||||||||||||||||||||||||||
58 | ||||||||||||||||||||||||||
59 | ||||||||||||||||||||||||||
60 | ||||||||||||||||||||||||||
61 | ||||||||||||||||||||||||||
62 | ||||||||||||||||||||||||||
63 | ||||||||||||||||||||||||||
64 | ||||||||||||||||||||||||||
65 | ||||||||||||||||||||||||||
66 | ||||||||||||||||||||||||||
67 | ||||||||||||||||||||||||||
68 | ||||||||||||||||||||||||||
69 | ||||||||||||||||||||||||||
70 | ||||||||||||||||||||||||||
71 | ||||||||||||||||||||||||||
72 | ||||||||||||||||||||||||||
73 | ||||||||||||||||||||||||||
74 | ||||||||||||||||||||||||||
75 | ||||||||||||||||||||||||||
76 | ||||||||||||||||||||||||||
77 | ||||||||||||||||||||||||||
78 | ||||||||||||||||||||||||||
79 | ||||||||||||||||||||||||||
80 | ||||||||||||||||||||||||||
81 | ||||||||||||||||||||||||||
82 | ||||||||||||||||||||||||||
83 | ||||||||||||||||||||||||||
84 | ||||||||||||||||||||||||||
85 | ||||||||||||||||||||||||||
86 | ||||||||||||||||||||||||||
87 | ||||||||||||||||||||||||||
88 | ||||||||||||||||||||||||||
89 | ||||||||||||||||||||||||||
90 | ||||||||||||||||||||||||||
91 | ||||||||||||||||||||||||||
92 | ||||||||||||||||||||||||||
93 | ||||||||||||||||||||||||||
94 | ||||||||||||||||||||||||||
95 | ||||||||||||||||||||||||||
96 | ||||||||||||||||||||||||||
97 | ||||||||||||||||||||||||||
98 | ||||||||||||||||||||||||||
99 | ||||||||||||||||||||||||||
100 |