| A | B | |
|---|---|---|
1 | Question | Reply |
2 | About reuse code and tests, in addition to Cypress commands, which other strategy is used? | If preferred you could always create additional helper functions in a separate file but you'd probably want to define what would be added as a helper vs a Cypress command. Another approach when reusing code in tests could be to create functions for the test(s), this means you can load in different preconditions/dependencies in a before/beforeEach and then just call the function to execute the same test steps across different configurations. |
3 | Pure UI flows, programmatic flows or combined? What is your prefered ratio? Thx | Personally I'd probably shift towards UI flows 60/40, I think it's important to think more about the front end use of what is being delivered. |
4 | Is your cypress repo integrated with your application's CI pipeline? Or is it running as an individual separate repo? | We have Cypress as a dependency of our repositories and CI just installs and runs it for us. |
5 | How did price impact the decision of moving to Cypress? Pricing has been a bit of an obstacle for us. | I haven't worked closely with pricing but it didn't impact our decision of moving to Cypress. However licensing costs have lead us to choose to initially roll out Cypress Dashboard iteratively to our teams. |
6 | What would be a good acceptable duration of test execution for sanity and a regression suite? Some ballpark numbers in hours or minutes | Personally I would go with <=10mins for Sanity as it runs on every pull request, we don't want it holding up the pipeline for too long when working on new features. For Regression I'd be happy with anything from 20 mins to 2 hours, this is because I would expect it to be a fully loaded workflow with environment dependencies that could be time consuming. Anything longer than an hour or two would make me nervous of the impact of time lost if an error occurred during the run. |
7 | How do you define the scope for each test block? i.e. having 10 test blocks vs doing all tests in a page in a single test block | We would define the spec file by the page or feature of our application, within the spec file our tests will then be broken down into describe blocks where required, the it blocks within those describe blocks would usually contain a single test or short flow with multiple assertions so it's easy to debug if it fails. I would avoid writing the tests too small though (as recommended in the best practices - https://docs.cypress.io/guides/references/best-practices#Creating-tiny-tests-with-a-single-assertion ) |
8 | It looks like pendo has integrated santiy and regression test suites in dev CI pipeline. Do you feel QA becomes the bottleneck when you do that? | I don't believe QA becomes a bottleneck (or at least it doesn't for us). The Sanity suite is small and fast enough to not impact frequent code pushes and the Regression suite runs out of hours as to not cause any impact. Generally we advocate shifting test left to avoid potential bottlenecks in our manual and automated testing, we often try to think about testing from the point of first designing a new feature. |
9 | How do you get developers interested in testing? | Interesting question and I'm not sure if there's a right answer (or if I have it at least) as this really boils down to individuals and their interests. I feel like the key to getting people interested is to engage them, show them the benefits of testing, make it as frictionless as possible. That's just my take though, hope it helps. |
10 | Do you use vanilla cypress API ? Do you use cucumber? | In my personal time I have just worked with standard API testing using Cypress (great blog post here - https://www.mariedrake.com/post/api-testing-with-cypress ). I haven't used Cucumber with it. |
11 | When using a Pendo CSS element in Cypress are those dynamic? For example, on ‘X’ button I see #pendo-close-guide-1738599e but is that static or will it change? | Interesting spot! That's correct, in the Guide preview (at least) the elements are dynamic so for our first iteration we identify these by the first part of the value that doesn't change. |
12 | For your full coverage through the regression suite, do you do that on multiple platforms and devices? Is this all possible combinations of the design studio? | For our first iteration we only targeted Chrome desktop, we do have some Sanity tests for our mobile Visual Design Studio though. Ideally we'd like to expand this coverage in the future and I'm confident we'd be able to do so with the Cypress viewport feature and extended browser support. |
13 | Do you start cypress test from scratch , or you migrate exist test cases? May I ask how many Cypress test cases you have, and how long it took? | We started ours from scratch and ended up with ~100 Sanity tests and ~200 Regression tests. The Regression tests are a lot larger than the Sanity ones due to the fact they can run independently. Writing the tests took us a good few months alongside our usual feature commitments. |
14 | Do you run cypress tests on cloud-based execution tools like BrowserStack or sauce labs? | Not currently but it could be considered if we need more than Cypress' current viewport feature and browser support |
15 | How has the transition to everyone owns quality gone? | Really well, everyone actively thinks about testing as well as writing lots of tests. |
16 | Are your QA's running auto/manual testing? Or do you have separate employees for that? | We as Quality Engineers run manual tests as well as contribute to the creation and maintenance of (mainly frontend) automated tests |
17 | Can you talk a little bit about coverage reports? Coverage being inherently incompatible with functional/e2e tests, how do you report the breadth of your suite? | Coverage for us comes from Cypress Dashboard as well as our test case management software. Cypress Dashboard allows us to breakdown our test suites by tags so we can monitor the coverage and stability of each one. We can monitor the number of tests in each suite to ensure our coverage is increasing with new features. Reporting from our test case management system allows us to see feature coverage breakdown for both manual and automated tests together (as our Cypress tests results are integrated). |
18 | How to handle tests requiring a large set of initial data to be added to a complex database schema? | We generally load the creation or mocking of required data into before or beforeEach calls so they are carried out before the test. If you have requirements on a larger or more complex scale it may be worth launching these at an earlier or different stage utilising the plugins file - https://docs.cypress.io/guides/tooling/plugins-guide |
19 | How do you take care about cross browser test? If all your tests are in Cypress, how do you safely skip Safari from business perspective? | Cypress supports multiple browsers for our test runs but we assess required coverage for browsers not supported by usage analytics. Knowing how many users use each browser allow us to scope the required manual testing. |
20 | What is up with the "describe" inside the "describe"? | Good spot! So this was an example taken from a larger spec file but we sometimes wrap describe blocks within each other to break it blocks into smaller groups. This also makes the tests easy to navigate and read in the headed test runner results. You could also do this with "context" which arguably could be clearer than using "describe" - https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests#Writing-tests |
21 | When using cy.intercept, have you found a way to intercept an api call to the same endpoint but with different body params? | I haven't had to do this personally but hopefully this answer is in the docs, glancing through it feels like a couple of the examples/sections could refer to your use case - https://docs.cypress.io/api/commands/intercept#Intercepting-a-request |
22 | why did you decide to make sanity environment-dependent? isn't it better to have them independent as well | I agree it'd be better if they were independent which is why we ended up deciding this for the Regression suite. Initially we decided to not do this for Sanity because we didn't want to build guides every time we run these high level tests, we wanted to balance the importance of coverage vs speed as these tests run on every pull request. I think we know we could always iterate something like this by making Sanity tests independent and bumping up our parallelisation numbers to speed up the test runs if it becomes a problem for us. |
23 | Do you have to deal with application state in your tests (testing against various settings)? If so, how do you accomplish this. | We do and we handle these in a few ways. If updating the settings should be part of the tests we carry them out as steps with the relevant assertions in the test. If updating the settings is a prerequisite and we are not really testing them we will load these in before/beforeEach calls, either as real data or as a fixture - https://docs.cypress.io/api/commands/fixture |
24 | How do you select which user scenarios to automate? | We generally judge these in our refinement sessions when the scenarios are first presented to the team. We will choose to automate the main scenarios of using the feature which includes the happy paths and any paths leading to implemented error handling etc. |
25 | How do you generate data for your tests ? Via API or Db or other ? | We are working with an API when writing our tests so it's either created as it would be by a user or added as a fixture - https://docs.cypress.io/api/commands/fixture |
26 | Does your dev team use Cypress for unit testing as well? | We currently do not use Cypress for unit testing |
27 | Did you encounter any flakiness with object timeout and page timeout? If so how yo dealt with that? | We did have timeouts causing flakiness but we handled them using Cypress test retries - https://docs.cypress.io/guides/guides/test-retries |
28 | Do you use BDD? | Not with Cypress although when writing tests we follow Chai's BDD assertions - https://docs.cypress.io/guides/references/assertions#BDD-Assertions |
29 | Can non developers/engineers create cypress tests? | With a helping hand from another engineer or the right amount of googling I would say so :) - A good place to get started is with the Real World app - https://www.cypress.io/blog/2020/06/11/introducing-the-cypress-real-world-app/ |
30 | How much time take your sanity and regression test to run? | Using parallelisation - ~5 mins for Sanity, ~18 mins for Regression |
31 | What are the best practices of Cypress you are using? | The majority of them but the key ones are linked as part of my Summary slides (selecting elements, unnecessary waiting etc) - https://cypress.slides.com/cypress-io/cypress-and-pendo#/8 |
32 | How do you handle pushback from other engineers about modifying code for testing purposes, such as adding data-cy attributes to elements. | Personally I've not had any pushback, I don't feel like the changes are particularly invasive or complicated so I would probably be trying to understand the reason for the pushback as a starting point and work together on solving any impediments from there. |
33 | How do you handle super flaky slow API responses | In case this isn't Umar asking this question (or if someone else is reading) I answered this one on Twitter - https://twitter.com/georgepalf/status/1380014713539559424 |
34 | Do you leverage runCount for anything? | We do not that I am aware of. |
35 | Did u use any plugins with in cypress ? | We currently use Cypress plugins but only for reporting - https://docs.cypress.io/plugins/directory#Reporting |