��A Unified Approach To Testing/QA

MICHELLE SOLLICITO

Who Am I?

• Michelle Sollicito
• 33 years experience in Software/Consultancy
• North Highland Director of Technology
• Cloud specialist
• AWS Solution Architect Professional Certification
• AWS Security Specialty Certification
• AWS SysOps Administrator
• Azure Certified
• See https://www.credly.com/users/michelle-sollicito
• Testing/QA Team leader

Unified Approach to Testing

• Good architectural design - Before you even think about testing
• Security built in from the start
• Isolation of layers/services, decoupling
• Microservices approach
• SOLID DRY principles
• Don’t repeat yourself (anything!!)
• Single responsibility
• Use interfaces where possible (esp useful for testing!)

Unified Approach to Testing

• Good architectural design – DevOps / DevSecOps

​

Unified Approach to Testing

• Good architectural design - Before you even think about testing
• Educate developers/testers in the whole architecture (reduce problems found in integration testing)
• Train them well!
• Explain how the layers work together
• Explain how queues decouple layers
• Ensure ALL testers know how all layers work – not just the layer they are testing
• Get them to work together FIRST on end to end tests so they understand each other’s components

​

​

Unified Approach to Testing

• Good architectural design - Before you even think about testing
• Break up the monolith (reduce problems found in unit testing)
• Microservices design makes testing easier
• Break up single page architecture into multiple pages
• Modularization is key
• It’s clearer which part of the architecture needs to be tested by which tests
• One tester / SDET responsible for each microservice typically

​

​

Unified Approach to Testing

• Good architectural design - Before you even think about testing
• Scalability (reduce problems found in Stress testing)
• Ensure the system can handle maximum expected load
• Load balancers, database replicas, serverless where possible

​

• Security and Governance (reduce problems found in Security testing)
• Developers should use plugins, tools and IDEs that highlight code quality issues
• Sonalint
• Micro Focus Fortify SAST
• Visual Code (many plugins)
• Don’t forget Infrastructure Security and Physical Security / Governance

​

​

​

​

​

Unified Approach to Testing

• Code Quality (reduce overall problems with code)
• Code Reviews
• Pull Requests before checkins
• Automated code quality checking tools such as Micro Focus Fortify SAST or Sonalint

​

• CI/CD Pipelines
• Tests at each stage of the pipeline
• Find problems as early in the pipeline as possible
• Gates to prevent code moving to next environment if failed tests

​

​

​

Unit tests – some best practices

Unified Approach

• DEV
• Unit testing
• STAGE
• Integration testing
• System testing
• Stress testing
• Security testing/scans
• PROD
• End to End testing – a full user path through the system
• Smoke testing
• User Acceptance testing
• Healthchecks
• General (not in scope for this presentation)
• Controls testing for Governance and Security
• Infrastructure controls/security controls

​

​

Unified Approach

• Why not omit some types of testing?
• Frustrating
• Wastes time and effort
• Costly
• Embarrassing/costly when bugs make it to Production
• Embarrassing/costly when security breaches happen
• Potentially lose business/customers
• Maintainability/monitoring/support difficult
• Difficult to support system with lots of bugs
• Difficult to identify why the system falls over

Unit Tests

• Developers write these with their code
• Test Driven Development ideally
• Junit or Nunit or Xunit typically
• Use Dependency Injection/Interfaces
• Moqs/Mocks simulate the interfaces
• White box with no dependencies typically

​

​

if (x>5)

{

y=y+1;

}

else

{

z=x+1;

}

​

Test x <5

x = 5

x >5

​

Can x be null? Undefined?

Unit testing tools

• Front end – jest / testing library for react js for example
• Test that “snapshots” are returned as expected
• Test that button clicks call the correct callback/function
• Test the callback/function itself through all paths, positive and negative

​

• Middleware – junit, xunit or nunit for example
• For each endpoint with each/any methods it supports (Get, Put, Post, Delete and Patch)
• Internal facing – testing the code inside the Get, Put, Post or Delete etc.

​

• Database – SpyProcedure
• Mock stored procedures when testing stored procedures

​

Unit tests – some best practices

• Dependency injection is key to great unit tests
• Rewrite your code if it is not already written to take interfaces throughout
• Key functionality should be 100% covered
• No need to “test java” or “test c#” or “test javascript/react js”

- model files generated for example

- tests that test that when you click on a button, a callback is invoked - silly

• Exclude/skip anything not covered

​

Unit tests – some best practices

• Aim for 80%+ coverage
• Cover every if and every else
• Cover every parameter passed
• Cover every max and min

​

Test Coverage

Unit tests – some best practices

• Test must not be brittle – must not easily break (one thing should break it)
• Test must not depend upon any other component (use Mocks)
• Must test positive paths and negative paths of each branch, mins/maxes for loops, edge conditions, parameters
• Code review/pull request should check that unit tests cover changes made before code is allowed into repo/branch for deploys

​

Unit tests – some best practices

• Unit test naming conventions are VERY important
• Unit test annotations are VERY important
• Category or Trait so that you can group tests
• Unit tests must always run on DEV/UAT/PROD pushes to ensure nothing got broken
• Break a unit test - you must fix before committing the code/pushing
• Measure/monitor coverage carefully using Resharper, dotCover or Sonaqube etc.

​

Annotations and naming

Integration Tests

• Different kinds of integration testing
• One type is a bit like unit testing but with dependencies (white box)
• One type is testing the interface between layers (black box)
• Usually you need BOTH!

​

General Integration Tests

• Once you have unit tests it should be easy to work out which general integration tests (white box tests) you need
• Mocks become real objects
• Anywhere you used a mock you need an integration test
• Real objects should respond the same as in the unit tests ideally
• Copy your unit tests into an Integration Test directory and change the naming and annotations

​

Difference between unit tests and integration tests (white box)?

• The main difference in the tests is that:
• They can take longer – think about async tests, only put into pipeline where time is ok
• They require the whole architecture to be in place
• So use only when deploying into Stage or Prod environments
• Often they are used with “Gates” in the CICD pipeline to prevent bad code getting into Stage or Prod

​

Integration Tests

• Layer testing (Black box testing)
• Postman is great for testing the interfaces
• Middleware/microservices Get, Post, Put, Delete etc. EXTERNAL testing
• Headers, authentication, security
• Payloads in
• Responses out
• May also need to Layer/Black box test anything that interconnects directly:
• Queues, Messaging services, Pipes, etc.
• Kafka, SQS, SNS, Events etc. in AWS,
• EventHub, ServiceBus, Azure Storage Queues in Azure etc.

​

System testing

• Often initiated on the client side / front end / gui
• Typically tests that front end talks to middleware which talks to the data layer etc.
• Can be manual or automated or both
• Automated system tests should be integrated into the CICD pipeline and should be part of the criteria for the final gate to get into PROD
• Depending upon the scenario they may also be part of Staging gate
• Selenium is great for automating this
• Gherkin / Cucumber (other Smartbear tools also)
• Checks that all the layers / components are working with each other together in a system
• Should test positive and negative paths (ensure errors are reported correctly, logged etc.)

​

End to end testing

• Simulate a user’s path (use case) through the system, typically:
• Front end passes info to the middleware
• Middleware passes to the database and
• Front end retrieves and displays result/amended data back to the screen

​

• Have “positive” paths and “negative” paths
• What should the system do if the user enters invalid data?
• What should the system do if the middleware is down/internet is down?
• What should the system do if the middleware response is not as expected?

Smoke testing (Regression testing?)

• More detailed test than End to End testing (can be many End to End tests)
• Have a subset of your system testing to use as a smoke test
• Automated
• Used on deploy into a new environment typically – Prod typically
• Where you do not want to do full testing which might:
• slow down the system or
• make the system unavailable or
• create test data in a production environment
• Quick way to ensure that deploying has not broken anything major
• Find configuration problems in new environment, differences in new environment

​

Security testing

• Mainly automated
• Test for a wide range of security issues:
• Coding mistakes
• Configuration mistakes (e.g firewall port open, password in config file, https not enforced)
• Sql injection attack prevention
• Vulnerabilities
• Automated security tests/scans should be integrated into the CICD pipeline and should usually be part of the criteria for the final gate to get into PROD
• Depending upon the scenario they may also be part of Staging gate
• Fortify on Demand, Sonarqube (SAST), Webinspect are great for automating this

​

OWASP Vulnerability Scans

Healthchecks

• Healthchecks are used in apis and infrastructure to indicate health/problems
• /health 200 – Good
• /health 400 – Bad
• /health 500 - Catastrophe
• Programmer writes a healthcheck for an api to respond on /health
• Can return a payload indicating dependencies’ health also
• Many pieces of infrastructure can automatically respond to healthchecks – many load balancers automatically failover on failed healthchecks, often database replicas with promote themselves automatically in response to healthchecks etc
• Use APM monitoring tools like APM, Datadog, Dynatrace, New Relic to monitor for them and alert help desk/support staff or to automate responses such as starting up a new server or failover to a different database etc.

​

Stress testing

• Usually automated
• Selenium, HP Winrunner (Micro Focus) and HP ALM/QC (Micro Focus) and other stress testing tools
• Usually, existing system tests are performed thousands of times, depending upon load SLAs
• Monitor carefully so you notice when the system starts to get sluggish
• Monitor for bottlenecks and memory increases

​

​

Stress testing

• Running these tests helps you learn to “read the signs” that your system is not coping
• Watch the APM/monitoring tools
• Watch the healthchecks
• When does it run out of memory?
• When can the database not keep up with load?
• When does the server fall over?
• When does CPU go sky high?
• Can help determine gaps in:
• Scalability
• Performance
• Architecture design (hopefully not but..)

​

​

User Acceptance testing

• This part should not and cannot be automated
• This part depends whether the whole system is being delivered or just an increment also
• Guide your users but they should create these
• Can be based on the System tests
• Trust users - Users know what they need to do in the system
• They know what the data means
• Usually this should happen in Prod or Pre-prod environment
• Usually should happen with backups/copies of real or realistic data (dependent upon security concerns)

​

After deployment..

• Change management and review for all changes
• Each change needs to be covered by appropriate tests
• Quality system constantly improves
• Feedback into the system to make each deployment more successful, reduce bugs
• For every bug found, how can we improve to catch future bugs like it?
• Track and monitor
• Bugs found
• Time to fix bug
• Time to deploy bug to each environment

Summary – A Unified Approach to Testing

• Build quality in from the start
• If you build testing correctly and use in your CICD pipeline effectively few problems will make it to Production
• Reuse whatever effort you can
• Don’t forget any type of testing

​

QA/Testing Help?

• Michelle.sollicito@northhighland.com
• 6783573661
• Also:
• DevOps
• DevSecOps
• Cloud Migrations
• Agile/Scrum
• Change Mgt / Strategy
• Digital Core
• Data Analytics and much much more..!

​

​

WITCHES

• Women in IT Cloud Hybrid Engineering and Software
• See https://www.meetup.com/witch-women-in-it-cloud-hybrid-engineering-and-software/
• Tech Career Progression

​

What?

Invest

Train

Certs

Help

Educate/Elevate

Succeed

​