1 of 18

January 20, 2021

​

�FOLIO QA Initiatives 2021

| www.folio.org

1

2 of 18

Agenda

  • Review 2020 progress
  • QA Audit goals and findings
  • Current state of affairs
  • Proposed improvements

| www.folio.org

2

3 of 18

Proposed Measures for FOLIO Quality Improvements in 2020

Done

Prototype CI/CD pipeline with additional build(test) steps

Done

Rollout Scratch Environments for all dev teams

Done

Achieve 80% unit test coverage

Done

Finish FOLIO performance test system

Partial

Finish end-2-end functional test

Cancel

Move current API performance tests into respective module pipelines

Cancel

Finish updating CI/CD pipelines for Platform Complete modules

Cancel

Implement contract tests for modules calling Core modules APIs.

Not started

Introduce industrial-grade infrastructure layer for Logging, Journaling, Monitoring

Not started

DevOps infrastructure for automated schema migrations testing

Not started

Identify a tool and create prototype for security testing

| www.folio.org

3

4 of 18

Assessment goals

  • Review current testing process and toolset
  • Measure current systems quality and process maturity
  • Identify main testing gaps and process bottlenecks
  • Suggest opportunities for improvement

| www.folio.org

4

5 of 18

QA Audit - Key Findings

TESTING

  • Unit tests coverage is good for both BE and FE
  • Defect Containment Effectiveness (DCE) is 91% (below industry standard 95%)
  • Number of defects is growing from release to release
  • Regression scope and time for regression grows from release to release

DOCUMENTATION

  • Very high level of documentation in general
  • Test Cases and Bugs have templates and are well written
  • Project is tracking all kinds of metrics and reporting
  • Test cases are written when PO has free time or when community finds inconsistencies

PROCESS

  • Project understands what improvements needs to be made
  • Impressive ​Community contributions and involvement in product testing during BugFests
  • Regression testing is late in the release cycle
  • Testing items have the lowest priority compared to other items

| www.folio.org

5

6 of 18

QA Audit → Improvement Direction is Right. However...

  • BE Unit Tests – coverage for API (est. 95%)
  • FE Unit Tests – coverage for UI (est. 78%)
  • Integration tests (Karate) – process started, tool selection is good, but est. coverage 25% (after 6 months since initiative started)
  • CI pipeline – all quality gates look good
  • Test cases are written and updated every release, albeit only once
  • Lots of metrics are gathered indicating clear issues with quality
  • Tool for E2E UI testing is selected, but writing Smoke test suite (109 TC) just started.

| www.folio.org

6

7 of 18

Last Day of Bug Fest Q3

| www.folio.org

7

8 of 18

Defect Containment Efficiency

​

Q2

Q3

Support

16

17

HotFix Releases

69

4

After Bug Fest Start

77

144

Before Bug Fest *

86

69

Total

248

234

​

​

​

​

​

​

DCE After GA

93.55%

92.74%

DCE before BF

34.68%

29.49%

Target 95%

​

​

* Not all bugs are filed before BugFest,

but BugFest defect count is still high

| www.folio.org

8

9 of 18

Release Capacity Allocated for Bugs

| www.folio.org

9

10 of 18

Current Process

SCRATCH

ENVIRONMENT

REFERENCE

ENVIRONMENT

Cover both BE & FE with Unit tests (DoD)

Create test cases in TestRail

Stories Acceptance

Developers

Automated

Product Owners

Community

RESPONSIBILITIES

PULL

REQUEST

Karate/Postman Integration Tests (DoD)

BUG FEST

ENVIRONMENT

Manual Testing

Bug Fixing

​

BUG FEST

START

​

GA

RELEASE

Reactive Problem Solving

PTF

| www.folio.org

10

11 of 18

Recommendation: Shift Testing to Scratch Environment

SCRATCH

ENVIRONMENT

REFERENCE

ENVIRONMENT

Cover both BE & FE with Unit tests (DoD)

UI Smoke Test

Create test cases in TestRail (DoD)

Stories Acceptance

Cover Test Debt TCs with Automation

Automated UI Smoke Test

Run Karate Regression Suite

Developers

Automated

Product Owners

SIG & Community

RESPONSIBILITIES

Manual Testing

PULL

REQUEST

Run Postman Regression Suite

UAT

← Manage

← Manage

Karate/Postman Integration Tests (DoD)

BUG FEST

ENVIRONMENT

Manual Testing

Automated UI Smoke Test

Bug Fixing

​

BUG FEST

START

​

GA

RELEASE

Performance Acceptance

Performance Regression Test

PTF

| www.folio.org

11

12 of 18

Proposed Next Steps

Priority

Action Item

R1

R2

R3

1

Move testing activities to teams scratch environments

50%

100%

​

2

Build API Integration Test Coverage (Karate framework)

35%

50%

75%

3

Implement UI Integration Test (New BigTest) - 109 test cases

40%

100%

​

4

Implement manual sprint regression using testing community

20%

50%

100%

5

Implement UAT events using SIG resources

20%

50%

100%

6

PO to complete specs for performance acceptance test

100%

​

​

7

Implement performance acceptance test

​

50%

100%

8

Extend BugFest time frame (Approved)

​

​

​

| www.folio.org

12

13 of 18

Appendix

| www.folio.org

13

14 of 18

No regression testing or smoke testing is executed during sprint development.

​

Start manual smoke testing at the end of every sprint:

Option 1: Use developers. Only a few test cases per developer.

Option 2: Use SIG members or Community. More work for PO to prepare a Test Run.

  • Create manual tests cases during story development
  • Update old test cases when testing the story
  • Story testing should include negative test cases
  • Deploy completed sprint work to Scratch environment for Smoke test

​

​

Manual Testing

| www.folio.org

14

15 of 18

PC requested SIGs to have early access to completed but not released features

​

  • Invite SIG members to UAT as soon as feature has been complete.
  • Deploy completed feature to scratch environment
  • Transfer related test cases from TestRail to Google Forms
  • Collect and process feedback:
    • File bugs
    • Create additional functional stories based on the feedback

​

​

UAT

| www.folio.org

15

16 of 18

Karate/Postman Integration Tests

  • 90% of the defects in the backend modules could have been found with Integration or Unit Tests
  • Progress in implementing of integration testing is very slow, though the plan is in place

https://wiki.folio.org/display/FOLIJET/API+Integration+Testing

  • Coverage % = “Implemented TC” / “Total TC in Test Plan” * 100

​

Current goals for integration testing

  • Create test plans for all BE modules
  • Cover all new and uncovered endpoints with Karate Integration tests
  • Run Integration testing suite:
  • Migrate all existing Postman tests to Karate (long term)

​

Actions required

  • Reprioritize integration testing stories (currently P = Low)
  • Respect 20% tech debt rule and prioritize accordingly

​

​

Run Karate Regression Suite

Run Postman Regression Suite

| www.folio.org

16

17 of 18

Performance Acceptance

Performance Regression Test

PTF created foundation for Performance Acceptance tests:

​

Recommended actions:

  • PO to finish Performance Acceptance tests requirements ASAP:
  • Teams to work with PTF to build these tests
  • Make PTF approval a part of required release verification and sign off

| www.folio.org

17

18 of 18

Manual Regression is done once per release by community members.

Both regression time and scope is growing from release to release.

One regression test cycle is not enough

​

GOAL: Decrease time of the release testing, bug fix and retesting by introducing automation testing earlier in the cycle.

​

SOLUTION:

  • Start writing automation E2E tests from Smoke suite (currently 109 TCs)
  • Continue with full Regression suite (currently 1107 TCs).

​

No action required from PO at this time

Zak Burke; Frontside Software + 1 EPAM Dev will be working on Smoke regression test.

​

​

​

UI Smoke Test

| www.folio.org

18