LFN-CNTI�

Testsuite improvement proposal

Martin Matyáš, Tietoevry

Konstantin Yarovoy, Tietoevry

Oliver Horecny, Tietoevry

LFN-CNTI – core assets

• Collection of best-practices
• Best practice documentation

​

• Collection of test cases
• Test case documentation, categorization, weighting, implementation
• interworking with 3^rd party tools

​

• Certification process
• Using CNTI test suite

​

• Interworking with environment
• Best practice sources
• End consumers of CNTI tests – service/infrastructure providers
• Other interactions - Anuket/Sylva etc.

​

LFN-CNTI – non-core assets

• Test framework
• Implemented using task management is crystal/sam
• Partly the framework is part of tests themselves

​

• Spec tests
• Test CNF yamls/helm charts

​

• Github actions
• Using spec tests

​

• Generic crystal libraries
• Git, tar, find, kubectl etc.

​

LFN-CNTI testsuite – Improvement motivations

• Make test case definitions better understandable for end users

​

• Lower entry barrier for new contributors

​

• Allow LFN-CNTI community focus on core assets
• Decrease needed work on non-core assets

​

• Improve test case traceability

​

• Non-core assets need improvements
• For example: Test logging, Test progress reporting, Spec test result reporting

​

​

LFN-CNTI testsuite – proposed improvement areas

(1) Test case definition readability

• Condense test case definition to one place
• Unify different levels of test case documentation and test step definition in one
• Use DSL for code that will serve as understandable documentation

​

(2) Unload some non-core assets

• Remove in-house test framework and replace it with an industry-standard test framework
• Remove in-house libraries and replace them with existing 3rd party libraries (where available)

​

(3) Introduce best practice traceability

• Allow end user to trace tests back to best practices

​

(4) Test execution logging/tracing/reporting

• Use available test framework features where possible
• Unify different logging approaches
• Unify test result reporting of regular tests and spec tests

​

(1) Test case definition readability (current state)

Test description

Best practice

Points

Categories

Test steps

(1) Test case definition readability

Use a “feature file” with Gherkin code.�For all test case definition information.

​

Use Cucumber test framework along with it.

(1) Test case definition readability

Test description

Best practice

Points

Categories

Test steps

configuration.feature

(1) Test case definition readability

Implication of Gherkin/Cucumber introduction:

​

Sorry, Crystal would need to go.

​

Would be replaced by Ruby (or alternatively Python)

(1) Test case definition readability

Example/demo of Cucumber test case�implementation

Mockup example

- Directory structure

- Feature files

- Step implementation in Ruby

​

Cucumber demo

- cucumber execution

- pretty formatter

- json formatter

​

(2) Non-core asset reduction

• Remove in-house developed task-based execution framework and replace it with Cucumber
• Testcase implementation:
• Crystal -> Ruby�(mostly straightforward transition)
• Or alternatively Crystal -> Python �(a bot more transition efforts but reaches�more developers, 49% vs. 6% - statista.com)
• Output generation:
• Rewrite to cucumber formatters in a modular way

• Some functionalities can be replaced by existing ruby and/or python 3^rd party libs

(2) Non-core asset reduction

Cucumber/ruby�(Behave/python)

Test definitions:

.feature

Test step implementations:

.rb / .py / ….

cnti_pretty_formatter

cnti_yaml_formatter

formatterN

stdout - Execution_progress

results.yml

custom_output.<ext>

logger

Logs (error/warning/info/debug)

cnti_cert_formatter

certification_protocol

anuket_report

anuket_formatter

tools

SUT

(3) Introduce best practice traceability

• Introduce tag(s) with Best Practice reference for test cases

​

​

• There are probably further improvements to be done for “Best Practices”
• Maybe reorganize tests accordingly to best practices.
• Not scope of this proposal

(4) Test execution logging/tracing/reporting

• Unify tooling and test definition styles also for current spec tests
• Use Gherkin/cucumber for them as well

​

• Unify logging approaches in regular tests and spec tests
• Use loggers and log levels

​

• Modularize outputs in Cucumber formatters
• Implement any output format from test execution by adding one file with formatter implementation

​

• Clearly separate progress and result information (typically stdout) from execution logging (typically logging targets)
• while still have options to direct any output to wherever is needed

​

​

(4) Test execution logging/tracing/reporting

Current spec to be converted to Gherkin/Cucubmer

(4) Test execution logging/tracing/reporting

Example/demo of Spec test implementation

Mockup example

- Feature files

- Step implementation in Ruby

LFN-CNTI testsuite – presented proposals wrap-up

1. Test case definition readability

All at one place, Use Gherkin

​

(2) Unload some non-core assets

Leverage Cucumber and libraries

​

(3) Introduce best practice traceability

Introduce reference to best practices

​

(4) Test execution logging/tracing/reporting

Clean-up, unify and modularize

​

​

Next steps

• Discuss/agree proposed direction in LFN-CNTI community
• If agreed, define steps leading to introduction
• Can be like:
1. Create a new branch (cucumber) (?)
2. Introduce cucumber/ruby deps
3. Convert current Spec tests to Gherkin/Ruby (or Python)
4. Implement needed cucumber formatters
5. Convert part of tests and supporting code to Gherkin/Ruby/Python
6. Convert the rest

LFN-CNTI improvement proposals - questions

Thanks for attention!