2021 Community Lab Retrospective

x

1

UNH-IOL Community Lab 2021 Retrospective

• SOW finalized the beginning of March, 2021
• This was the date when all assigned stories were finalized / handed off
• 25 of 31 stories completed (81% complete)
• Across 9 Epics
• Official list of 2021 objectives are available here

2

UNH-IOL Community Lab 2021 Epics

• Percentage Performance Numbers
• 4 completed stories
• All work completed
• Coverity Alternatives
• 2 completed story
• 1 open stories (pending DPDK patch)
• Added Q3 story - Sonar Cloud (nearly complete)
• Expanding test coverage
• 8 completed stories
• 1 open stories
• 3+ open bugs
• Both depend on DPDK action items: BUG-761 & BUG-756
• ASAN: https://bugs.dpdk.org/buglist.cgi?quicksearch=[asan]
• FIPS Testing
• 1 completed story
• 1 open story
• 1 new story added
• Expected by end of January 2022

• Expand Periodic Testing
• 3 completed story
• 1 open stories
• Community Access
• 6 completed stories
• Deploy RTE Flow Testing
• 1 open story
• Deploy Compression Performance Testing
• 1 open story, not started, push to 2022
• Deploy DTS virtio test suite
• 1 open story
• Expected by end of January 2022

3

UNH-IOL Community Lab Test Coverage (1)

Beginning of 2021

End of 2021

4

UNH-IOL Community Lab Test Coverage (2)

End of 2021:

• Compile (24) / Unit (15) / ABI (15) tests over multiple operating systems
• Combination of gcc and clang
• Includes both x86_64 and aarch64 for some operating systems
• Cross-compile tests
• x86_64 to aarch32 & aarch64
• aarch64 SVE emulated unit tests
• Bare-metal performance and function testing
• Performance: 6 NICs from 3 vendors
• Functional: 10
• Includes both x86_64 and aarch64

Start of 2021:

• Compile (9) / Unit (1) / ABI (2) tests over multiple operating systems
• Bare-metal performance and function testing
• Performance: nic_single_core_perf (6 NICs)
• Functional: 7

5

Coverity Alternatives

• Spell checker was created, deployed, community requested updates
• Updated tooling proposed as patch to dpdk/devtools, still pending merge https://patchwork.dpdk.org/project/dpdk/patch/20211201165954.16153-1-hnadeau@iol.unh.edu/
• Deployment into CI is pending above code merge
• Sonar Cloud was requested
• Test harness has been created and should be deployed first week of January (avoiding rollouts before holidays)
• DPDK “org” has been created, similar to Coverity setup
• CPPcheck, Clang Static Analyzer / scan-build, Flaw finder
• Rollout scheduled for January, with latest Jenkins update (December) providing required plugins.

6

Expanding test coverage

• Compile / Unit / ABI testing expanded to X operating system
• Added better support for aarch64 and cross-compile jobs
• Added emulated support for aarch64 SVE
• Note, this was added as additional story / work outside of the SOW
• Open Stories
• ASAN is ready in UNH-IOL CI, but are pending dpdk “readiness”: https://bugs.dpdk.org/buglist.cgi?quicksearch=[asan]
• UBSAN is ready, but dpdk readiness not yet started.
• FreeBSD 13.0 unit testing is pending bug fixes in DPDK
• Alpine Linux unit testing is pending bug fixes in DPDK
• Code coverage reports (gcov / lcov) were not completed and should be pushed to 2022.

7

FIPS Testing

• Communicated with NIST to confirm and setup test harness
• Harness can automatically pull required test vectors from NIST API server and submit result for pass/fail grading.
• This API replaces the original NIST “sold” vectors, which are no longer available
• DPDK FIPS Sample Test
• Does not support the current NIST vector format (maintainers were reusing originally acquired vectors).
• UNH is working to update the app to support the new NIST vector format.
• Expecting to complete the app by end of January and role out automated testing into CI.

8

Expand Periodic Testing

• CI jobs created to automatically test DPDK LTS branches and appropriate next- repos.
• Maintainers are automatically notified of jobs / results.
• Jobs check the repos/branches and only test if the “tip” has moved to a new commit.
• Dashboard upgraded to better show “periodic” jobs
• Open Stories
• gcov / lcov not yet available to track test coverage over time.

9

Community Access

• Updated all DTS deployments to a uniform version across bare-metal systems.
• Patch categorization to avoid testing on documentation changes, etc.
• Patch apply issues report as warnings back to patchworks.
• Jenkins pipelines are pulled from a git repo.
• Need to create public facing documentation for the repo.
• Job definitions need to be moved to the repo as well (2022).
• Monitoring system updated to notify the community of issues in the lab.

10

Deploy RTE Flow Testing

• Updates / fixes to the DTS test case were completed in Q3 / Q4 of 2021.
• Need to deploy the updated version of DTS to the bare-metal hardware to roll out these changes.
• This was deferred until after the 21.11 LTS release and the Community Lab maintenance updates.
• Planning to role this out towards the end of January 2022.

11

Deploy DTS virtio test suite

• Work has started to develop the VM required for testing
• DTS includes no documentation for the VM setup or requirements
• UNH-IOl is creating this, along with scripting to automate the VM setup, as a patch back to DTS
• Expecting to complete this work during January 2022

12

Deploy Compression Performance Testing

• Did not get to start this work during 2021
• Push this as a story to 2022

13

DPDK Community Issues

• Vendors with bare-metal systems need to better support the CI lab / infrastructure.
• Performance tuning and maintenance is the responsibility of the vendor, not UNH-IOL.
• UNH-IOL can ensure DTS performance and functional tests run on the vendor platform, but performance stability (and requirements) is the responsibility of the PMD vendor.
• Note, bare-metal performance systems are purpose setup, for single vendor / architecture, and are not running “other” jobs, so the CPU is “dedicated” to DPDK DTS testing.
• Some stories are dependent on DPDK community
• Multiple open bugs preventing story completion (spell check, FreeBSD, Alpine Linux, etc.)
• Need a path forward to enable testing that will include failures, such as ASAN, UBSAN, RTE Flow, etc.

14

2022 Planning

Initial planning and ideas are being captured here: https://docs.google.com/document/d/1l38GZwaMuIu8hq3kMjoAHDU_LanXXTt-LYS4y7Sv4t8/edit

​

Requests to the community is to review, provide ideas, feedback, and priorities

15