1 of 52

Solidity Summit 2023 | Test Your Tests

2 of 52

Test Your Tests

The Dos and Don’ts of testing

@trailofbits

phaze

@lovethewired

Solidity Summit 2023 | Test Your Tests

3 of 52

Overview

  • Motivation for topic
  • Examples: Testing shortcomings
  • Exploring various testing strategies
  • Offensive Testing: case study
  • Takeaways

Solidity Summit 2023 | Test Your Tests

4 of 52

The Role of Testing

  • Fault identification
  • Invariant validation
  • Spec adherence
  • Build up confidence and trust in performance
  • Guarding code functionality: Regression tests

​

But: No testing method is foolproof.

Testing is an ongoing process of refinement, not a final endpoint!

Solidity Summit 2023 | Test Your Tests

5 of 52

Motivation for Topic

  • Reflections on past shortcomings
  • Improving development and testing process
  • Driver for becoming security-oriented

Solidity Summit 2023 | Test Your Tests

6 of 52

Improving ERC721A

Solidity Summit 2023 | Test Your Tests

7 of 52

ERC721A

Solidity Summit 2023 | Test Your Tests

8 of 52

ERC721A Optimized

  • Save gas on all future transfers
  • Store boolean nextTokenDataSet in ownership slot
  • Only touch subsequent token data if !nextTokenDataSet

Solidity Summit 2023 | Test Your Tests

9 of 52

Development & Testing Approach

  • Test-driven
  • “Sufficient testing will uncover flaws”
  • Quality through quantity

​

But: Lost sight of the bigger picture

Solidity Summit 2023 | Test Your Tests

10 of 52

Unit-Testing Functionality

Solidity Summit 2023 | Test Your Tests

11 of 52

Unit-Testing Functionality

Solidity Summit 2023 | Test Your Tests

12 of 52

Unit-Testing Functionality

How??!

Solidity Summit 2023 | Test Your Tests

13 of 52

Apply Patch and Test

Fixed!!

All tests pass…

Solidity Summit 2023 | Test Your Tests

14 of 52

Apply Patch and Test

Fixed!!

All tests pass…

Fixed!! (Not quite)

All tests pass…

Solidity Summit 2023 | Test Your Tests

15 of 52

What Went Wrong?

  • Lacking systematic testing approach and structure

Solidity Summit 2023 | Test Your Tests

16 of 52

What Went Wrong?

  • Lacking systematic testing approach and structure
  • Missing important edge-cases

Solidity Summit 2023 | Test Your Tests

17 of 52

What Went Wrong?

  • Lacking systematic testing approach and structure
  • Missing important edge-cases
  • Testing multiple things at once

Solidity Summit 2023 | Test Your Tests

18 of 52

What Went Wrong?

  • Lacking systematic testing approach and structure
  • Missing important edge-cases
  • Testing multiple things at once
  • Lacking expressive and meaningful fuzz tests
    • Multiple transfers
    • Random ids
    • Arbitrary actors

Solidity Summit 2023 | Test Your Tests

19 of 52

What Went Wrong?

  • Lacking systematic testing approach and structure
  • Missing important edge-cases
  • Testing multiple things at once
  • Lacking expressive and meaningful fuzz tests
    • Multiple transfers
    • Random ids
    • Arbitrary actors

100% code line & branch coverage != 100% state coverage

Solidity Summit 2023 | Test Your Tests

20 of 52

Good Testing is Hard…

Solidity Summit 2023 | Test Your Tests

21 of 52

Shortcomings Exemplified:

Testing WAD Conversions

Solidity Summit 2023 | Test Your Tests

22 of 52

Testing WAD Conversions

Solidity Summit 2023 | Test Your Tests

23 of 52

Testing WAD Conversions

Solidity Summit 2023 | Test Your Tests

24 of 52

Testing WAD Conversions

Testing WAD Conversions (DON’T)

Solidity Summit 2023 | Test Your Tests

25 of 52

Testing WAD Conversions

Testing WAD Conversions (DON’T)

Solidity Summit 2023 | Test Your Tests

26 of 52

Testing WAD Conversions

Testing WAD Conversions (DON’T)

Solidity Summit 2023 | Test Your Tests

27 of 52

Testing WAD Conversions

Testing WAD Conversions (DON’T)

  • Know your tool!

Solidity Summit 2023 | Test Your Tests

28 of 52

Testing WAD Conversions

Testing WAD Conversions (DON’T)

  • Know your tool!
  • Don’t solely rely on one type of tests

Solidity Summit 2023 | Test Your Tests

29 of 52

Test WAD Conversions (DO)

​

​

​

​

    • Reduce complex decision trees
  • Know your tool!
  • Don’t solely rely on one type of tests
  • Restructure tests

Solidity Summit 2023 | Test Your Tests

30 of 52

Test WAD Conversions (DO)

​

​

​

​

    • Reduce complex decision trees
    • Split tests by outcome/behavior
  • Know your tool!
  • Don’t solely rely on one type of tests
  • Restructure tests

Solidity Summit 2023 | Test Your Tests

31 of 52

Test WAD Conversions (DO)

​

​

​

​

    • Reduce complex decision trees
    • Split tests by outcome/behavior
    • Ensure coverage around boundary points
  • Know your tool!
  • Don’t solely rely on one type of tests
  • Restructure tests

Solidity Summit 2023 | Test Your Tests

32 of 52

Test WAD Conversions (DO)

​

​

​

​

    • Reduce complex decision trees
    • Split tests by outcome/behavior
    • Ensure coverage around boundary points
    • Expect specific revert
  • Know your tool!
  • Don’t solely rely on one type of tests
  • Restructure tests

Solidity Summit 2023 | Test Your Tests

33 of 52

Shortcomings Exemplified:

Testing WAD Multiplication

Solidity Summit 2023 | Test Your Tests

34 of 52

Testing WAD Multiplication

Solidity Summit 2023 | Test Your Tests

35 of 52

Testing WAD Multiplication

Solidity Summit 2023 | Test Your Tests

36 of 52

Testing WAD Multiplication

Testing WAD Multiplication (DON’T)

Solidity Summit 2023 | Test Your Tests

37 of 52

Testing WAD Multiplication

Testing WAD Multiplication (DON’T)

&&

&&

Solidity Summit 2023 | Test Your Tests

38 of 52

Testing WAD Multiplication

Testing WAD Multiplication (DON’T)

&&

&&

  • Don’t blindly reuse code

​

Solidity Summit 2023 | Test Your Tests

39 of 52

Exploring Various Testing Strategies

  • Don’t blindly reuse code
  • Re-implement logic from a different angle

​

Solidity Summit 2023 | Test Your Tests

40 of 52

Exploring Various Testing Strategies

  • Don’t blindly reuse code
  • Re-implement logic from a different angle
  • Test multiple properties

​

Solidity Summit 2023 | Test Your Tests

41 of 52

Exploring Various Testing Strategies

  • Don’t blindly reuse code
  • Re-implement logic from a different angle
  • Test multiple properties
  • Include unit tests for special cases

​

Solidity Summit 2023 | Test Your Tests

42 of 52

Exploring Various Testing Strategies

  • Don’t blindly reuse code
  • Re-implement logic from a different angle
  • Test multiple properties
  • Include unit tests for special cases
  • Use differential fuzzing

​

Solidity Summit 2023 | Test Your Tests

43 of 52

Offensive Testing

Case Study:

Primitive Finance - Hyper

Solidity Summit 2023 | Test Your Tests

44 of 52

Primitive Finance - Hyper

  • CFMM with time-dependent curves (options-like trading)
  • Central pool balance accounting and batch swapping functionality
  • Non-trivial function approximations
  • Use of assembly and inconsistent rounding methods

​

=> Fuzz the swap function

​

Solidity Summit 2023 | Test Your Tests

45 of 52

Fuzz Test

​

Swapping

Back

And Forth

Solidity Summit 2023 | Test Your Tests

46 of 52

Solidity Summit 2023 | Test Your Tests

47 of 52

Refining

the Testing

Strategy

  • Address reverts and bound parameters

Solidity Summit 2023 | Test Your Tests

48 of 52

Refining

the Testing

Strategy

  • Address reverts and bound parameters
  • Sanity check setup

Solidity Summit 2023 | Test Your Tests

49 of 52

Refining

the Testing

Strategy

  • Address reverts and bound parameters
  • Sanity check setup
  • Question assumptions

Solidity Summit 2023 | Test Your Tests

50 of 52

Offensive Testing Conclusions

  • Requires a persistent, dynamic approach
  • Aim to actively find potential cracks
  • Question assumptions and validate your setup

Solidity Summit 2023 | Test Your Tests

51 of 52

Key Takeaways

  • Treat your tests as production code
  • Be aware of limitations of testing and tooling
  • Explore different testing strategies and techniques
  • Examine assumptions, preconditions, and conclusions of tests
  • Test your tests

Solidity Summit 2023 | Test Your Tests

52 of 52

Stay in Touch

Questions?

52

phaze

@lovethewired

Blog post

lovethewired.github.io

/blog/2023/test-your-tests

Solidity Summit 2023 | Test Your Tests