1 of 8

Common Goals of Benchmarking Studies

Carola Doerr, Thomas Bartz-Beielstein,

Boris Naujoks

2 of 8

Good Benchmarking Practice

Common interest to improve benchmarking

  • No good resource available? WRONG!
  • Benchmarking in Optimization: Best Practice and Open Issues
    • Thomas Bartz-Beielstein, Carola Doerr, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, Manuel Lopez-Ibanez, Katherine M. Malan, Jason H. Moore, Boris Naujoks, Patryk Orzechowski, Vanessa Volz, Markus Wagner, Thomas Weise
    • 10 chapters, 11/54 pages of reference
    • Available here: https://arxiv.org/abs/2007.03488
  • Benchmarking Network: https://sites.google.com/view/benchmarking-network/

3 of 8

Performance Extrapolation

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development

4 of 8

​

​

  • G1.1 Basic Assessment of Performance and Search Behavior

​

  • G1.2 Algorithm Comparison

​

  • G1.3 Competition

​

  • G1.4 Assessment of the Optimization Problem

​

  • G1.5 Illustrating Algorithms’ Search Behavior

Performance Extrapolation

​

​

​

​

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development

5 of 8

​

​

  • G2.1 Testing Invariances

​

  • G2.2 Algorithm Tuning

​

  • G2.3 Understanding the Influence of Parameters and Algorithmic Components

​

  • G2.4 Characterizing Algorithms’ Performance by Problem Features
  • G1.1 Basic Assessment of Performance and Search Behavior

​

  • G1.2 Algorithm Comparison

​

  • G1.3 Competition

​

  • G1.4 Assessment of the Optimization Problem

​

  • G1.5 Illustrating Algorithms’ Search Behavior

Performance Extrapolation

​

​

​

​

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development

6 of 8

  • G3.1 Performance Regression

​

  • G3.2 Automated Algorithm Design, Selection, and Configuration

​

  • G2.1 Testing Invariances

​

  • G2.2 Algorithm Tuning

​

  • G2.3 Understanding the Influence of Parameters and Algorithmic Components

​

  • G2.4 Characterizing Algorithms’ Performance by Problem Features
  • G1.1 Basic Assessment of Performance and Search Behavior

​

  • G1.2 Algorithm Comparison

​

  • G1.3 Competition

​

  • G1.4 Assessment of the Optimization Problem

​

  • G1.5 Illustrating Algorithms’ Search Behavior

Performance Extrapolation

​

​

​

​

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development

7 of 8

  • G3.1 Performance Regression

​

  • G3.2 Automated Algorithm Design, Selection, and Configuration

​

  • G2.1 Testing Invariances

​

  • G2.2 Algorithm Tuning

​

  • G2.3 Understanding the Influence of Parameters and Algorithmic Components

​

  • G2.4 Characterizing Algorithms’ Performance by Problem Features
  • G1.1 Basic Assessment of Performance and Search Behavior

​

  • G1.2 Algorithm Comparison

​

  • G1.3 Competition

​

  • G1.4 Assessment of the Optimization Problem

​

  • G1.5 Illustrating Algorithms’ Search Behavior

Performance Extrapolation

  • G4.1 Cross- Validation and Complementation of Theoretical Results

​

  • G4.2 Source of Inspiration for Theoretical Studies

​

  • G4.3 Benchmarking as Intermediary between Theory and Practice

​

​

​

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development

8 of 8

  • G3.1 Performance Regression

​

  • G3.2 Automated Algorithm Design, Selection, and Configuration

​

  • G2.1 Testing Invariances

​

  • G2.2 Algorithm Tuning

​

  • G2.3 Understanding the Influence of Parameters and Algorithmic Components

​

  • G2.4 Characterizing Algorithms’ Performance by Problem Features
  • G1.1 Basic Assessment of Performance and Search Behavior

​

  • G1.2 Algorithm Comparison

​

  • G1.3 Competition

​

  • G1.4 Assessment of the Optimization Problem

​

  • G1.5 Illustrating Algorithms’ Search Behavior

Performance Extrapolation

  • G4.1 Cross- Validation and Complementation of Theoretical Results

​

  • G4.2 Source of Inspiration for Theoretical Studies

​

  • G4.3 Benchmarking as Intermediary between Theory and Practice

​

  • G5.1 Code Validation

​

  • G5.2 Algorithm Development

​

Common Goals of Benchmarking Studies

Sensitivity of Performance

Visualization and Basic Assessment

Theory- Oriented Goals

Benchmarking in Algorithm Development