Measuring performance with BenchmarkDotNet

Libraries/Frameworks/Tooling around .NET and .NET Core ecosystem

Albert Rodríguez, December 13th 2016

A little about myself

  • In development industry since 2008
  • Started with .NET 3.5 and XNA
  • Some of my interests:
    • Real time systems
    • ALM
    • Development productivity
    • Quality of Software
    • And of course communities!
  • Backend Dev @ Adaptive Financial Consulting: http://weareadaptive.com/careers/

The performance problem

  • Performance optimization is an art
  • Mainly based on experimentation
  • Some options:
    • End to end load tests
    • Real-time profiling tools
      • Visual Studio
      • JetBrains dotTrace, dotMemory
      • Redgate Ants
      • Telerik JustTrace
      • Glimpse
      • MiniProfiler
    • Benchmarking

Benchmarking

  • Analyze a specific part of the system
    • Isolate hot path
    • Resolve dependencies (if any)
    • Run one or multiple times
    • Gather results
    • Interpret results

Benchmarking is hard

  • Precise time measuring? > ms
  • Memory usage?
  • Garbage Collection?
  • Results influenced by other processes in the OS?
  • Different CLRs?
  • Graphical representation of data?

Introducing BenchmarkDotNet

  • Part of .NET Foundation (announced November 10th)
  • Used by many open source projects
  • Very common tool to use in performance PR discussions on GitHub
  • Easy to start with
  • Highly configurable
  • Focuses on microbenchmarks, it’s not a golden hammer!

“BenchmarkDotNet doesn't just blindly run your code - it tries to help you to conduct a qualitative performance investigation.”

Andrey Akinshin, Project Lead on BenchmarkDotNet

Getting started

Endless configuration combinations!

Config

Jobs

Runtime, JIT, GC, # Iterations

Columns

Statistics, Rank, Params

Exporters

Ascii, CSV, Markdown, R

Loggers

Diagnosers

Plugins to get useful info

GC allocation, JIT inlining

Analyzers

Validators

Set conditions before running benchmarks

Choose your config style

Attribute

Object

Command

You can also create your own Attributes and Objects!

Example, sorting algorithms

What happens under the covers?

Benchmark1.notcs

Benchmark2.notcs

BenchmarkN.notcs

CSV

Markdown

Html

R Plots

Run Project

Read Benchmark Config

ForEach config: Generate code

ForEach File: Compile code

Run benchmarks

Consolidate

Results

Export

Feature enabler, performance requirements!

  • Start tracking critical hot paths with regression tests
  • Review performance results over time
  • Integrate with CI system to warn and reject code merges if necessary
  • Always consider the pros and cons to deal with performance as a feature

How do I get started?

Performance references

Contact me

Measuring performance with BenchmarkDotNet - Google Slides