HDK
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Benchmarker Class Reference

#include <benchmark.h>

Public Types

enum  Unit : int {
  Unit::autounit, Unit::ns, Unit::us, Unit::ms,
  Unit::s
}
 

Public Member Functions

 Benchmarker ()
 
template<typename FUNC , typename... ARGS>
double operator() (string_view name, FUNC func, ARGS &&...args)
 
double avg () const
 
double stddev () const
 
double range () const
 
double median () const
 
Benchmarkeriterations (size_t val)
 
size_t iterations () const
 
Benchmarkertrials (size_t val)
 
size_t trials () const
 
Benchmarkerwork (size_t val)
 
size_t work () const
 
Benchmarkerexclude_outliers (int e)
 
int exclude_outliers () const
 
Benchmarkerverbose (int v)
 
int verbose () const
 
Benchmarkerindent (int spaces)
 
int indent () const
 
Benchmarkerunits (Unit s)
 
Unit units () const
 
const std::stringname () const
 

Friends

OIIO_API std::ostream & operator<< (std::ostream &out, const Benchmarker &bench)
 

Detailed Description

Benchmarker is a class to assist with "micro-benchmarking". The goal is to discern how long it takes to run a snippet of code (function, lambda, etc). The code will be run in some number of trials, each consisting of many iterations, yielding statistics about the run time of the code.

Tne number of trials is user-selectable, with a reasonable default of 10 trials. The number of iterations per trial may be set explicitly, but the default is to automatically compute a reasonable number of iterations based on their timing. For most use cases, it's fire and forget.

Generally, the most and least expesive trials will be discarded (all sorts of things can happen to give you a few spurious results) and then the remainder of trials will be used to compute the average, standard deviation, range, and median value, in ns per iteration as well as millions of executions per second. The default behavior it just to echo the relevant statistics to the console.

The basic use illustrated by this example in which we try to assess the difference in speed between acos() and fast_acos():

Benchmarker bench;
float val = 0.5f;
clobber (val);  // Scrub compiler's knowledge of the value
bench ("acos", [&](){ DoNotOptimize(std::acos(val)); });
bench ("fast_acos", [&](){  // alternate indentation style
    DoNotOptimize(OIIO::fast_acos(val));
});

Which produces output like this: acos : 4.3 ns, 230.5 M/s (10x2097152, sdev=0.4ns rng=31.2%, med=4.6) fast_acos : 3.4 ns, 291.2 M/s (10x2097152, sdev=0.4ns rng=33.0%, med=3.4)

Some important details:

After declaring the Benchmarker, a number of options can be set: number of trials to run, iterations per trial (0 means automatic detection), verbosity, whether (or how many) outliers to exclude. You can chain them together if you want: bench.iterations(10000).trials(10);

It can be VERY hard to get valid benchmarks without the compiler messing up your results. Some tips:

  • Code that is too fast will not be reliable. Anything that appears to take less than 1 ns actually prints "unreliable" instead of full stats, figuring that it is likely that it has been inadvertently optimized away.
  • Use the DoNotOptimize() call on any final results computed by your benchmarked code, or else the compiler is likely to remove the code that leads to any values it thinks will never be used.
  • Beware of the compiler constant folding operations in your code – do not pass constants unless you want to benchmark its performance on known constants, and it is probably smart to ensure that all variables acccessed by your code should be passed to clobber() before running the benchmark, to confuse the compiler into not assuming its value.

Definition at line 142 of file benchmark.h.

Member Enumeration Documentation

enum Benchmarker::Unit : int
strong
Enumerator
autounit 
ns 
us 
ms 
s 

Definition at line 232 of file benchmark.h.

Constructor & Destructor Documentation

Benchmarker::Benchmarker ( )
inline

Definition at line 144 of file benchmark.h.

Member Function Documentation

double Benchmarker::avg ( ) const
inline

Definition at line 162 of file benchmark.h.

Benchmarker& Benchmarker::exclude_outliers ( int  e)
inline

Definition at line 205 of file benchmark.h.

int Benchmarker::exclude_outliers ( ) const
inline

Definition at line 210 of file benchmark.h.

Benchmarker& Benchmarker::indent ( int  spaces)
inline

Definition at line 224 of file benchmark.h.

int Benchmarker::indent ( ) const
inline

Definition at line 229 of file benchmark.h.

Benchmarker& Benchmarker::iterations ( size_t  val)
inline

Definition at line 170 of file benchmark.h.

size_t Benchmarker::iterations ( ) const
inline

Definition at line 175 of file benchmark.h.

double Benchmarker::median ( ) const
inline

Definition at line 165 of file benchmark.h.

const std::string& Benchmarker::name ( ) const
inline

Definition at line 245 of file benchmark.h.

template<typename FUNC , typename... ARGS>
double Benchmarker::operator() ( string_view  name,
FUNC  func,
ARGS &&...  args 
)
inline

Definition at line 151 of file benchmark.h.

double Benchmarker::range ( ) const
inline

Definition at line 164 of file benchmark.h.

double Benchmarker::stddev ( ) const
inline

Definition at line 163 of file benchmark.h.

Benchmarker& Benchmarker::trials ( size_t  val)
inline

Definition at line 178 of file benchmark.h.

size_t Benchmarker::trials ( ) const
inline

Definition at line 183 of file benchmark.h.

Benchmarker& Benchmarker::units ( Unit  s)
inline

Definition at line 238 of file benchmark.h.

Unit Benchmarker::units ( ) const
inline

Definition at line 243 of file benchmark.h.

Benchmarker& Benchmarker::verbose ( int  v)
inline

Definition at line 215 of file benchmark.h.

int Benchmarker::verbose ( ) const
inline

Definition at line 220 of file benchmark.h.

Benchmarker& Benchmarker::work ( size_t  val)
inline

Definition at line 193 of file benchmark.h.

size_t Benchmarker::work ( ) const
inline

Definition at line 198 of file benchmark.h.

Friends And Related Function Documentation

OIIO_API std::ostream& operator<< ( std::ostream &  out,
const Benchmarker bench 
)
friend

The documentation for this class was generated from the following file: