Sltbench Versions Save

The cheapest to compile and the most popular part of sltbench is extracted to BenchCore.h header. It significantly reduces compilation cost of sltbench code layer in comparison to Bench.h. "Core" functionality covers:

• benchmarking of functions without arguments (SLTBENCH_FUNCTION)
• benchmark driver start (SLTBENCH_MAIN)

Observed speedup on benchcompile metric:

	dataset=simple, compilation time, sec	dataset=all, compilation time, sec
sltbench 2.4.0	2.5	246.5
sltbench 2.3.0	17.9	261.7

(results are obtained on Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz, clang-6.0.0)

Usage:

// #include <sltbench/Bench.h>  // compiles slower
#include <sltbench/BenchCore.h>  // compiles faster

void BM_fun()
{
    // ...
}
SLTBENCH_FUNCTION(BM_fun);  // ok, simple function benchmark is core API

SLTBENCH_MAIN();  // ok, driver start is core API

Release goal is to improve benchmarks (not library!) compilation time in order to speedup development cycle for huge projects.

Improvements:

• benchcompile metric shows 1.7x compilation time speedup:

codebase	compilation time, sec
sltbench r-2.3.0	263.1
sltbench r-2.2.0 + benchcompile unification patch	441.8

measured on Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz, clang-6.0, dataset=all

• Secondary and quite funny achievement is a binary size reduction:

codebase	libsltbench.a size, Kb, gcc 7-5.0
sltbench r-2.3.0	397
sltbench r-2.2.0	471

sltbench library is not targeting IOT devices obviously, so it was done just for achievement.

Followup notes:

• Compilation speed was improved by reducing includes and templates instantiation during benchmark compilation. Testing dataset is refined from user-layer code and targeted mostly to optimize compilation of sltbench layer. If user-layer code is the most expensive part of compilation, project might not gain from improvements.
• Another one significant compilation speedup might be done by breaking reporters API (smth. like change std::string to char* and chrono::nanoseconds to long long). Change is under consideration because there shouldn't be big number of custom reporters in a wild. If it will be implemented it would be a major version update.
• The most expensive part of sltbench-layer compilation is based on arguments conversion to std::string using std::ostream and arguments generators using std::vector. Problem fix would require to change all argument-based benchmarks in user code and will never be implemented. It is a drawback of initial architecture decisions. Relying on STL in library API probably was a bad idea.
• There is an intent to extract "core" part of sltbench layer to a single header (without arguments, generators and fixtures support). Users might include "core" part only and significantly speedup compilation if they do not need complex features.

The goal of this release is to introduce simplified API for managing fixtures.

New

• Added simplified fixtures: SLTBENCH_FUNCTION_WITH_FIXTURE_BUILDER. It allows simple fixtures without implementing custom Fixture class. Variants with arguments and arguments generators are also provided.
• Added runtime warning to console about DEBUG build configuration.
• Added DoNotOptimize utility function to prevent the code under testing being optimizated out by compiler.
• --name_prev and --name_next command line arguments of performance results comparator script allows to customize columns names with previous and next results.

Changes

• Startup heatup time decreased to 3 seconds (was 7 seconds).

The goal of this release is the documentation regorganization. README looks huge. Documentation restructuring seems a good way to decrease sltbench intergration cost.

• Reorganized documentation.
• Added "why sltbench" section to documentation to mension key features.
• Added comparison with googlebench by key metrics.

The goals of release 2.0.0:

• Fix and speedup cases when functions under testing takes about hundreds of nanoseconds.
• Implement tools for easier performance results comparison.
• Define the key product metrics and implement evaluation tools.

Changes:

• Speeded up benchmarking of fast functions (less than 0.00001 sec) in about 3x - 100x times.
• Removed 20-nanoseconds (approx.) overhead on function call, which is critical for functions with execution time less then 0.0000005 sec (500 nanoseconds). For other functions 20 nanoseconds fits into timing error and does not influe on performance results.
• Significantly improved reproducibility, quality and benchmarking time for functions with execution time about tens of nanoseconds.
• Now sltbench is about 4x times faster than googlebench on training dataset and produce more stable results (see metrics section below).
• Added tool to compare performance results. Usable to check was functions dataset speeded up or not. See tools/compare/main.py
• Added tools to compare key project metrics for sltbench and competitors like googlebench (see tools/metrics): benchmark time, performance results reproducibilty (rel. error) and compilation time.
• Added IMPRECISE status for datasets when sltbench cannot guarantee reproducible results. If you see IMPRECISE status, the problem is in a test in 99% cases. See the corresponding documentation notes about the most frequent cases of imprecise behaviour.
• Added --heatup off command line option to skip heatup stage.
• Added --reporter command line option to choose report format.
• Added warning log about the same test suite registered twice.

The goal of release 1.1.0 is to add lazy generators feature for memory-consuming benchmarks.

• Added lazy generators to save RAM for large input datasets.
• Speeded up benchmarking of fast functions (less than 1 ms) in about 20%.
• Added tarvis and appveyor CI.
• Added build instructions to README.

Hotfix

• Fixed --filter option when positional command line argument is provided.

Initial release

• Added benchmarks for procedures, fixtures, arguments and generators.
• Added bencmharks filtering by command line --filter option with regexp.
• Added instruction HOWTO implement and run benchmarks.
• Measure algorithm is learned on training dataset.
• The first sltbench release test functions about 3x times faster than googlebench. Looks reasonable.