NBench Versions Save

v1.0.0 March 14 2017

NBench v1.0.0 represents support for .NET Standard 1.6 for the NBench core library and a .NET Core 1.0 version of the NBench runner.

This release introduces a breaking change to NBench:

• NBench core library .NET Framework target increased from 4.5 -> 4.5.2. Client projects must target 4.5.2 or greater.

How to use the .NET Core NBench runner:

• In your .NET Core performance test project, add the following dependency element to the .csproj file:

  <ItemGroup>
    <DotNetCliToolReference Include="NBench.Runner.DotNetCli" Version="1.0.0" />
  </ItemGroup>
  

• Save the .csproj file (if using Visual Studio 2017) or run dotnet restore in the project location.

• From a command prompt within the project's parent directory, run dotnet nbench project_name.dll arguments...

You can see the full list of changes in NBench 1.0.0 here

v0.3.4 December 15 2016

NBench v0.3.4 is a bugfix for RunMode.ThroughPut benchmarks where we regularly had false negatives on asserting number of operations per second.. This patch fixes this issue.

0.3.3 December 07 2016

NBench v0.3.3 includes a handful of bug fixes, but also enables TeamCity output formatting for NBench specifications.

To enable TeamCity output formatting explicitly, pass in the following argument to the NBench.Runner.exe

NBench.Runner.exe [assembly names] [teamcity=true]

Implemented first cut of TeamCity support, but had a bug with printing out failures

0.3.1 August 15 2016

NBench v0.3.1 introduces full Mono support for cross-platform benchmarks to both NBench.Runner as well as the core NBench library.

No major new changes have been added feature wise, but we've worked around issues specific to Linux permissions and have tailored the core NBench package to be able to run benchmarks on any platform.

Here's an example of some side-by-side NBench benchmarks on Mono vs. .NET 4.6.2.

NBench.PerformanceCounters is platform-specific to Windows and thus can't be supported on Linux.

Read the full list of changes in NBench v0.3.1 here.

0.3.0 May 24 2016

This release introduces some breaking changes to NBench:

Tracing The biggest feature included in this release is the addition of tracing support, which is exposed directly to end-users so they can capture trace events and include them in the output produced by NBench.

You can access the IBenchmarkTrace object through the BenchmarkContext passed into any of your PerfSetup, PerfBenchmark, or PerfCleanup methods like such:

public class TracingBenchmark
{

    [PerfSetup]
    public void Setup(BenchmarkContext context)
    {
        context.Trace.Debug(SetupTrace);
    }

    [PerfBenchmark(TestMode = TestMode.Test, NumberOfIterations = IterationCount, RunTimeMilliseconds = 1000)]
    [MemoryMeasurement(MemoryMetric.TotalBytesAllocated)]
    [MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThan, ByteConstants.EightKb)]
    public void Run1(BenchmarkContext context)
    {
        context.Trace.Debug(RunTrace);
    }

    [PerfCleanup]
    public void Cleanup(BenchmarkContext context)
    {
        context.Trace.Info(CleanupTrace);
    }
}

NBench.Runner.exe now takes a trace=true|false commandline argument, which will enable the new tracing feature introduced in this release.

Tracing is disabled by default.

Skippable Warmups You can now elect to skip warmups altogether for your specs. This feature is particularly useful for long-running iteration benchmarks, which are often used for stress tests. Warmups don't add any value here.

Here's how you can skip warmups:

[PerfBenchmark(TestMode = TestMode.Test, NumberOfIterations = IterationCount, RunTimeMilliseconds = 1000, SkipWarmups = true)]
[MemoryMeasurement(MemoryMetric.TotalBytesAllocated)]
[MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThan, ByteConstants.EightKb)]
public void Run1(BenchmarkContext context)
{
    context.Trace.Debug(RunTrace);
}

Just set SkipWarmups = true on your PerfBenchmark attribute wherever you wish to skip a warmup.

Foreground thread is no longer given high priority when concurrent mode is on.

If you are running the NBench.Runner with concurrent=true, we no longer give the main foreground thread high priority as this resulted in some unfair scheduling during concurrent tests. All threads within the NBench.Runner process all share the same priority now.

Markdown reports include additional data All markdown reports now include:

• The concurrency setting for NBench
• The tracing setting for NBench
• A flag indicating if warmups were skipped or not

All of these were added in order to make it easy for end-users reading the reports to know what the NBench settings were at the time the report was produced.

0.2.2 May 03 2016

Warmup count is now equal to iteration count on all benchmarks, useful for users with long-running macro benchmarks and stress tests.

0.2.1 March 22 2016

Fixed issue with NuGet logos and concurrency settings - we now still keep the process priority set to high, as we did before.

0.2.0 March 22 2016

Major changes in NBench v0.2, beginning with our new logo:

First, we've added an extensible plugin API for capturing third-party metrics not natively provided by NBench. We will be providing more detailed documentation for this in a later release.

NBench.PerformanceCounters The first example of this can be found in NBench.PerformanceCounters, a brand new NuGet package that allows you to instrument any arbitrary Windows PerformanceCounter on any of your tests.

PS> Install-Package NBench.PerformanceCounters

This package introduces the three following attributes you can use on your benchmarks and performance tests:

• PerformanceCounterMeasurementAttribute - measures any available performance counter.
• PerformanceCounterThroughputAssertion - asserts a performance counter's per-second value.
• PerformanceCounterTotalAssertion - asserts a performance counter's total value.

TimingMeasurement and ElapsedTimeAssertion Somewhat related to traditional CounterMeasurements, we've added two new attributes which allow you to measure and assert against the total amount of elapsed time it took to run a particular block of code.

• TimingMeasurementAttribute - reports on the elapsed time A run of a benchmark took in milliseconds. Designed to work with RunMode.Iterations benchmarks.
• ElapsedTimeAssertionAttribute - performs a bounds-checking assertion on amount of time it took to run a particular benchmark. Designed to work with RunMode.Iterations benchmarks.

These are now available as part of the core NBench package.

Additional NBench.Runner Options NBench.Runner now supports a new flag argument, concurrent=true|false

NBench.Runner.exe [assembly names] [output-directory={dir-path}] [configuration={file-path}] [include=MyTest*.Perf*,Other*Spec] [exclude=*Long*] [concurrent={true|false}]

concurrent=true|false - disables thread priority and processor affinity operations for all benchmarks. Used only when running multi-threaded benchmarks. Set to false (single-threaded) by default.

0.1.6 February 15 2016

Includes following features:

• Support for config files