Allure integrations for C# test frameworks
🚀 New adapter for Reqnroll ⚠️ Namespace changes in Allure.NUnit and Allure.Xunit 🎯 New target frameworks of Allure.Xunit 🔬 Improvements New implementation of AddScreenDiff (#435) Standard failed/broken classification Other improvements 🐞 Bug fixes Deprecations and removals Other changes ⬆️ Dependency Updates 📦 Packaging changes 👻 Internal changes Pull requests and contributors New contributors
Meet Allure.Reqnroll - a new Allure adapter for Reqnroll.
Give it a try by installing the Allure.Reqnroll package to your Reqnroll project. Once it's done, you're all set! Run your tests and create the report from the allure-results in the project output directory.
The documentation is currently in progress. Once it's done, the link will be added to this description and the root README file of the repo.
Since Reqnroll is a fork of SpecFlow (it was forked back by Gáspár Nagy (gasparnagy), the original creator of SpecFlow), adapters for Reqnroll and SpecFlow have some similarities. However, some notable changes are listed below.
Allure.Reqnroll correctly works with all unit test frameworks supported by Reqnroll out-of-the-box: NUnit, xUnit.net, and MSTest. In contrast, Allure.SpecFlow currently can't work with xUnit.net due to problems with Allure context isolation (see #431).
Allure.Reqnroll integrates with Reqnroll in a way that differs from how Allure.SpecFlow integrates with SpecFlow. Because of that, users are not required to alter the Reqnroll configuration to enable the adapter.
SpecFlow, on the other hand, requires users to add an entry to the stepAssemblies list in specflow.json. A missing entry leads to an error and is a common source of problems.
Allure.SpecFlow features the specflow section of allureConfig.json where you may define Gherkin tag patterns and configure the table-to-parameter conversion.
In Allure.Reqnroll that section was moved to allure/gherkinPatterns. Some properties were renamed. The packages sub-section was removed (the package label is automatically filled by Allure.Reqnroll).
{
"allure": {
"gherkinPatterns": {
"stepArguments": {
"createFromDataTables": false,
"nameColumn": "ParameterName",
"valueColumn": "ParameterValue"
},
"grouping": {
"suites": {
"parentSuite": "allure\\.parentSuite:(.+)",
"suite": "allure\\.suite:(.+)",
"subSuite": "allure\\.subSuite:(.+)"
},
"behaviors": {
"epic": "allure\\.epic:(.+)",
"story": "allure\\.story:(.+)"
}
},
"metadata": {
"owner": "allure\\.owner:(.+)",
"severity": "(normal|blocker|critical|minor|trivial)",
"label": "allure\\.label\\.([^:]+):(.+)"
},
"links": {
"link": "allure\\.link:(.+)",
"issue": "allure\\.issue:(.+)",
"tms": "allure\\.tms:(.+)"
}
}
}
}
The schema can be accessed at https://raw.githubusercontent.com/allure-framework/allure-csharp/2.12.0/Allure.Reqnroll/Schemas/allureConfig.schema.json.
Allure.SpecFlow allows users to define patterns to convert Gherkin tags to Allure data. Allure.Reqnroll provides reasonable defaults for such patterns. That also contributes to the zero-configuration experience of Allure Reqnroll (see #439).
The default patterns used by Allure Reqnroll are equivalent to having the following allureConfig.json in place:
{
"allure": {
"gherkinPatterns": {
"stepArguments": {
"createFromDataTables": false
},
"grouping": {
"suites": {
"parentSuite": "allure\\.parentSuite:(.+)",
"suite": "allure\\.suite:(.+)",
"subSuite": "allure\\.subSuite:(.+)"
},
"behaviors": {
"epic": "allure\\.epic:(.+)",
"story": "allure\\.story:(.+)"
}
},
"metadata": {
"owner": "allure\\.owner:(.+)",
"severity": "(normal|blocker|critical|minor|trivial)",
"label": "allure\\.label\\.([^:]+):(.+)"
},
"links": {
"link": "allure\\.link:(.+)",
"issue": "allure\\.issue:(.+)",
"tms": "allure\\.tms:(.+)"
}
}
}
}
Unlike Allure SpecFlow, Allure.Reqnroll doesn't utilize runtime patching to implement test plans. That means features that depend on test plans (e.g., selective run) will work with Allure.Reqnroll in areas where they might not work with Allure.SpecFlow. That includes:
In this release, we target some historical discrepancies between package names and namespaces of Allure.NUnit and Allure.Xunit.
The root namespace of Allure.NUnit has been NUnit.Allure up until now. That's the namespace Nick Chursin (unickq) used in his original allure-nunit project until the project was transferred to allure-framework. It matched the package name at the time (NUnit.Allure; it is still available at nuget.org, although deprecated).
Once we changed the package name according to our general convention Allure.<Framework>, the namespace stopped matching the package name. That has created an extra barrier for users who usually expect the root namespace to be the same as the package name. That may also cause users to mix the code of Allure.NUnit and the code of NUnit.
In Allure.XUnit, there are two namespaces: Allure.Xunit and Allure.XUnit (note the casing of u). The public API has been split between both of them at the same time. The attributes have been contained in Allure.Xunit while some other APIs, like AllureStepAttribute - are in Allure.XUnit.
Now, the root namespace of Allure.NUnit is Allure.NUnit. The root namespace of Allure.Xunit is Allure.Xunit.
Additionally, the NUnit.Allure.Core.AllureNUnitAttribute should now be accessed via the root namespace instead of the Core sub-namespace, i.e., as Allure.NUnit.AllureNUnitAttribute.
We advise you to transition accordingly.
For example, given the following code:
using NUnit.Framework;
using NUnit.Allure.Attributes;
using NUnit.Allure.Core;
namespace MyNamespace;
[AllureNUnit]
class MyTestClass
{
[Test]
public void MyTest()
{
/* ... */
}
[AllureStep]
void MyStep()
{
/* ... */
}
}
It should be change to the following one:
using NUnit.Framework;
using Allure.NUnit.Attributes;
using Allure.NUnit;
namespace MyNamespace;
[AllureNUnit]
class MyTestClass
{
[Test]
public void MyTest()
{
/* ... */
}
[AllureStep]
void MyStep()
{
/* ... */
}
}
IDE features like "find & replace all" might assist you in the transition.
The transition becomes trivial if you've added Allure namespaces to the list of global usings. Given the following part of the .csproj file:
<Project Sdk="Microsoft.NET.Sdk">
<!-- ... -->
<ItemGroup>
<Using Include="Allure.Xunit.Attributes">
<Using Include="Allure.XUnit.Attributes.Steps">
<!-- Other usings -->
<!-- ... -->
</ItemGroup>
</Project>
Just a single character should be changed:
<Project Sdk="Microsoft.NET.Sdk">
<!-- ... -->
<ItemGroup>
<Using Include="Allure.Xunit.Attributes">
<Using Include="Allure.Xunit.Attributes.Steps"> <!-- Note the casing here -->
<!-- Other usings -->
<!-- ... -->
</ItemGroup>
</Project>
The old namespaces remain in place for both Allure.NUnit and Allure.Xunit. Accessing the API through them now triggers the CS0618 warning. While we advise you to transition to new namespaces, you may ignore the warnings and continue using the old ones.
We will remove the public API access from the old namespaces in the next major release.
Allure.Xunit now targets netstandard2.0 (down from netstandard2.1) and netcoreapp3.1 (increased from netcoreapp2.0 because of security issues; for example: CVE-2021-26701).
Due to how xUnit.net dependencies are managed, that means that the package works in the following runtimes:
Other runtimes are excepted to work except the following:
The general rule is this:
The runtime is supported by Allure.Xunit if, given that runtime, Nuget selects an assembly of
xunit.runner.utilitiesthat is compatible with the chosen assembly ofAllure.Xunit. I.e.:
- If the
netcoreapp3.1version ofAllure.Xunitis selected, thenetcoreapp1.0(*) version ofxunit.runner.utilitiesis also selected, OR- If the
netstandard2.0version ofAllure.Xunitis selected, thenetstandard1.5(*) version ofxunit.runner.utilitiesis also selected.(*) Specific versions of
xunit.runner.utilities's target frameworks are subject to change by the xUnit.net team in the future.
AllureApi.AddScreenDiff now aligns with allure-framework/allure2#1145. The new implementation allows multiple screen diffs to be attached to a single test result.
Previously, each adapter has used its own rules to distinguish between failed and broken tests/steps/fixtures:
brokenTestData configuration property. Usually, those were type names of exceptions.Xunit.Sdk namespace as failed; all other exceptions have corresponded to the broken status.Now, a general algorithm is implemented in Allure.Net.Commons. It checks the exception's type against a pre-defined list of type names associated with assertion errors. Each adapter provides its own list of such exception types. Users may redefine the list via the failExceptions property of allureConfig.json:
{
"allure": {
"failExceptions": [
"NUnit.Framework.AssertionException",
"MyCustomAssertionException"
]
}
}
If an exception matches an entry from the list, the status is resolved as failed. Otherwise, it's resolved as broken. An entry is considered a match if it equals to the full name of one of the following types:
Currently, if a type denoted by failExceptions is generic, it must be a closed constructed class.
Steps and fixtures added via AllureApi, ExtendedApi, [AllureStep], [AllureBefore], or [AllureAfter] use this algorithm automatically.
Allure.Xunit also uses that algorithm to resolve a test's status.
Allure.Reqnroll and Allure.SpecFlow both use it when resolving statuses of steps and features created from bindings, as well as statuses of test results created from scenarios.
Due to the limitations of NUnit, Allure.NUnit uses a different algorithm that analyzes assertion results provided by NUnit to distinguish between tests with unhandled exceptions and tests with a failed assertion. The new algorithm is more reliable than the old one and requires no extra configuration.
AllureLifecycle.ScheduleTestCase(TestResult testResult) was added to schedule a test case. The test case should then be started via AllureLifecycle.StartTestCase().ToString() in case serialization fails (#438, #448).dotnet test now doesn't return exit code 1 if Harmony fails to patch xUnit.net, but all tests pass (#441).[OneTimeTearDown] methods of a test class are annotated with [AllureAfter].[AllureAfter] now doesn't throw IndexOutOfRangeException if applied to a method that has parameters (#419).[AllureDisplayIgnored] is applied to the class.*reporters* pattern when trying to find a second reporter. That improves the startup time and prevents assembly loading exceptions in many scenarios (#412).[Theory(Skip = "...")] (#421).The following previously deprecated API is removed from Allure.Net.Commons:
CurrentTestIdGetter (it's not currently in use)StartTestContainer(string parentUuid, TestResultContainer container)
UpdateTestContainer(string uuid, Action<TestResultContainer> update)
StopTestContainer(string uuid)
WriteTestContainer(string uuid)
StartBeforeFixture(FixtureResult result, out string uuid)
StartBeforeFixture(string uuid, FixtureResult result)
StartBeforeFixture(string parentUuid, FixtureResult result, out string uuid)
StartBeforeFixture(string parentUuid, string uuid, FixtureResult result)
StartAfterFixture(string parentUuid, FixtureResult result, out string uuid)
StartAfterFixture(string parentUuid, string uuid, FixtureResult result)
UpdateFixture(string uuid, Action<FixtureResult> update)
StopFixture(string uuid)
StartTestCase(string containerUuid, TestResult testResult)
UpdateTestCase(string uuid, Action<TestResult> update)
StopTestCase(string uuid)
WriteTestCase(string uuid)
StartStep(StepResult result, out string uuid)
StartStep(string uuid, StepResult result)
StartStep(string parentUuid, string uuid, StepResult result)
UpdateStep(string uuid, Action<TestResult> update)
StopStep(string uuid)
StepFailedException
IStepLogger
IStepActionLogger
CoreStepsHelper (AllureApi/ExtendedApi should be used instead)The following API is deprecated now in Allure.NUnit:
AllureExtensions:
WrapSetUpTearDownParams - it has no effect.WrapInStep (all overloads) - use AllureApi.Step instead.The following previously deprecated API is removed from Allure.NUnit:
AllureNUnitHelper(*):
WrapInStep - use AllureApi.Step or [AllureStep] instead.SaveOneTimeResultToContext - it wasn't meant to be public and shouldn't be used.AddOneTimeSetupResult - it wasn't meant to be public and shouldn't be used.StepsHelper - [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.AllureNUnitAttribute:
AllureNUnitAttribute(bool wrapIntoStep) - the wrapIntoStep parameter has no effect and can safely be removed.AllureExtensions:
AddScreenDiff - use AllureApi.AddScreenDiff instead.(*) The AllureNUnitHelper and AllureExtensions classes are now internal.
The following previously deprecated API is removed from Allure.Xunit:
Allure.Xunit.AllureBefore
Allure.Xunit.AllureAfter
Allure.Xunit.AllureStep
Allure.Xunit.AllureStepBase
Use [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.AllureAttachments - use AllureApi.AddAttachment or Allure.Xunit.Attachments instead.AllureXunitHelper:
StartTestContainer
StartTestCase
MarkTestCaseAsFailedOrBroken
MarkTestCaseAsPassed
MarkTestCaseAsSkipped
FinishTestCase
Those methods weren't meant to be public and shouldn't be used at all.Steps - [AllureStep], [AllureBefore], [AllureAfter], AllureApi, or ExtendedApi instead.AllureXunitAttribute and AllureXunitTheoryAttribute - use [Fact] and [Theory] instead.The following APIs are internal now (those wasn't meant to be public in the first place):
AllureXunitFacade
AllureXunitHelper
AllureMessageSink
Commons and Adapters solution configuration were replaced with Publish. The Publish configuration includes all package projects in Release mode.Allure.Xunit.reporters.dll.dotnet test against the samples project.The full list of pull requests of the release:
Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.11.0...2.12.0
This release introduces Runtime API that can be used to affect the report during test execution. Previously, users had to utilize AllireLifecycle's methods, such as AllureLifecycle.UpdateTestCase, to achieve that. Now we advise all users to migrate to the new API whenever possible.
The API is accessible as static methods of Allure.Net.Commons.AllureApi. The full list of the class's methods includes:
SetTestName(string newName)
SetFixtureName(string newName)
SetStepName(string newName)
SetDescription(string description)
SetDescriptionHtml(string descriptionHtml)
AddLabels(params Label[] labels)
AddLabel(string name, string value)
AddLabel(Label newLabel)
SetSeverity(SeverityLevel severity)
SetOwner(string owner)
SetAllureId(int allureId)
AddTags(params string[] tags)
AddLink(string url)
AddLink(string name, string url)
AddIssue(string url)
AddIssue(string name, string url)
AddTmsItem(string url)
AddTmsItem(string name, string url)
AddLink(string name, string type, string url)
AddLinks(params Link[] links)
AddParentSuite(string parentSuite)
AddSuite(string suite)
AddSubSuite(string subSuite)
AddEpic(string epic)
AddFeature(string feature)
AddStory(string story)
Step(string name)
Step(string name, Action action)
Step<T>(string name, Func<T> function): T
async Step(string name, Func<Task> action): Task
async Step<T>(string name, Func<Task<T>> function): Task<T>
AddAttachment(string name, string type, string path)
AddAttachment(string name, string type, byte[] content, string fileExtension = "")
AddAttachment(string path, string? name = null)
AddScreenDiff(string expectedPng, string actualPng, string diffPng)
AddTestParameter(string name, object? value)
AddTestParameter(string name, object? value, ParameterMode mode)
AddTestParameter(string name, object? value, bool excluded)
AddTestParameter(string name, object? value, ParameterMode mode, bool excluded)
AddTestParameter(Parameter parameter)
Some less frequently used API methods are available as static methods of Allure.Net.Commons.ExtendedApi. Those are:
StartBeforeFixture(string name)
StartAfterFixture(string name)
PassFixture()
PassFixture(Action<FixtureResult> updateResults)
FailFixture()
FailFixture(Action<FixtureResult> updateResults)
BreakFixture()
BreakFixture(Action<FixtureResult> updateResults)
StartStep(string name)
StartStep(string name, Action<StepResult> updateResults)
PassStep()
PassStep(Action<StepResult> updateResults)
FailStep()
FailStep(Action<StepResult> updateResults)
BreakStep()
BreakStep(Action<StepResult> updateResults)
Before(string name, Action action)
Before<T>(string name, Func<T> function): T
async Before(string name, Func<Task> action): Task
async Before<T>(string name, Func<Task<T>> function): Task<T>
After(string name, Action action)
After<T>(string name, Func<T> function): T
async After(string name, Func<Task> action): Task
async After<T>(string name, Func<Task<T>> function): Task<T>
[!Note]
Allure.Net.Commons.Steps.CoreStepsHelperas its derivatives (NUnit.Allure.Core.StepsHelperandAllure.Xunit.Steps) are now deprecated in favor of the new API. The attachment-related methods fromAllure.Net.Commons.AllureLifecycleare also deprecated.
(implements #404 by @delatrie via #414)
Allure Report supports masked and hidden parameters of a test. Allure TestOps does, and the future major release of Allure Report will support excluded parameters of a test. Additionally, many other Allure integrations support excluded parameters of a test when calculating historyId of the test.
This release adds the mode and exclude fields to the Allure.Net.Commons.Parameter class as well as some overloads of the new AllureApi.AddTestParameter method to fill those fields. Allure NUnit, Allure xUnit.net, and Allure SpecFlow now ignore excluded parameters when calculating historyId (implements #425 by @delatrie via #414).
Link.Issue(string name) and Link.Tms(string name) now correctly fill the url field of the link they create. The parameter was renamed to url for both methods (fixes #416 by @delatrie via #414).historyId filled correctly (fixes #422 by @delatrie via #414).Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.10.0...2.11.0
Concurrency support by allure-csharp integrations was very limited. In this release we extend it. Steps from parallel/async tests as well as steps from parallel/async methods, sub-steps and operations now work as expected. That fixes a lot of concurrency-related errors that were usually manifested as NullPointerException (fixes #83, #106 and #367 by @delatrie via #371, #383 and #388 by @delatrie via #393).
Selective run is now supported for all integrations (implements #372 by @delatrie via #392).
Id properties (fullName, testCaseId, historyId) are fixed to prevent the following potential problems:
(implements #373 and #387 by @delatrie via #395)
Test and step parameters are now converted to strings using JSON conversion algorithm. It is much more versatile than the plain ToString() call.
The same goes for step parameters interpolated into the step's title (by @delatrie via #395).
A user can add an implementation of Allure.Net.Commons.TypeFormatter[T] via the AllureLifecycle.Instance.AddTypeFormatter[T] method. Previously, such formatters were just ignored. Now, they are automatically used to format test and step parameters as well as step titles in case the type of a parameter value is the same as the formatter's generic type argument (fixes #377 by @delatrie via #395).
JSON schemas are available for allureConfig.json at https://raw.githubusercontent.com/allure-framework/allure-csharp/<version>/<package>/Schemas/allureConfig.schema.json. You can use the schemas to validate the config file or explore available properties (by @delatrie via #395).
The language and framework labels are now added to all test results by all integrations (by @delatrie via #395).
Debug symbols (.snupkg files) are now available on NuGet for all packages (by @delatrie via #382).
InvalidCastException when underlying step returns Task<T> masked as non-generic Task (by @eranikid via #343).ArgumentOutOfRangeException was thrown (fixes #375 by @delatrie via #380).broken. A placeholder scenario is added instead to indicate an error (by @delatrie via #371).Fact and Theory are supported out-of-the-box (implements #344 by @delatrie via #366).Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.9.1...2.10.0
Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.9.5-preview.1...2.10.0-preview.1
Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.9.4-preview.6...2.9.5-preview.1
Full Changelog: https://github.com/allure-framework/allure-csharp/compare/2.9.4-preview.5...2.9.4-preview.6