An extensible multiplatform framework to test GUIs in WinForms, WPF, GtkSharp and Xamarin.Mac.
A very simple platform to implement GUI tests on WinForms, WPF, GTK# and Xamarin.Mac applications.
Click on the following images to watch the videos of this system working!
This is the automated GUI testing system we use internally to develop Plastic SCM, and we thought it would be a good idea to share it for anyone to benefit from it.
It heavily depends on our own PNUnit, a modification to NUnit to run distributed tests we contributed more than a decade ago.
We develop GUIs for Windows, GNU/Linux, and macOS, and while we used a well-known commercial tool on Windows for GUI automated testing, we didn't have a solution for Linux and macOS. The existing solutions were not cross-platform.
So, we decided to go ahead and implement our own GUI testing system.
The beautiful parts are:
The less beautiful parts are:
ClickOkButton()
call, it all depends on how this method is implemented. If it invokes some GUI button.Click()
code, then it will be almost as good as actually sending the event to the app. If the implementer wants to fool you, it could be invoking the underlying logic, and then the test would be pointless. More on this in the how does it work section.The PNUnit framework, apart from the pnunit.framework.dll
library itself, consists on two applications: agent.exe
and launcher.exe
. The launcher
is responsible of telling the agent
which tests should run, and of gathering and reporting the results. The agent
is in charge of launching the application being tested, and terminating it if it doesn't exit on its own.
Because the tests need access to the actual application (the entire GUI, and not only the classes that make up your bussiness logic, the tests must run inside the application's context, and not inside the typical NUnit test runner. So, the agent
is not only in charge of launching the target application, but of serializing the test data and using the right arguments to pass it to the application.
This way, the application is not entirely test agnostic, as it ends up depending on the testing framework, but this is a small price to pay compared to the benefits.
While your application runs on the GUI's thread (and on any additional thread it might create, for example for I/O bound work), the test runs on another thread, and it manipulates UI elements through Invoke
on WinForms, and the like on other platforms. It is the developer's duty to write the code that manipulates the UI as faithful as possible to what the GUI framework would do / behave as with a real user.
The test code is written in C#, taking advantage of the NUnit platform, along with the custom WaitingAssert
class.
A typical section of a text looks at follows:
Assert.AreEqual(
"Goodbye", window.GetItemInListAt(0),
"The item 0 in the list does not match the expected one.");
Assert.AreEqual(
"Hello", window.GetItemInListAt(1),
"The item 1 in the list does not match the expected one.");
window.ChangeText("Hello");
window.ClickRemoveButton();
WaitingAssert.AreEqual(
"Removing element Hello...",
window.GetProgressMessage,
"The progress message does not match the expected one while" +
"removing 'Hello'.");
WaitingAssert.IsTrue(
window.AreButtonsEnabled,
"The window buttons were not re-enabled in a reasonable " +
"time after removing 'Hello'.");
As you can see, there are very specific interfaces like ChangeText
and ClickRemoveButton
. This framework is not a general purpose test framework, but something very simple to develop and maintain.
We built a sample GUI application to explain how testing works. In fact, we built 3 different ones: WinForms on Windows, GTKSharp on GNU/Linux, and Xamarin.Mac for macOS, but it is extensible to any GUI framework, as long as there are bindings to work with it from C#. You'd only need to adapt the UI specific code of the testing stack.
The sample application is really simple. It consist on a single window that has:
Add
and Remove
).Add
button, the application launches a background operation. While the operation runs, the application displays a progress text at the bottom describing what is happening under the hood (it is really just a Sleep
simulating some CPU bound or I/O). When the operation ends, the text introduced by the user is added to the list in alphabetical order. Then the text input is cleared.Remove
button the text is removed from the list.In short, we simply simulate what a regular GUI application does. You can see the multplatform application under testing bellow:
/src
: the main code of the example.
/application
: the source code of the application.
/lib
: the multiplatform bussiness logic. This library does not have code that depends on any UI framework or that is OS-specific.
/gui
: the GUI code.
/windows
: the Windows GUI application, based on WinForms./linux
: the Linux GUI application, based on GTKSharp (GTK+)./macos
: the macOS GUI application, based on Xamarin.Mac./testing
: the testing related source code. On this repository there is only GUI testing code, but unit tests would be here too.
/guitest
: the library that contains the tests./guitestinterfaces
: the interfaces (and some utility classes) that all of the applications (Windows, macOS & GNU/Linux) need to implement so the tests can interact with them./lib
: the third party libraries (source code or library assemblies).
/log4net
: the logging library used by the pnunit
framework./mono
: the library that the testing framework needs to identify the specific OS version and hardware platform when running under the Mono Runtime./nunit
: the well known NUnit testing framework, in its last 2.x release./pnunit
: Códice's contribution to the NUnit framework, stands for Parallel NUnit. It allows you to run tests in parallel both between them and to the application that is being tested. This version is a little bit different to the one you can find in the NUnit 2.6 release, with changes to make GUI testing work. Maybe in the future we'll release it as a standalone library, but for now you can use this version.You can build the solutions in this repository with Visual Studio, MonoDevelop, MSBuild, or XBuild.
The Windows application is based on WinForms, so with a standard Visual Studio install you should already have everything you need.
/src/lib/pnunit/pnunit.sln
and build it (Ctrl + Shift + B
by default).
/bin/pnunit
, including the agent
and the launcher
./src/application/gui/windows/windows.sln
and build it.
/bin/application
./bin/pnunit
./bin/pnunit
in two different terminals and launch the following:
> agent.exe agent.conf
launcher
issues a command.> launcher.exe wintest.conf
MSBuild.exe
is in your %PATH%
. It normally isn't.MSBuild.exe lib/pnunit/pnunit.sln
MSBuild.exe src/application/gui/windows/windows.sln
/bin/pnunit
in two different terminals and launch the following:
> agent.exe agent.conf
launcher
issues a command.> launcher.exe wintest.conf
You need to do the following before starting:
mono-complete
monodevelop
gtk-sharp2
These instructions work both on a complete GNU/Linux OS environment and on the WSL (Windows Subsystem for Linux). For the later, you will need:
DISPLAY=:0
for your session.Running Linux's GUI tests on Windows shows how flexible and rock-solid this system is!
/src/lib/pnunit/pnunit.sln
and build it (Ctrl + Shift + B
by default).
/bin/pnunit
, including the agent
and the launcher
./src/application/gui/windows/windows.sln
and build it.
/bin/application
./bin/pnunit
./bin/pnunit
in two different terminals and launch the following:
> agent agent.conf
launcher
issues a command.> launcher wintest.conf
You might need to give execution permissions to both the agent
and the launcher
scripts. To do so:
chmod u+x agent
chmod u+x launcher
msbuild.exe
is in your $PATH
. It should be after installing mono-complete
or monodevelop
. It is now the preferred tool for compiling C# code, replacing xbuild
.msbuild lib/pnunit/pnunit.sln
msbuild src/application/gui/windows/windows.sln
/bin/pnunit
in two different terminals and launch the following:
> agent agent.conf
launcher
issues a command.> launcher wintest.conf
You might need to give execution permissions to both the agent
and the launcher
scripts. To do so:
chmod u+x agent
chmod u+x launcher
Note: if you are using Windows Subsystem for Linux to test the GNU/Linux example using Windows (obscure scenario, but you might want to give it a try!), make sure you don't have the Windows' MSBuild executable in your Windows' PATH. Because you can execute Windows binaries from the WSL, if the Windows variant of MSBuild is in your PATH, when you try to compile the application using Mono's MSBuild, it will use some of the Windows' executables. The compile process will succeed, but the application won't launch.
You need to do the following before starting, if you don't have already done it in the past:
/src/lib/pnunit/pnunit.sln
and build it (cmd + Shift + B
by default).
/bin/pnunit
, including the agent
and the launcher
./src/application/gui/macos/macos.sln
and build it.
macos.app
) and its dependencies to /bin/application
./bin/pnunit
./bin/pnunit
in two different terminals and launch the following:
> mono agent.exe agent.conf
launcher
issues a command.> mono launcher.exe mactest.conf