What is PNUnit?

PNUnit is an expansion to the NUnit framework we have developed to test our software. The "P" stands for "Parallel" and basically means that you can run several tests at the same time on different machines.

The PNUnit development

Our team is developing a new Software Configuration Management system. During the last year, starting from scratch, we have developed a new SCM server and some client tools (GUI, command line, and IDE integrations). Almost everything has been written in C#. We always wanted to make Plastic SCM run on non-windows platforms using Mono and now, it does.

One of our main concerns has always been testing. We have written unit tests with NUnit since the project start-up, but after a couple of sprints we needed to test the server's behavior under a heavy load, and under certain race conditions. We have evaluated several commercial tools (some of them we have been working with in previous projects with very good results) to automate client actions (both on the GUI and on the command line) performing several operations.

To really test our system we needed to simulate hundreds of clients against the same server. Unfortunately, the tools we could afford didn't work on both Windows and Linux platforms. Then we thought the NUnit platform could be the basis to distribute our tests over the network and reproduce real scenarios. We could write tests in NUnit automating our client command line tool (both directly writing tests using our classes and launching the tool as a separate process).

When we started, NUnit lacked the following: the ability to manage test launching on remote machines, gather the results, and test synchronization between different machines to create more complex scenarios.

The PNUnit structure

The scripts with the test configuration are read by the launcher, which is also responsible for creating the Runner. The Runner will launch the tests in the specified machines, and it provides the synchronization mechanism and the collection of results. The agent is able to launch tests remotely by creating instances of PNUnitAgent. Every time the PNUnitAgent receives a test to run, it creates an instance of PNUnitTestRunner,which will then run the test, gather the results, and send them to the Runner.

PNUnit basic Structure diagram

A very basic test

Let's check the result of a sum operation using a PNUnit test. We suppose we have a function called add, which is able to do the sum of two numbers. The code of the test itself (included in the tests.zip file below) is: 

[TestFixture]
public class Testing
{
  [Test]
  public void EqualTo19()
  {
    Assert.AreEqual(19, Cmp.Add(15,4));
  }
  ...
}

Notice that the structure is the same as NUnit. We generate the .dll file (TestLibraries.dll in this case) and we put it into the directory with all the PNUnit files included in the test.zip file. We then have to create a script with the configuration of the test. The following is the test.conf file. 

<TestGroup>
  <ParallelTests>
    <ParallelTest>
      <Name>Testing</Name>
      <Tests>
        <TestConf>
          <Name>Testing</Name>
          <Assembly>TestLibraries.dll</Assembly>
          <TestToRun>TestLibraries.Testing.EqualTo19</TestToRun>
          <Machine>localhost:8080</Machine>
          <TestParams>
            <string>..\server</string> <!-- server dir-->
            <string></string> <!-- database server -->
            <string></string><!-- conn string -->
          </TestParams>
        </TestConf>
      </Tests>
    </ParallelTest>
  </ParallelTests>
</TestGroup>

In Assembly, we have to put the name of the .dll file with the tests. In TestToRun we have to specify the individual test we want to run. In Machine we put the ip address (localhost for local testing) and the port. In agent.conf, it is stated that the port is 8080, so we can change it in both files and the test will still run properly.

Results of a PNUnit run

Then, we have to start the agent. In the command line:

C:\pnunit>start agent agent.conf

And to launch the test:

C:\pnunit>launcher test.conf

A synchronized test

One advantage of PNUnit is that it is possible to synchronize several tests. We will run a very simple test, with just a client and a server.

The client will wait for the server to start before issuing any commands. Here's the test code:

[Test]
  public void Server()
  {
    PNUnitServices.Get().InitBarrier("BARRIER");
    PNUnitServices.Get().WriteLine("Server started");
    Thread.Sleep(10000);

    PNUnitServices.Get().EnterBarrier("BARRIER");
    Assert.IsTrue(false, "The test failed");
  }

[Test]
  public void Client()
  {
    PNUnitServices.Get().WriteLine("The client should wait until the server starts");
    PNUnitServices.Get().InitBarrier("BARRIER");

    PNUnitServices.Get().EnterBarrier("BARRIER");

    Console.WriteLine("Server should be started now");
    Assert.IsTrue(true, "The test failed");
  }

The structure of the test.conf file is the same as in the previous case. The only difference is that in the new test, TestToRun, we write: 

<TestToRun>TestLibraries.Testing.Server</TestToRun>

<TestToRun>TestLibraries.Testing.Client</TestToRun>

The synchronization mechanism

The way PNUnit provides synchronization facilities is by means of barriers. The tests wait until all clients hit the barrier, and then they can pass.

How do we use it?

We have been using PNUnit for more than six months now. We have smoke tests that each developer runs locally when he finishes a task (together with normal NUnit tests). Smoke tests don't take a long time to run, and rapidly notify developers about broken things. They have proved to be very useful to us. We also have a bigger test battery that is run on each release. Those tests cover merge functionality, too. During the release process, both smoke and merge tests run on different platform combinations: Windows-Windows, Linux-Windows, and so on. All of our supported platforms are covered this way. In the following video, there are six machines running smoke tests from different platforms.

Then we have load and stress tests. Those are a different kind of test suite specifically written to be scalable. Using the config file, we can set up very different load test scenarios, configuring both the platforms and the number of machines involved. We are currently using a cluster to test our system under heavy load.

What can you do with PNUnit?

You can probably use PNUnits as we do. We have subtly modified the NUnit framework, and we are obtaining good results out of it, so we thought sharing it would be a good idea.

Download