Getting your App Continuously Tested

Being able to identify issues quickly and having an automated system that tells us whether or not the application is broken, sounds like a must have practice for software development companies nowadays. Nevertheless, we are still facing many challenges specially on large and complex systems.

So where do we start?

In my opinion, the first step in getting automated is having a CI (Continuous Integration) system in place and making sure the builds cover major branches.

The CI system needs to be able to deploy and run the application in isolation per branches, for example using different virtual machines / databases. This way we can make sure we have a known state of all resources and that we are able to roll back to that state after the tests run. Having the hardware/software resources needed for this is key to success.

In many cases, we may require changing the way the application is built and deployed, using tools and creating scripts that runs steps automatically in CI.

Being able to run tests automatically in CI

Select the tools that allows you to run automated tests unattended and getting tests results reports in CI. This way we make sure all the tests are executed after every code change, and that we also are able to run many tests on different environments/configurations.

Creating the tests

Get the whole team involved and make the creation of different levels of automated tests a part of development activities. Consider the team may need to improve their code design skills, which leads to testable code. Most of us have heard high cohesion and low coupling, for a long time; unfortunately it is common not seeing these applied.

Automated tests general guidelines

1. The test should communicate intent: it should be clear and simple what the test is verifying, and how the functionality is used by the application.
2. The test must have an assert.
3. The test must pass and fail reliably. The test should not have code branches i.e. if/else statements that cause it to not give a reliable pass/fail.
4. If for some reason, the test has code branches, there must not be one that doesn’t have an assert.
5. Keep tests independent: As tests grow, running the tests sequentially may be unpractical, so we need to make sure we can run tests in parallel and get quick feedback.
6. There must be a way to run separately unit, integration and end to end tests. The distinction between these must be clearly understood.
7. Unit tests must run fast.
8. Do not comment tests when they start failing, fix them.

This list can grow very long but at least this can be a good start =)

Hope this summary helps you and your team getting automated.

xUnit.Net – Running the tests (RunAfterTestFailed Custom Attribute)

On my last post I created a Custom TestCategory Attribute and custom xUnit.Net TestClassCommand to run test by category. Now I want to create a Custom RunAfterTestFailed attribute to run a method whenever a test fails. We have the following test class

using Xunit;
using Xunit.Extensions;

namespace xUnitCustomizations
{
    [CustomTestClassCommand]
    public class TestClass
    {
        [Fact, TestCategory("Unit")]
        public void FailedTest1()
        {
            Assert.True(false);
        }

        [Fact, TestCategory("Unit")]
        public void FailedTest2()
        {
            throw new InvalidOperationException();
        }

        [RunAfterTestFailed]
        public void TestFailed()
        {
            Debug.WriteLine("Run this whenever a test fails");
        }
    }
}

We will use CustomTestClassCommandAttribute and CustomTestClassCommand previously created, and made the following changes to EnumerateTestCommands method, since we need a custom command to be able to handle errors when executing test methods:

public class CustomTestClassCommand : ITestClassCommand
{
    .....

    #region ITestClassCommand Members

    .....

    public IEnumerable<ITestCommand> EnumerateTestCommands(IMethodInfo testMethod)
    {
        string skipReason = MethodUtility.GetSkipReason(testMethod);

        if (skipReason != null)
            yield return new SkipCommand(testMethod, MethodUtility.GetDisplayName(testMethod), skipReason);
        else
            foreach (var testCommand in cmd.EnumerateTestCommands(testMethod))
                yield return new AfterTestFailedCommand(testCommand);
    }
    ....
    #endregion
}

AfterTestFailedCommand will handle calling the execution of the test method and calling the proper method whenever a test fails

public class AfterTestFailedCommand : DelegatingTestCommand
{
    public AfterTestFailedCommand(ITestCommand innerCommand)
    : base(innerCommand)
    {}

    public override MethodResult Execute(object testClass)
    {
        MethodResult result = null;
        Type testClassType = testClass.GetType();
        try
        {
            result = InnerCommand.Execute(testClass);
        }
        finally
        {
            if (!(result is PassedResult))
            {
                foreach (MethodInfo method in testClassType.GetMethods())
                    foreach (var attr in method.GetCustomAttributes(typeof(RunAfterTestFailedAttribute), true))
                        method.Invoke(testClass, null);
            }
        }
        return result;
    }
}

I liked the extensibility that xUnit.Net provides, even when there isn’t a built in solution for TestCategory and RunAfterTestFailed attributes, I was able to build it by looking at the source code, the unit tests and examples available in CodePlex (the framework has unit tests!). Hope this series of posts was helpful to get you started in migrating from MSTest to xUnit.Net and writing custom extensions for this framework.

xUnit.Net – Running the tests (TestCategory)

In my previous post I showed an example about converting a MSTest class to xUnit.Net, and now I want to provide a solution for converting MSTest TestCategory attribute to an equivalent implementation in xUnit.Net.

MSTest allowed us to run the test that belongs to an specific category, let’s see a solution on how this can be accomplished in xUnit.Net.

using Xunit;
using Xunit.Extensions;

namespace xUnitCustomizations
{
    [CustomTestClassCommand]
    public class TestClass
    {
        [Fact, TestCategory("Unit")]
        public void FastTest()
        {
            Debug.WriteLine("fast test executed");
            Assert.True(true);
        }

        [Fact, TestCategory("Integration")]
        public void SlowTest()
        {
            Thread.Sleep(5000);
            Debug.WriteLine("slow test executed");
            Assert.True(true);
        }
    }
}

Create TestCategory attribute

 

namespace Xunit.Extensions
{
    public class TestCategoryAttribute : TraitAttribute
    {
        public TestCategoryAttribute(string category)
        : base("TestCategory", category) { }
    }
    ...
}

CustomTestClassCommandAttribute attribute is used to indicate that a custom test runner will be used

public class CustomTestClassCommandAttribute : RunWithAttribute
{
    public CustomTestClassCommandAttribute() : base(typeof(CustomTestClassCommand)) { }
}

CustomTestClassCommand  is the class that implements ITestClassCommand and acts as the runner for the test fixture

public class CustomTestClassCommand : ITestClassCommand
{
    // Delegate most of the work to the existing TestClassCommand class so that we
    // can preserve any existing behavior (like supporting IUseFixture<T>).
    readonly TestClassCommand cmd = new TestClassCommand();

    #region ITestClassCommand Members
    public object ObjectUnderTest
    {
        get { return cmd.ObjectUnderTest; }
    }

    public ITypeInfo TypeUnderTest
    {
        get { return cmd.TypeUnderTest; }
        set { cmd.TypeUnderTest = value; }
    }

    public int ChooseNextTest(ICollection<IMethodInfo> testsLeftToRun)
    {
        return cmd.ChooseNextTest(testsLeftToRun);
    }

    public Exception ClassFinish()
    {
        return cmd.ClassFinish();
    }

    public Exception ClassStart()
    {
        return cmd.ClassStart();
    }

    public IEnumerable<ITestCommand> EnumerateTestCommands(IMethodInfo testMethod)
    {
        return cmd.EnumerateTestCommands(testMethod);
    }

    public bool IsTestMethod(IMethodInfo testMethod)
    {
        return cmd.IsTestMethod(testMethod);
    }

    public IEnumerable<IMethodInfo> EnumerateTestMethods()
    {
        string category;

        foreach (IMethodInfo method in cmd.EnumerateTestMethods())
        {
            category = string.Empty;
            foreach (IAttributeInfo attr in method.GetCustomAttributes(typeof(TestCategoryAttribute)))
                category = attr.GetPropertyValue<string>("Value");

            if (category.ToLower().Contains("unit")) // We can make this configurable
                yield return method;
        }
    }
    #endregion
}

The Method public IEnumerable<IMethodInfo> EnumerateTestMethods() filters the tests methods by TestCategory's attribute Value, note that we can make this configurable so we can configure our CI server to run unit test as soon as there are changes in the repository to provide quick feedback and schedule (e.g. once a day) the execution slower test like integration or Web UI test.