Getting your App Continuously Tested

Being able to identify issues quickly and having an automated system that tells us whether or not the application is broken, sounds like a must have practice for software development companies nowadays. Nevertheless, we are still facing many challenges specially on large and complex systems.

So where do we start?

In my opinion, the first step in getting automated is having a CI (Continuous Integration) system in place and making sure the builds cover major branches.

The CI system needs to be able to deploy and run the application in isolation per branches, for example using different virtual machines / databases. This way we can make sure we have a known state of all resources and that we are able to roll back to that state after the tests run. Having the hardware/software resources needed for this is key to success.

In many cases, we may require changing the way the application is built and deployed, using tools and creating scripts that runs steps automatically in CI.

Being able to run tests automatically in CI

Select the tools that allows you to run automated tests unattended and getting tests results reports in CI. This way we make sure all the tests are executed after every code change, and that we also are able to run many tests on different environments/configurations.

Creating the tests

Get the whole team involved and make the creation of different levels of automated tests a part of development activities. Consider the team may need to improve their code design skills, which leads to testable code. Most of us have heard high cohesion and low coupling, for a long time; unfortunately it is common not seeing these applied.

Automated tests general guidelines

1. The test should communicate intent: it should be clear and simple what the test is verifying, and how the functionality is used by the application.
2. The test must have an assert.
3. The test must pass and fail reliably. The test should not have code branches i.e. if/else statements that cause it to not give a reliable pass/fail.
4. If for some reason, the test has code branches, there must not be one that doesn’t have an assert.
5. Keep tests independent: As tests grow, running the tests sequentially may be unpractical, so we need to make sure we can run tests in parallel and get quick feedback.
6. There must be a way to run separately unit, integration and end to end tests. The distinction between these must be clearly understood.
7. Unit tests must run fast.
8. Do not comment tests when they start failing, fix them.

This list can grow very long but at least this can be a good start =)

Hope this summary helps you and your team getting automated.

xUnit.Net – Running the tests (ClassInitialize – ClassCleanup)

I started using xUnit.Net few weeks ago. My first question was how to do the things I was used to do in other testing frameworks like MSTest or NUnit, specially when using these frameworks not only for unit testing but for higher level tests like Selenium RC web tests. So far, the framework seems to be very good and extensible.

I am going to show some scenarios I have run into converting MSTest to xUnit.Net, having the following class

using Microsoft.VisualStudio.TestTools.UnitTesting;
[TestClass]
public class AdminPageTest
{
    static SeleniumWebTestContext test = new SeleniumWebTestContext();

    [ClassInitialize()]
    public static void ClassInitialize() 
    {
        new LoginPage(test).LoginUser(“maria”, “******”);
    }

    [ClassCleanup()]
    public static void ClassCleanup() 
    {
        new AdminPage(test.Driver).Logout();
        test.StopWebTestContext();
    }

    [TestCategory("WebTest"), TestMethod]
    public void TestViewMyProfile()
    {
        var profileWindow = new AdminPage(test.Driver).SelectMyProfile();
        Assert.IsTrue(profileWindow.CheckFirstName(“Maria”));
        profileWindow.Close();
    }

    [TestCategory("WebTest"), TestMethod]
    public void TestAdminSearchUser()
    {
        var userWindow = new AdminPage(test.Driver).SelectUserManagement();
        userWindow.Search("Marcano Maria");
        Assert.IsTrue(adminTab.VerifyEmail("my-email@domain.com"));
    }
}

Note that SeleniumWebTestContext is holding information about Selenium RC and starts the server.

ClassInitialize – ClassCleanup / IUseFixture<T>

Sometimes we require sharing information among the methods defined in a class, for example in web tests we want to share the logged in the user information, execute different actions and validations and log out at the end:

using Xunit;
public class AdminPageTest : IUseFixture<LoggedUserContext>
{
    SeleniumWebTestContext test;
    public AdminPageTest()
    {  
    }

    public void SetFixture(LoggedUserContext usrContext)
    {
        test = usrContext.Test;
    }
    ...
}

and the LoggedUserContext class will look like this:

public class LoggedUserContext : IDisposable
{    
    public SeleniumWebTestContext Test;    
    public LoggedUserContext()    
    {        
        Test = new SeleniumWebTestContext();        
        new LoginPage(Test).LoginUser(“maria”, “******”);    
    }    

    public void Dispose()    
    {       
        new AdminPage(Test.Driver).Logout();
        Test.StopWebTestContext();
    }
}

Selenium’s How-To

Recently I ran into a couple of issues that started affecting some selenium automated tests I was developing, below the issues and solutions implemented:

XHR ERROR: Response_Code = –1 Request Error

When executing the open command the tests started failing with the following error

XHR ERROR: URL = https:// Response_Code = -1 Error_Message = Request Error. on Firefox version 3.6.12 and Selenium RC version 1.0.3

I found that the issue was reported by other users (issue 408 on selenium project page). The open command started working again after making the suggested workaround on comment 4,  calling the open command and passing the second parameter true:

public void CustomOpen(string url)
{     
    commandProcessor.DoCommand("open", new String[] { url, "true" });
}

According to comment 4, the issue was caused by a feature that was supposed to be disabled.

The issue seems to be related to that second parameter (ignoreResponseCode) on the open command and the default value assigned when it is null or empty, there is more details about the fix on commend 14. The fix hasn't been released yet, so if you run into this issue you can use this workaround.

Https pages

When running selenium on development / test environment, usually there is a need to deal with the certificate exceptions thrown by the browser.

I was experiencing the following behavior when running the tests using *firefoxproxy mode, this mode will remember the certificate exception added but if the site is redirected automatically to other location, Selenium throws permission denied exception (I supposed this is related to Same Origin Policy restriction).

Reading a little further, *firefoxproxy mode is supported only for backward compatibility and *firefox mode should be use in Selenium-RC 1.0 beta 2 and later.

I tried the following solutions mentioned here running selenium using *firefox mode:

• Creating a firefox profile and run selenium with that profile
  
• Use RCE - Remember Certificate Exception firefox add-on
  

Both approaches worked, but the test execution experiences a small delay when using RCE.

CodeProject

Automating MyAppInFlex.swf – Useful FlexPilot commands

I continue automating my Flex application with FlexPilot. One clear advantage I found when using FlexPilot is that I was able to access more of the UI elements of a complex Flex interface. The other tools I tried weren’t picking all those elements, but I must admit that I didn’t investigate further.

An interesting feature is FlexPilot’s chain syntax, that allows to access flex UI elements by different properties, like id, label or text. Another feature I liked is easier integration to work with selenium RC and selenium IDE.

Below some usage examples to create automated tests for MyAppInFlex.swf, and also serve as a quick reference on how to interact/access the flex UI.

Click on a Button

The easiest way to access an element is using the element’s id, for example click on a button:

selenium.flexClick("id=MyAppInFlex", "chain=id:okButton");

If the button you are trying to access doesn't have an id, you are able to find it using the label property:

selenium.flexClick("id=MyAppInFlex", "chain="id:buttonBar/label:Search");

Access an element on a Grid

Click an element that has an specific value on a Grid

selenium.flexClick("id=MyAppInFlex", "chain="id:userGrid/name:AdvancedListBaseContentHolder*/name:AdvancedListBaseContentHolder*/text:Maria Marcano");

Check the value on a TextInput

Check a textbox has a specific value

selenium.flexAssertProperty("id=MyAppInFlex", "chain="id:emailTextBox, validator=text|emailvalue@domain.com", );

To improve the application testability, add id’s to the objects you want to access. This also applies for web application in general (accessing elements by id’s is faster than processing xpath expressions).

Some limitations

Elements on a grid can’t be accessed by position, for example click on the first/second element on a grid.

I’ve been experiencing some issues with some application builds that prevent the use of the flex explorer and recorder (reported the bug, hopefully it gets fixed soon). You can still run the tests, just not using those tools to capture the elements.

CruiseControl.Net and MSTest

I wanted to put together some notes about CruiseControl.Net and MSTest to have it as reference:

First, to be able to generate reports the following element is required on ccnet.config file.

<xmllogger />     

this element is used to create the log files read by the CruiseControl.NET web page, if it isn't defined the web page will throw the following error:

Exception Message
Request processing has failed on the remote server: Unable to find Log Publisher for project so can't find log file.

To view the MSTest report, we need to add the result file in ccnet.config to merge the content in the build report, for example :

..................
<exec>
  <executable>C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe</executable>        
  <baseDirectory>.\</baseDirectory>
  <buildArgs> /testcontainer:SeleniumSearch\bin\Debug\SeleniumSearch.dll /resultsfile:Results\results1.trx</buildArgs>
</exec>     
tasks>
<publishers>
<merge>
  <files>
    <file>.\Results\results1.trx</file>
  </files>
</merge> 
<xmllogger />
</publishers>
..................

Include the MSTest plugin to generate the report on the web page, for example dashboard.config file

..................
<buildPlugins>
  <buildReportBuildPlugin>
    <xslFileNames>
      <xslFile>xsl\header.xsl</xslFile>
      <xslFile>xsl\modifications.xsl</xslFile>
      <xslFile>xsl\MsTestSummary2008.xsl</xslFile>          
    </xslFileNames>
  </buildReportBuildPlugin>
  <buildLogBuildPlugin />
  <xslReportBuildPlugin description="MSTest2008 Report" actionName="MSTESTReport" xslFileName="xsl\MsTestReport2008.xsl"/>             
</buildPlugins>
..................

And include the xls file for MSTest report on ccservice.exe.config/ccnet.exe.config

..................
<!-- Specifies the stylesheets that are used to transform the build results when using the EmailPublisher -->
<xslFiles>
    <file name="xsl\header.xsl"/>
    <file name="xsl\compile.xsl"/>
    <file name="xsl\unittests.xsl"/>
    <file name="xsl\MsTestSummary2008.xsl"/>        
    <file name="xsl\fit.xsl"/>
    <file name="xsl\modifications.xsl"/>
    <file name="xsl\fxcop-summary_1_36.xsl"/>
</xslFiles>
..................

The xls used on the above example is the one included in ccnet installation. I tested the report generation running tests on VS 2010 and VS 2008 and both reports are generated correctly. The only thing that caused issues was having MSTest on the path windows environment variables on a computer that has both VS 2010 an VS 2008

Also, if MSTest needs to connect to external resources, like accessing remote files or connecting to SQL Server, make sure the account that runs CruiseControl.Net Server windows service (by default is Local System Account) has access to those resources.

CruiseControl.Net driving Selenium Tests

This is how you can quickly get CruiseControl.Net running and driving your selenium test – in 6 steps:

1. Donwload & Install CruiseControl.Net
2. Start “CruiseControl.NET Server“ Windows Service
3. Download SeleniumTest sample project and extract it on a folder C:\Projects\.
4. Edit CruiseControl.NET configuration file (C:\Program Files (x86)\CruiseControl.NET\server\ccnet.config) and replace it with the information in C:\Projects\SeleniumTest\CruiseControl.Net\ccnet.config. Modify it with proper locations of MSBuild.exe, MsBuild.dll and MSTest.exe 
5. Start selenium server by running C:\Projects\SeleniumTest\SeleniumServerPort4441.bat.
6. Open http://localhost/ccnet and you will se the project SeleniumSearchTest1 click on Force. It should build the project and start the tests.

The tests will run using chrome, you can change this parameter in C:\Projects\SeleniumTest\SeleniumSearch\SeleniumSearchTest.cs and use "*iehta" for Internet Explorer or  "*firefoxproxy" for Firefox.

// "*iehta" IE -  "*firefoxproxy" FF - *googlechrome
selenium = new DefaultSelenium("localhost", 4441, "*googlechrome", "http://seleniumhq.org/"); 

This is how the ccnet.config file looks

<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
  <project>
    <name>SeleniumSearchTest1</name>
    <triggers>
      <scheduleTrigger time="23:30" buildCondition="ForceBuild">
        <weekDays>
            <weekDay>Monday</weekDay>
            <weekDay>Tuesday</weekDay>
            <weekDay>Wednesday</weekDay>
            <weekDay>Thursday</weekDay>
            <weekDay>Friday</weekDay>
        </weekDays>
      </scheduleTrigger>
    </triggers>
    <workingDirectory>C:\Projects\SeleniumTest</workingDirectory>
    <tasks>
      <msbuild>
        <executable>C:\Windows\Microsoft.NET\Framework\v3.5\MSBuild.exe</executable>
        <workingDirectory>C:\Projects\SeleniumTest</workingDirectory>
        <projectFile>SeleniumTest.sln</projectFile>
        <buildArgs>/noconsolelogger /p:Configuration=Debug /v:diag</buildArgs>
        <targets>Build</targets>
        <timeout>900</timeout>
        <logger>C:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
      </msbuild>
      <exec>
        <executable>DelResults.bat</executable>
        <baseDirectory>.\</baseDirectory>
      </exec>     
      <exec>
        <executable>C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe</executable>        
        <baseDirectory>.\</baseDirectory>
        <buildArgs> /testcontainer:SeleniumSearch\bin\Debug\SeleniumSearch.dll /resultsfile:Results\results1.xml</buildArgs>
      </exec>     
    </tasks>
    <publishers>
      <merge>
        <files>
          <file>.\Results\results1.xml</file>
        </files>
      </merge>
      <xmllogger />
    </publishers>
  </project>
</cruisecontrol>

This example applies for VS 2008 Professional, ccnet 1.5.7256.1 and Selenium RC 1.0.3

Links:

http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET

http://github.com/downloads/mariangemarcano/seleniumtest/SeleniumTest-Code.zip

Selenium Testing with FlexPilot

I continued my quest for a solution to create automated tests using seleniumRC and C# and this time I took a look at this new project called Flex Pilot.

FlexPilot is a open source testing tool that integrates with selenium,  it has a bootstrapper to make the application testable, it is able to use a selenium IDE recorder, and you can access elements using chain syntax (like accessing with xpath).

Here is what I did to start building tests with flex pilot:

• Rebuilded flexpilot: source is http://github.com/mde/flex-pilot/archives/master and excecuted build.py this will update FlexPilot.swf and FPBootstrap.swf  located under org/flex_pilot folder (place this file where the app is located)
• Copied the content from src/org/flex_pilot folder to the flex's app libs folder example C:\source\MyAppInFlex\libs
• Imported Bootstrap in the flex app

  import org.flex_pilot.FPBootstrap;

  
  
• Set FPBootstrap.flex_pilotLibPath the path on the server where FPBootstrap can find FlexPilot.swf. The Loader fetches FlexPilot.swf via a Loader class.
  
  

  FPBootstrap.flex_pilotLibPath = '/flash/org/flex_pilot/FlexPilot.swf';
  

  
  
• Initialize FPBootstrap this fetch and load FlexPilot.swf, and gives FlexPilot Flash a context to use for testing. This context is usually a reference to app’s Stage.
  
  

  <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" xmlns:local="*" creationComplete="init() 
  applicationComplete="initFlexPilot()">
  ......
  <mx:Script>
          <![CDATA[
              ...............
              import mx.utils.ObjectUtil;    
              import org.flex_pilot.FPBootstrap;
  
              ...............
              private function initFlexPilot():void
              {
                  FPBootstrap.flex_pilotLibPath = 'FlexPilot.swf';
                  FPBootstrap.init(stage);
              }
  ......

  
  
• Compiled application: mxmlc -source-path=. -source-path+=../libs MyAppInFlex.mxml -o MyAppInFlex.swf
  
• If present, remove tags like <noscript> around the flex object in you html.
  
• If the configuration is working Firebug will return “function” when calling document.getElementById('MyAppInFlex').fp_click 
• This is the “hello world” example of a test method on c#
  
  

  [TestMethod]
  public void TestMethodFlexPilot()
  {
      selenium.Open("http://localhost/testapp.html");
      selenium.RunScript("document.getElementById('MyAppInFlex').fp_type({name:'usernameTextInput', text:'Flex Pilot'})");
      selenium.RunScript("document.getElementById('MyAppInFlex').fp_click({name:'secureCheckBox'})");
  }

FlexPilot has seleniumRC client drivers available for java, phyton and rubi (no c# client driver available yet)

Setting up flex testing with selenium

Recently I was looking for tools to support automated testing for flex applications. I have a test suite in SeleniumRC and C# I was looking for options to continue using this environment. Here’s what I found:

FlashSelenium, Selenium Flex API

These two projects provides capabilities to interact with Flex UI components and web pages through selenium RC.

Selenium Flex API automatically exposes Flex APP UI and FlashSelenium  allowing us to call ActionScript methods to interact with Flex elements. Note that this approach requires us to compile our flex applications with Selenium Flex API library.

To start coding your test:

• Rebuild your Flex application with sfapi.swc add the compiler argument:  -include-libraries "..\libs\sfapi.swc"
• Include FlashSelenium.dll library in the seleniumRC test project.

To be able to run test on firefox you need to specify the browserString *firefoxproxy instead of *firefox since firefox doesn't like javascript calling flash when javascript comes from another window (the way selenium calls flash objects).

This a “hello world” example:

[TestClass]
public class MyAppInFlexTest
{
    private ISelenium selenium;
    private FlashSelenium.FlashSelenium flashApp;

    [TestInitialize()]
    public void SetupTest()
    {
        selenium = new DefaultSelenium("localhost", 4444, @"*firefoxproxy", @"http://localhost/testapp.html");
        //selenium = new DefaultSelenium("localhost", 4444, @"*iexplore", @"http://localhost/testapp.html");
        //selenium = new DefaultSelenium("localhost", 4444, @"*googlechorme", @http://localhost/testapp.html);
        selenium.Start();
        flashApp = new FlashSelenium.FlashSelenium(selenium, "MyAppInFlex");
    }

    [TestCleanup()]
    public void TeardownTest()
    {
            selenium.Stop();
    }

    [TestMethod]
    public void TestMethodFlashSelenium()
    {
        flashApp.Call("doFlexType", "usernameTextInput", "from selenium flex");
        flashApp.Call("doFlexClick", "secureCheckBox", "");
    }
}