Visual Studio Debugging and Remote Debugging

This time I would like to share some notes gathered from different sources about debugging: how visual studio handles different builds (Debug and Release) and how we can debug code that has been deployed to a remote machine.

Debug, Release Build

Like most of you know visual studio projects has some predefined build configurations: Debug and Release. Using Debug, the program is compiled with full symbolic debug information and no optimizations.

With Release the program is compiled using Optimize code and  pdb-only options. The pdb-only option does not generate the DebuggableAttribute that tells the JIT compiler that debug information is available, but generates a .pdb (program database) file to allow viewing information such as source filenames and line numbers in the application’s stacktrace.

Optimized code is harder to debug since the compiler repositions and reorganizes instructions to get a more efficient compiled code, so generated instructions may not correspond directly to the source code. Some optimizations are always performed by the compiler and others only performed when the Optimized code option is set; optimization may include things like: constant propagation, dead code elimination. .Net runs optimizations in 2 steps: one optimization run by the compiler when generating the IL code and other when running the application and transforming IL to machine code, most optimizations are left to the JIT compiler.

While reviewing more about it, it seems that optimization is separate from pdb generation so pdb-only generation shouldn’t affect performance in most scenarios and it is recommended to produce PDB files, even if you don't want to ship them with the executable.

In order to debug an application with visual studio, we require the matching pdb file: The debugger looks for the PDB files using the dll or executable name and looks for [applicationname].pdb. When .dll, executable, and PDB files are generated it creates and stores identical GUIDs in each file. The GUID is used to determine if a given PDB file matches a DLL or an executable file.

You can use DUMPBIN to see if pdb files are found for a given dll, for example:

e:\Program Files (x86)\Microsoft Visual Studio 9.0\VC>dumpbin /pdbpath:verbose 
"e:\Projects\VS 2008\RemoteDebugging\WpfApplication1\bin\Release\WpfApplication1.exe"

Microsoft (R) COFF/PE Dumper Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file e:\Projects\VS 2008\RemoteDebugging\WpfApplication1\bin\Release\WpfApplication1.exe

File Type: EXECUTABLE IMAGE
  PDB file found at 'e:\Projects\VS 2008\RemoteDebugging\WpfApplication1\bin\Release\WpfApplication1.pdb'

  Summary

        2000 .reloc
        2000 .rsrc
        2000 .text

e:\Program Files (x86)\Microsoft Visual Studio 9.0\VC>

and GUID information can be inspected using DUMPBIN as well:

e:\Program Files (x86)\Microsoft Visual Studio 9.0\VC>dumpbin /headers 
"e:\Projects\VS 2008\RemoteDebugging\WpfApplication1\bin\Release\WpfApplication1.exe"
.....

Dump of file e:\Projects\VS 2008\RemoteDebugging\WpfApplication1\bin\Release\WpfApplication1.exe
.....

Debug Directories

      Time Type       Size      RVA  Pointer
  -------- ------ -------- -------- --------
  4D97EE82 cv           6C 00003708     1908    Format: RSDS, 
  {9873ECF8-BA29-4C84-9AF5-BC54B1E6FFD4}, 1, E:\Projects\VS 2008\RemoteDebugging\WpfApplication1\obj\Release\WpfApplication1.pdb

Another interesting feature around .net debugging is configuring an executable image for debugging,  if you want to debug [application].exe, create a text file named [application].ini, in the same folder having this information:

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

This tells JIT compiler to generate tracking information and not run optimizations, making it possible for the debugger to match up MSIL and not optimizing resulting machine code.

Remote debugging

To remote debug a .net application you can use the visual studio remote debugger (msvsmon.exe),
Both machines should have access/permissions to connect to each other. Start  visual studio remote debugger on the machine running the application, this will start visual studio remote debugging monitor displaying the following information.

Msvsmon started a new server named 'Domain\user@machinename'. Waiting for new connections.
machine\user connected. 

On the machine you want to debug, start visual studio and on the attach to process windows, set Qualifier to the remote debugger server name. In this example 'Domain\user@machinename' use the mane as it appears.

xUnit.Net – Running the tests (RunAfterTestFailed Custom Attribute)

On my last post I created a Custom TestCategory Attribute and custom xUnit.Net TestClassCommand to run test by category. Now I want to create a Custom RunAfterTestFailed attribute to run a method whenever a test fails. We have the following test class

using Xunit;
using Xunit.Extensions;

namespace xUnitCustomizations
{
    [CustomTestClassCommand]
    public class TestClass
    {
        [Fact, TestCategory("Unit")]
        public void FailedTest1()
        {
            Assert.True(false);
        }

        [Fact, TestCategory("Unit")]
        public void FailedTest2()
        {
            throw new InvalidOperationException();
        }

        [RunAfterTestFailed]
        public void TestFailed()
        {
            Debug.WriteLine("Run this whenever a test fails");
        }
    }
}

We will use CustomTestClassCommandAttribute and CustomTestClassCommand previously created, and made the following changes to EnumerateTestCommands method, since we need a custom command to be able to handle errors when executing test methods:

public class CustomTestClassCommand : ITestClassCommand
{
    .....

    #region ITestClassCommand Members

    .....

    public IEnumerable<ITestCommand> EnumerateTestCommands(IMethodInfo testMethod)
    {
        string skipReason = MethodUtility.GetSkipReason(testMethod);

        if (skipReason != null)
            yield return new SkipCommand(testMethod, MethodUtility.GetDisplayName(testMethod), skipReason);
        else
            foreach (var testCommand in cmd.EnumerateTestCommands(testMethod))
                yield return new AfterTestFailedCommand(testCommand);
    }
    ....
    #endregion
}

AfterTestFailedCommand will handle calling the execution of the test method and calling the proper method whenever a test fails

public class AfterTestFailedCommand : DelegatingTestCommand
{
    public AfterTestFailedCommand(ITestCommand innerCommand)
    : base(innerCommand)
    {}

    public override MethodResult Execute(object testClass)
    {
        MethodResult result = null;
        Type testClassType = testClass.GetType();
        try
        {
            result = InnerCommand.Execute(testClass);
        }
        finally
        {
            if (!(result is PassedResult))
            {
                foreach (MethodInfo method in testClassType.GetMethods())
                    foreach (var attr in method.GetCustomAttributes(typeof(RunAfterTestFailedAttribute), true))
                        method.Invoke(testClass, null);
            }
        }
        return result;
    }
}

I liked the extensibility that xUnit.Net provides, even when there isn’t a built in solution for TestCategory and RunAfterTestFailed attributes, I was able to build it by looking at the source code, the unit tests and examples available in CodePlex (the framework has unit tests!). Hope this series of posts was helpful to get you started in migrating from MSTest to xUnit.Net and writing custom extensions for this framework.

xUnit.Net – Running the tests (TestCategory)

In my previous post I showed an example about converting a MSTest class to xUnit.Net, and now I want to provide a solution for converting MSTest TestCategory attribute to an equivalent implementation in xUnit.Net.

MSTest allowed us to run the test that belongs to an specific category, let’s see a solution on how this can be accomplished in xUnit.Net.

using Xunit;
using Xunit.Extensions;

namespace xUnitCustomizations
{
    [CustomTestClassCommand]
    public class TestClass
    {
        [Fact, TestCategory("Unit")]
        public void FastTest()
        {
            Debug.WriteLine("fast test executed");
            Assert.True(true);
        }

        [Fact, TestCategory("Integration")]
        public void SlowTest()
        {
            Thread.Sleep(5000);
            Debug.WriteLine("slow test executed");
            Assert.True(true);
        }
    }
}

Create TestCategory attribute

 

namespace Xunit.Extensions
{
    public class TestCategoryAttribute : TraitAttribute
    {
        public TestCategoryAttribute(string category)
        : base("TestCategory", category) { }
    }
    ...
}

CustomTestClassCommandAttribute attribute is used to indicate that a custom test runner will be used

public class CustomTestClassCommandAttribute : RunWithAttribute
{
    public CustomTestClassCommandAttribute() : base(typeof(CustomTestClassCommand)) { }
}

CustomTestClassCommand  is the class that implements ITestClassCommand and acts as the runner for the test fixture

public class CustomTestClassCommand : ITestClassCommand
{
    // Delegate most of the work to the existing TestClassCommand class so that we
    // can preserve any existing behavior (like supporting IUseFixture<T>).
    readonly TestClassCommand cmd = new TestClassCommand();

    #region ITestClassCommand Members
    public object ObjectUnderTest
    {
        get { return cmd.ObjectUnderTest; }
    }

    public ITypeInfo TypeUnderTest
    {
        get { return cmd.TypeUnderTest; }
        set { cmd.TypeUnderTest = value; }
    }

    public int ChooseNextTest(ICollection<IMethodInfo> testsLeftToRun)
    {
        return cmd.ChooseNextTest(testsLeftToRun);
    }

    public Exception ClassFinish()
    {
        return cmd.ClassFinish();
    }

    public Exception ClassStart()
    {
        return cmd.ClassStart();
    }

    public IEnumerable<ITestCommand> EnumerateTestCommands(IMethodInfo testMethod)
    {
        return cmd.EnumerateTestCommands(testMethod);
    }

    public bool IsTestMethod(IMethodInfo testMethod)
    {
        return cmd.IsTestMethod(testMethod);
    }

    public IEnumerable<IMethodInfo> EnumerateTestMethods()
    {
        string category;

        foreach (IMethodInfo method in cmd.EnumerateTestMethods())
        {
            category = string.Empty;
            foreach (IAttributeInfo attr in method.GetCustomAttributes(typeof(TestCategoryAttribute)))
                category = attr.GetPropertyValue<string>("Value");

            if (category.ToLower().Contains("unit")) // We can make this configurable
                yield return method;
        }
    }
    #endregion
}

The Method public IEnumerable<IMethodInfo> EnumerateTestMethods() filters the tests methods by TestCategory's attribute Value, note that we can make this configurable so we can configure our CI server to run unit test as soon as there are changes in the repository to provide quick feedback and schedule (e.g. once a day) the execution slower test like integration or Web UI test.

xUnit.Net – Running the tests (ClassInitialize – ClassCleanup)

I started using xUnit.Net few weeks ago. My first question was how to do the things I was used to do in other testing frameworks like MSTest or NUnit, specially when using these frameworks not only for unit testing but for higher level tests like Selenium RC web tests. So far, the framework seems to be very good and extensible.

I am going to show some scenarios I have run into converting MSTest to xUnit.Net, having the following class

using Microsoft.VisualStudio.TestTools.UnitTesting;
[TestClass]
public class AdminPageTest
{
    static SeleniumWebTestContext test = new SeleniumWebTestContext();

    [ClassInitialize()]
    public static void ClassInitialize() 
    {
        new LoginPage(test).LoginUser(“maria”, “******”);
    }

    [ClassCleanup()]
    public static void ClassCleanup() 
    {
        new AdminPage(test.Driver).Logout();
        test.StopWebTestContext();
    }

    [TestCategory("WebTest"), TestMethod]
    public void TestViewMyProfile()
    {
        var profileWindow = new AdminPage(test.Driver).SelectMyProfile();
        Assert.IsTrue(profileWindow.CheckFirstName(“Maria”));
        profileWindow.Close();
    }

    [TestCategory("WebTest"), TestMethod]
    public void TestAdminSearchUser()
    {
        var userWindow = new AdminPage(test.Driver).SelectUserManagement();
        userWindow.Search("Marcano Maria");
        Assert.IsTrue(adminTab.VerifyEmail("my-email@domain.com"));
    }
}

Note that SeleniumWebTestContext is holding information about Selenium RC and starts the server.

ClassInitialize – ClassCleanup / IUseFixture<T>

Sometimes we require sharing information among the methods defined in a class, for example in web tests we want to share the logged in the user information, execute different actions and validations and log out at the end:

using Xunit;
public class AdminPageTest : IUseFixture<LoggedUserContext>
{
    SeleniumWebTestContext test;
    public AdminPageTest()
    {  
    }

    public void SetFixture(LoggedUserContext usrContext)
    {
        test = usrContext.Test;
    }
    ...
}

and the LoggedUserContext class will look like this:

public class LoggedUserContext : IDisposable
{    
    public SeleniumWebTestContext Test;    
    public LoggedUserContext()    
    {        
        Test = new SeleniumWebTestContext();        
        new LoginPage(Test).LoginUser(“maria”, “******”);    
    }    

    public void Dispose()    
    {       
        new AdminPage(Test.Driver).Logout();
        Test.StopWebTestContext();
    }
}

CruiseControl.Net and MSTest

I wanted to put together some notes about CruiseControl.Net and MSTest to have it as reference:

First, to be able to generate reports the following element is required on ccnet.config file.

<xmllogger />     

this element is used to create the log files read by the CruiseControl.NET web page, if it isn't defined the web page will throw the following error:

Exception Message
Request processing has failed on the remote server: Unable to find Log Publisher for project so can't find log file.

To view the MSTest report, we need to add the result file in ccnet.config to merge the content in the build report, for example :

..................
<exec>
  <executable>C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe</executable>        
  <baseDirectory>.\</baseDirectory>
  <buildArgs> /testcontainer:SeleniumSearch\bin\Debug\SeleniumSearch.dll /resultsfile:Results\results1.trx</buildArgs>
</exec>     
tasks>
<publishers>
<merge>
  <files>
    <file>.\Results\results1.trx</file>
  </files>
</merge> 
<xmllogger />
</publishers>
..................

Include the MSTest plugin to generate the report on the web page, for example dashboard.config file

..................
<buildPlugins>
  <buildReportBuildPlugin>
    <xslFileNames>
      <xslFile>xsl\header.xsl</xslFile>
      <xslFile>xsl\modifications.xsl</xslFile>
      <xslFile>xsl\MsTestSummary2008.xsl</xslFile>          
    </xslFileNames>
  </buildReportBuildPlugin>
  <buildLogBuildPlugin />
  <xslReportBuildPlugin description="MSTest2008 Report" actionName="MSTESTReport" xslFileName="xsl\MsTestReport2008.xsl"/>             
</buildPlugins>
..................

And include the xls file for MSTest report on ccservice.exe.config/ccnet.exe.config

..................
<!-- Specifies the stylesheets that are used to transform the build results when using the EmailPublisher -->
<xslFiles>
    <file name="xsl\header.xsl"/>
    <file name="xsl\compile.xsl"/>
    <file name="xsl\unittests.xsl"/>
    <file name="xsl\MsTestSummary2008.xsl"/>        
    <file name="xsl\fit.xsl"/>
    <file name="xsl\modifications.xsl"/>
    <file name="xsl\fxcop-summary_1_36.xsl"/>
</xslFiles>
..................

The xls used on the above example is the one included in ccnet installation. I tested the report generation running tests on VS 2010 and VS 2008 and both reports are generated correctly. The only thing that caused issues was having MSTest on the path windows environment variables on a computer that has both VS 2010 an VS 2008

Also, if MSTest needs to connect to external resources, like accessing remote files or connecting to SQL Server, make sure the account that runs CruiseControl.Net Server windows service (by default is Local System Account) has access to those resources.

CruiseControl.Net driving Selenium Tests

This is how you can quickly get CruiseControl.Net running and driving your selenium test – in 6 steps:

1. Donwload & Install CruiseControl.Net
2. Start “CruiseControl.NET Server“ Windows Service
3. Download SeleniumTest sample project and extract it on a folder C:\Projects\.
4. Edit CruiseControl.NET configuration file (C:\Program Files (x86)\CruiseControl.NET\server\ccnet.config) and replace it with the information in C:\Projects\SeleniumTest\CruiseControl.Net\ccnet.config. Modify it with proper locations of MSBuild.exe, MsBuild.dll and MSTest.exe 
5. Start selenium server by running C:\Projects\SeleniumTest\SeleniumServerPort4441.bat.
6. Open http://localhost/ccnet and you will se the project SeleniumSearchTest1 click on Force. It should build the project and start the tests.

The tests will run using chrome, you can change this parameter in C:\Projects\SeleniumTest\SeleniumSearch\SeleniumSearchTest.cs and use "*iehta" for Internet Explorer or  "*firefoxproxy" for Firefox.

// "*iehta" IE -  "*firefoxproxy" FF - *googlechrome
selenium = new DefaultSelenium("localhost", 4441, "*googlechrome", "http://seleniumhq.org/"); 

This is how the ccnet.config file looks

<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
  <project>
    <name>SeleniumSearchTest1</name>
    <triggers>
      <scheduleTrigger time="23:30" buildCondition="ForceBuild">
        <weekDays>
            <weekDay>Monday</weekDay>
            <weekDay>Tuesday</weekDay>
            <weekDay>Wednesday</weekDay>
            <weekDay>Thursday</weekDay>
            <weekDay>Friday</weekDay>
        </weekDays>
      </scheduleTrigger>
    </triggers>
    <workingDirectory>C:\Projects\SeleniumTest</workingDirectory>
    <tasks>
      <msbuild>
        <executable>C:\Windows\Microsoft.NET\Framework\v3.5\MSBuild.exe</executable>
        <workingDirectory>C:\Projects\SeleniumTest</workingDirectory>
        <projectFile>SeleniumTest.sln</projectFile>
        <buildArgs>/noconsolelogger /p:Configuration=Debug /v:diag</buildArgs>
        <targets>Build</targets>
        <timeout>900</timeout>
        <logger>C:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
      </msbuild>
      <exec>
        <executable>DelResults.bat</executable>
        <baseDirectory>.\</baseDirectory>
      </exec>     
      <exec>
        <executable>C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe</executable>        
        <baseDirectory>.\</baseDirectory>
        <buildArgs> /testcontainer:SeleniumSearch\bin\Debug\SeleniumSearch.dll /resultsfile:Results\results1.xml</buildArgs>
      </exec>     
    </tasks>
    <publishers>
      <merge>
        <files>
          <file>.\Results\results1.xml</file>
        </files>
      </merge>
      <xmllogger />
    </publishers>
  </project>
</cruisecontrol>

This example applies for VS 2008 Professional, ccnet 1.5.7256.1 and Selenium RC 1.0.3

Links:

http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET

http://github.com/downloads/mariangemarcano/seleniumtest/SeleniumTest-Code.zip

Selenium Testing with FlexPilot

I continued my quest for a solution to create automated tests using seleniumRC and C# and this time I took a look at this new project called Flex Pilot.

FlexPilot is a open source testing tool that integrates with selenium,  it has a bootstrapper to make the application testable, it is able to use a selenium IDE recorder, and you can access elements using chain syntax (like accessing with xpath).

Here is what I did to start building tests with flex pilot:

• Rebuilded flexpilot: source is http://github.com/mde/flex-pilot/archives/master and excecuted build.py this will update FlexPilot.swf and FPBootstrap.swf  located under org/flex_pilot folder (place this file where the app is located)
• Copied the content from src/org/flex_pilot folder to the flex's app libs folder example C:\source\MyAppInFlex\libs
• Imported Bootstrap in the flex app

  import org.flex_pilot.FPBootstrap;

  
  
• Set FPBootstrap.flex_pilotLibPath the path on the server where FPBootstrap can find FlexPilot.swf. The Loader fetches FlexPilot.swf via a Loader class.
  
  

  FPBootstrap.flex_pilotLibPath = '/flash/org/flex_pilot/FlexPilot.swf';
  

  
  
• Initialize FPBootstrap this fetch and load FlexPilot.swf, and gives FlexPilot Flash a context to use for testing. This context is usually a reference to app’s Stage.
  
  

  <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" xmlns:local="*" creationComplete="init() 
  applicationComplete="initFlexPilot()">
  ......
  <mx:Script>
          <![CDATA[
              ...............
              import mx.utils.ObjectUtil;    
              import org.flex_pilot.FPBootstrap;
  
              ...............
              private function initFlexPilot():void
              {
                  FPBootstrap.flex_pilotLibPath = 'FlexPilot.swf';
                  FPBootstrap.init(stage);
              }
  ......

  
  
• Compiled application: mxmlc -source-path=. -source-path+=../libs MyAppInFlex.mxml -o MyAppInFlex.swf
  
• If present, remove tags like <noscript> around the flex object in you html.
  
• If the configuration is working Firebug will return “function” when calling document.getElementById('MyAppInFlex').fp_click 
• This is the “hello world” example of a test method on c#
  
  

  [TestMethod]
  public void TestMethodFlexPilot()
  {
      selenium.Open("http://localhost/testapp.html");
      selenium.RunScript("document.getElementById('MyAppInFlex').fp_type({name:'usernameTextInput', text:'Flex Pilot'})");
      selenium.RunScript("document.getElementById('MyAppInFlex').fp_click({name:'secureCheckBox'})");
  }

FlexPilot has seleniumRC client drivers available for java, phyton and rubi (no c# client driver available yet)

Setting up flex testing with selenium

Recently I was looking for tools to support automated testing for flex applications. I have a test suite in SeleniumRC and C# I was looking for options to continue using this environment. Here’s what I found:

FlashSelenium, Selenium Flex API

These two projects provides capabilities to interact with Flex UI components and web pages through selenium RC.

Selenium Flex API automatically exposes Flex APP UI and FlashSelenium  allowing us to call ActionScript methods to interact with Flex elements. Note that this approach requires us to compile our flex applications with Selenium Flex API library.

To start coding your test:

• Rebuild your Flex application with sfapi.swc add the compiler argument:  -include-libraries "..\libs\sfapi.swc"
• Include FlashSelenium.dll library in the seleniumRC test project.

To be able to run test on firefox you need to specify the browserString *firefoxproxy instead of *firefox since firefox doesn't like javascript calling flash when javascript comes from another window (the way selenium calls flash objects).

This a “hello world” example:

[TestClass]
public class MyAppInFlexTest
{
    private ISelenium selenium;
    private FlashSelenium.FlashSelenium flashApp;

    [TestInitialize()]
    public void SetupTest()
    {
        selenium = new DefaultSelenium("localhost", 4444, @"*firefoxproxy", @"http://localhost/testapp.html");
        //selenium = new DefaultSelenium("localhost", 4444, @"*iexplore", @"http://localhost/testapp.html");
        //selenium = new DefaultSelenium("localhost", 4444, @"*googlechorme", @http://localhost/testapp.html);
        selenium.Start();
        flashApp = new FlashSelenium.FlashSelenium(selenium, "MyAppInFlex");
    }

    [TestCleanup()]
    public void TeardownTest()
    {
            selenium.Stop();
    }

    [TestMethod]
    public void TestMethodFlashSelenium()
    {
        flashApp.Call("doFlexType", "usernameTextInput", "from selenium flex");
        flashApp.Call("doFlexClick", "secureCheckBox", "");
    }
}