In my previous blog entry I wrote about how to extend Visual Studio in order to get transactional tracing for the applications that I develop in my Visual Studio Solution. This could be a console application, a WinForms or WPF Rich Client Application, an ASP.NET Web Application or a SharePoint Extension (WebPart, Custom List, …)

What about Unit Testing?

Visual Studio Team System Test Edition offers a set of different test types, e.g.: Unit-, Manual-, Web- and Load. Those tests can be executed via MSBuild and are therefore suitable to be executed by Team Foundation Server to verify the correctness of every build that comes out of your Continuous Integration Environment.

Lets focus on Unit Testing: If I - as a developer - start coding my unit tests I execute them locally before I checkin my code changes. Visual Studio allows me executing the unit tests from within the IDE showing me the status of my tests, error messages and even the execution time. Here is a sample test result view:

Unit Test Result View in Visual Studio Team System

What are the limitations of Unit Testing in Visual Studio?

My unit test class implements 3 test cases that all call a Web Service hosted on a test machine in my development environment. The test machine runs the latest build of the application that I am working on and the unit tests make sure that the Web Service calls are working as expected - at least from a functional perspective.

With Unit Test support as it is right now I run into the following limitations:

• I don’t see how much time is actually spend in my ClassInitialize and ClassCleanup methods. I could have introduced code changes that takes several seconds to execute. Several seconds might not be long for my 3 tests that I run in my test suite. But what if my colleagues that work on the other hundreds or thousands of unit test suites do the same thing without knowing it? This would cause the overall unit test execution cycle to take very long and nobody would know why that is because the current set of tools doesn’t tell us
• Code coverage can help me to find out which methods have actually been called by my unit tests. But I run into two issues here: I don’t know which test method actually covered which code (this seems to be a feature that will be addressed in the next version of Visual Studio). The other limitation is that code coverage only works for the code that is exectuted in the unit test container (VSTestHost.exe). In my case I call a remote web service which is not automatically covered by code coverage
• With a Unit Test I can only verify the functional correctness of the components that I am testing. I cannot verify if those components would also run well outside the unit test environment or whether the components are configured correctly to use them in the most efficient way

Transactional Tracing with dynaTrace

dynaTrace offers transactional tracing with its unique PurePath technology. It allows tracing every single unit test transaction/execution from the actual run method in the unit test runner (Microsoft Unit Test or NUnit) through all relevant methods that are executed by the test and is even able to trace the transaction across runtime and technology (Java and .NET) boundaries. Along the PurePath information like execution times, method arguments, exceptions, logging messages and database statements are captured.

How to integrate transactional tracing for Visual Studio Unit Test Executions?

I extended the Add-In I created in my previous post to enable injecting the dynaTrace Agent into the VSTestHost process that is launched by Visual Studio when executing any type of test. By doing that I get PurePaths for every single test that is executed. The PurePath not only includes the test method execution itself but also includes initialization and cleanup methods, gives me timing on every single method along the execution path and it also traces the calls across runtime and technology boundaries. The following illustration shows the same 3 unit tests executed with transactional tracing enabled:

Transactional Tracing for Unit Tests executed by Visual Studio

It seems that there are several things that we can see from the PurePath of my TestAccountInformation test case. Expanding the PurePath for this test case we see that there is a significant amount of time spent in the ClassInitialize method. Drilling into it we see that exceptions have been thrown because the XmlSerializers have not been deployed with the Web Service Proxies - this resulting in an in-memory compilation of the Serializers that are needed.

Insight into ClassInitializers - reveiling configuration/deployment problem

Furthermore I can drill into the actual web service implementations on the test server that is hosting those services and figuring out what those services are doing and why they take a certain amount of time.

Unit Test PurePath showing trace to remote Web Service implementation

The two examples above give you an indication about how much more we can get out of our unit tests. Analyzing the remoting or database behaviour of the components that we test can help us understand if the current implementation will actually scale in a production environment.

Looking at memory behavior will get us insight about memory consumption per component or use case that we test - allowing us to make predictions about the environment we need in order to host our application and run it with the predicted user load.

And finally - looking at timings - not only on the top test method level but on the individual methods that we call will help us to address performance issues from the beginning - allowing our unit test executions to complete within our continuous integration cycles as well as fixing performance issues before they make it to the test - staging or production environment.

Conclusion

The ability to do transactional tracing for Unit Tests executed from within the development environment enables developers to better understand what is going on in the code that they are testing, in the frameworks that they base their app on and in the services that their code relies on. Visual Studio offers these extension mechnisms and dynaTrace - with the PurePath technology - enables capturing the necessary data.

ASP.NET Page Lifecycle methods for instance that are called in the different stages of page processing are a good example where debugging definitely helps you to understand which methods of your page are called in while ASP.NET processes the page request.

Limitations of debugging

There are however limitations with debugging that makes it a bit hard to analyze the exact workflow of an application when based on complex frameworks and optionally calling out to other (local or remote) components:

• While debugging, you only see the current stack trace but you don’t see the complete transaction trace including ALL different execution paths
• You can normally only debug through your own code. Its hard to figure out what is going on in the frameworks or libraries that are used
• Calling external (local or remote) services are treated like a blackbox - unless you have the chance to install a remote debugger and as long as the remote technology can be handled by Visual Studio (Native C++, .NET)
• Debugging changes the runtime behavior of your application and may even influence the execution paths, e.g.: running into timeouts, …

Last week at the PDC2008 in Los Angeles - Microsoft gave a preview of Visual Studio 2010 and a new feature that they call Historical Debugging. This feature will adress the first bullet item on my list but we still need to wait till the next release of Visual Studio - and - what about the other 3 points?

Transactional Tracing with dynaTrace

dynaTrace offers transactional tracing with its unique PurePath technology. It allows tracing every single transaction from where it enters your application, e.g.: ASP.NET Page Handler through all relevant methods that are executed within the same transaction context and is even able to trace the transaction across runtime and technology (Java and .NET) boundaries. Along the PurePath information like execution times, method arguments, exceptions, logging messages and database statements are captured.

How to integrate transactional tracing with Visual Studio?

As a developer I want to be able to launch my application as I am used to - from within Visual Studio by hitting Ctrl+F5 to run the app or F5 to debug it. In order to extend this capability with transactional tracing for my app I created a Visual Studio Add-In that launches the active project of the open solution with dynaTrace support. Binding the new command to the shortcut Ctrl-Shift-D now allows me to get transactional tracing support from within Visual Studio with a simple keystroke.

Launch Web Project with dynaTrace support to trace transactions

Transactional tracing now allows me to really see what is going on within my application. I can explore the PurePath to analyze how the ASP.NET Page is actually executed:

PurePath of an ASP.NET Page Request containing a GridView

It also allows me to look at the captured data from different perspectives, like looking at all database statements that have been executed in context of a single transaction:

Database statements executed by a single ASP.NET Web Request

Or even follow a call to an external service and see what was executed within that service - no matter whether it is implemented in .NET or Java:

Transaction Trace across runtime boundaries and technologies

Conclusion

The ability to do transactional tracing triggered from within the development environment enables developers to better understand what is going on in their code, in the frameworks that they base their app on and in the services that their code relies on. Visual Studio offers these extension mechnisms and dynaTrace - with the PurePath technology - enables capturing the necessary data.

Exceptions are a great way to communicate error state back to the caller. The exception class basically defines the type of error, e.g.: AccessDenied that happened. The detailed message then explains the exact reason for the problem, e.g.: “Your user has no rights to access file xyz”.

Documentation of frameworks usually lists all types of exceptions that could potentially be thrown - allowing the caller to implement try-catch routines to handle those exceptions that make sense in order to implement program logic.

“The Good” thing about exceptions

The above described usage scenarios are a good thing about exceptions. Getting a detailed explanation in case a problem happened can make it easy for you to address the issue or even allows you to publish this error to the user interface. Using exceptions to control program logic is also a way to go (but still keep in mind the good old return values )

“The Evil” thing about exceptions

In the past weeks I came across different scenarios where the above explained usage scenarios didn’t work. Here is why:

Scenario a) The Web Application returned a 403 Access Forbidden

The application used a 3rd party web control to display a questionnaire form. Some users of the web application could use it fine - others always got an UnauthorizedAccessException thrown by the Page_Load method of the 3rd party web control leading to a 403 HTTP Error. The exception had no detailed error message specified so it was impossible to find the exact root cause. 
Using dynaTrace it was possible to find the root cause. It turned out that the web control accessed a local configuration file on the web server. Opening that file threw an AccessViolationException with a detailed error message that this user just had no access rights. The original message unfortunately was never thrown up to the application itself. Knowing the original exception helped to understand and address the problem.

Scenario b) The Application allowed restricted access to an unauthorized user

The application called into an authorization framework that would throw different kinds of exceptions in case the user did not have the appropriate access rights. The application would catch all different exceptions and implement different program logic to handle the situation.
In order to implement a logging facility, a new layer was introduced that was supposed to catch the exceptions thrown from the authorization framework, log the calls, and then rethrow the exception. Due to a bug in this layer, several exceptions indicating an authorization problem never made it to the actual caller. The application therefore continued the normal application flow as if the user would have had access.

Solution

Getting visibility into your application and all its frameworks it depends on enables you to better understand how to use your 3rd party frameworks. It also allows you to diagnose errors down to the root cause in case you don’t get the detailed explanation that you need.

Analyzing Exceptions in the context of a single transaction

The above image shows how dynaTrace tracks every exception that happens in your system. It also shows it in the context of a single transaction (PurePath) with all its contextual information, e.g: method parameters, …

Conclusion

Relying on frameworks is a mandatory thing these days. But not knowing what’s actually going on in those frameworks can lead to the above described side effects. With a solution like dynaTrace it is possible to analyze all exceptions that occurred within the application and not only within the custom application code.