Silverlight Unit Test Framework: Breaking changes from Beta 1 to Release Candidate

Update your test projects to the first Silverlight Unit Test Framework update in half a year, it should take two minutes to update and get the awesome new features and functionality.

The improvements and changes made to the surface area of the test framework and runtime are a good thing. I've organized the high-level changes into the following sections: Highlights and important notes, new functionality, for test developers, test harness changes, and miscellaneous notes.

Although it looks and feels like the previous release we did back in Vegas, this one has more extension points and has reduced the learning curve of the complex harness types by making the programming interfaces more concise & easier to discover.

I hope that you find this unit test framework release for the developer-targeted Silverlight 2 Release Candidate to be a nice step forward. The developer release candidate should be complete now.

Go ahead and fire up Reflector, the simplified namespaces are easy to look at. That'd be more interesting than reading this post - almost as exciting as watching real-time stock quotes today. Oh, and if you've done any cool customizations, you'll want to check out the bottom, boring part of this post.

Download (MSDN Code Gallery) the updated test bits today.

Happy Silverlight unit testing!
- Jeff

Highlights & important notes

Simplified, improved APIs and developer experience

Less is more. Fewer namespaces, fewer assemblies, fewer framework types, more discoverable with IntelliSense. For most test development tasks, you should only need to reference two namespaces:

• Microsoft.VisualStudio.TestTools.UnitTesting: For desktop unit test framework compatibility, contains the Visual Studio unit test framework metadata.
• Microsoft.Silverlight.Testing: Contains the UnitTestSystem (used to start the test project at runtime), as well as all the other custom attributes and types that are needed for developing tests. Includes the base test classes for working with the presentation system through a "test panel."

From 3 to 2 assemblies

The Microsoft.Silverlight.Testing.dll and Microsoft.Silverlight.Testing.Framework.dll assemblies have been merged into a single assembly, Microsoft.Silverlight.Testing.dll.

Test projects should remove any references to Microsoft.Silverlight.TestingFramework.dll. Also, assembly file versions have changed to 2.0.20929.1929 from 1.0.0.0. A simple rebuild of your test projects is all that is required.

The TestSurface functionality has been renamed

The TestPanel is a special container (formerly known as the TestSurface); when you access the panel during a test case, the children of the panel are automatically cleared after the TestMethod executes. It makes testing with controls and shapes quick and easy.

If you make use of the test surface, you will need to modify your tests ever so slightly. This only impacts tests that derive from SilverlightTest and include a call to the exposed Silverlight property.

Before:

using System;
using Microsoft.Silverlight.Testing;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Windows.Controls;

namespace Test
{
    [TestClass]
    public class TestPanelTest : PresentationTest
    {
        [TestMethod]
        public void Bug484164()
        {
            Button b = new Button();
            Silverlight.TestSurface.Children.Add(b);
        }
    }
}

After:

using System;
using Microsoft.Silverlight.Testing;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Windows.Controls;

namespace Test
{
    [TestClass]
    public class TestPanelTest : PresentationTest
    {
        [TestMethod]
        public void Bug484164()
        {
            Button b = new Button();
            TestPanel.Children.Add(b);
        }
    }
}

Simple asynchronous enqueue method name changes

The Enqueue* methods that are present on SilverlightTest are now found in the WorkItemTest class. SilverlightTest still inherits this functionality. The following methods are exposed by WorkItemTest and are similar to the earlier names, no longer using custom delegates:

• Enqueue(Action), EnqueueCallback(Action)
• EnqueueConditional(Func<bool>)
• EnqueueSleep(int)
• EnqueueTestComplete()
• EnqueueWorkItem(IWorkItem)
• Sleep(int, Action)
• TestComplete()

First debugger-attached run notes

The first time that you run a Silverlight test project with these new bits, you may receive a FileNotFoundException depending on the exception handling settings you're using in Visual Studio. This is an artifact of the move to using the IsolatedStorage system for optional settings.

This is not an unhandled exception, but if you are setup to break on first chance CLR exceptions, you might think that is what is happening here. Just press F5 to continue the test run.

New functionality

There are additional features built into the framework now, which will be documented in detail in coming blog posts. Since we're sharing the same bits we use on our team, these features are more specific than a simple unit test harness and are totally unsupported. But they're there none the less!

• TagExpressions
• Enhanced interactive results explorer
• BugAttribute
• Visual Studio Team Test log format output provider
• Test service provider implementation
• Dynamic test methods
• Dynamic test method attributes

For test developers

New runtime dependencies: System.Core, System.Xml.Linq

With the RC0 release of the Silverlight 2 runtime, the System.Core runtime is built in to the platform. However, System.Xml.Linq will need to be included with your test projects, since the Visual Studio log provider requires it.

Updated list of framework assemblies that the test framework is dependent on:

• mscorlib
  
• System
  
• System.Core
  
• System.Net
  
• System.Windows
  
• System.Windows.Browser
  
• System.Xml
  
• System.Xml.Linq

Custom delegate types removed. Please use Action and Func<T> objects.

The original release used custom delegates named VoidDelegate, ConditionalDelegate, and GenericDelegate. Please instead use the standard framework's Action, Action<T>, and Func<T> types.

TestPanel has been changed from a StackPanel to a Grid

The TestPanel (Silverlight.TestSurface before) is now a Grid by default. It was a StackPanel in the original release. To enable namescope testing properly, the TestPage is now instantiated using XamlReader.

UnitTestSystem changes, CreateTestPage no longer requires casting

Although the test framework did not need to be updated for Beta 2, the Beta 2 runtime did require that the UnitTestSystem.CreateTestPage() result be cast to UIElement. The method signature has been updated to no longer require this explicit cast.

Several of the static methods of UnitTestSystem are now public, and the type itself can be instantiated. These changes make it easier to customize the settings used when initializing the system, swapping out the TestService, TestHarness, LogProviders, and other components.

Known parameters and settings

There are a number of settings that can be stored in the TestHarnessSettings.Parameters string dictionary, and are recognized during a unit test run. Of interest are the "tagExpression", "computerName", "userName", "enablePluginResize", and "testColumnWidth."

A number of these address specific concerns, workaround needs, or logging fields that have been brought to my attention.

Exception handling changes

A simple fix, there is no longer a Debugger.Break() statement in the exception handlers for test methods.

New test class hierarchy

Although SilverlightTest is still supported, the unique functionality of the Silverlight Unit Test Framework has been split out into multiple test classes that you are free to inherit from.

When inheriting from any of the four types, do realize that you are breaking the built-in compatibility with the desktop Visual Studio test framework, and it may require some creative juices to cross-compile or otherwise share tests between other projects.

The idea of the test classes is to promote a "bite off as much as you need" story for test case development. All these types are found in the standard Microsoft.Silverlight.Testing namespace.

No need to make changes if you're already inheriting from SilverlightTest.

Browser property removed from SilverlightTest

The property has been removed and is now available as the static helper class Browser.

Test harness changes

If you dug in with Reflector to make your own customizations, here are some public types used in the construction of the test harness implementation that may affect your extensions:

UnitTestProvider changes: The interfaces have been updated to allow instantiated IAssembly types to store a reference to the underlying ITestHarness used to request the unit test provider. There is no longer a singleton providing access to the UnitTestHarness. You may need to update your IUnitTestProvider, IAssembly, and ITestMethod implementations.

Asynchronous test dispatcher: The RunMethodManager type has been renamed to RunDispatcher. This dispatcher that handles driving test run execution forward and running asynchronous work items can also now be restarted after a test run is complete.

Many associated types and interfaces have had small name changes to help clarify their purpose, and shorten unhelpful long names. The AsyncTestTask type has been removed as it is redundant with the WorkItem base class.

UserInterfaceContext removed: The UI context concept and implementation was not simple enough to prove useful, it has been removed.

Logging changes: The initial implementation relied on a complex type hierarchy that has been replaced with a simple LogMessage type. More like the standard .NET framework's EventLog handling, the new LogMessage can be decorated with any type of additional reference, a type category, and other properties that permit the same functionality. LogMessageFactory and LogMessageWriter are new types to simplify working with the newer logging system.

The ParameterizedLogProvider is now available through the ITestSettingsLogProvider interface. The LogManager and related interfaces have been removed.

Earlier custom providers will need significant updates to work with the release candidate build.

TestOutcome enum replaces TestResult: The TestResult type now matches the name and enum values of the desktop unit test framework's TestOutcome. The unit test engine best understands the Passed state, but the others are provided for use by developers as it makes sense. The VisualStudioLogProvider implementation counts and tracks all legitimate TestOutcome values for reporting purposes.

BaseTestHarness changes:

• The BaseTestHarness base class has been renamed to TestHarness and moved into a different namespace.
• The TestHarnessState type now allows for tracking total scenarios and failing scenarios.
• TestHarness is able to hook into the reporting TestService provider, if present. It exposes a public TestService property.
• LogProviders exposes the collection of active log providers.
• The test harness implementation now has a WriteLogFile method that can be used to optionally store a log file when used with the test service.
• The harness also exposes a Publishing event that is fired right before the complete events to offer your log providers the opportunity to subscribe and save results before the harness completes.

Web interface changes: Many changes were made to the coupling and implementation of the various HTML log providers and objects. A strongly typed, richer HTML control library is used for many of the interactive HTML DOM elements.

Closing note

Overall the process of updating existing test projects to this framework should be very simple. Let me know how you like the new bits, I should be posting a lot more on this subject in the coming weeks.

Also do post a comment if you run into any issues with your existing projects. I've also enabled a few more tabs on the MSDN Code Gallery site for the framework.