← About Jeff

Visual Studio unit testing intro

August 16, 2008

Visual Studio's unit testing framework isn't widely covered in books or in other places, and this makes it tougher for teams to adopt strict test guidelines for their developers. At the end of the day, if you aren't unit testing from day one of your project, you're in for a world of hurt down the line.

If you've new to Visual Studio unit testing, or just wanting to learn about Silverlight unit testing, you may feel a little lost. Here's some information that just scratches the surface of the Visual Studio unit testing world.

So here are some of the questions people may have, let me know if there are others that you have! If you're already familiar with unit testing this will all be redundant.

What makes up a unit test project?

Most unit tests are simple class libraries written in your language of choice. They reference your production application's projects or built assemblies, and also the unit test framework.

Metadata decorates the tests, and the tests themselves typically include calls to the test framework's assertion classes.

Where do I go to get started?

Must-read resources:

Patterns and examples:

  • The Composite WPF guidance by Microsoft's Patterns & Practices includes tests
  • TypeMock has a useful, general article about unit testing patterns. Note that their examples aren't using the Visual Studio metadata, but instead nUnit's... but it's pretty much the same.

Videos:

For testing desktop applications, class libraries, and other code built for the full .NET framework, you're in for a treat: Visual Studio 2008 Professional and other SKUs put great built in unit test support right inside your development environment. You'll always be a click away from running a test or seeing where you stand.

What's a unit test?

  • Simple.
  • Quick to run.
  • Independent and self-contained: Should not rely on any previous test results or any particular execution order.
  • Doesn't cross any unit test boundaries (as defined by Phil Haack)
  • Easy to maintain over time. Documented. Not archaic.
  • Lacking variability: a valid unit test should always consistently pass.
  • Positive or negative: They can validate exceptions and error conditions as well as expected results and states.

What isn't a unit test?

  • A test that validates integration (your database and your class library).
  • A test that interacts with UI, network resources, or the file system.
  • Something that takes a long while to run.
  • An end-to-end test of a user scenario.
  • A performance, stress, load test, security, or other advanced test.
  • Complex.

What other kinds of tests are written?

There are integration tests, end-to-end tests, performance and stress tests, and many, many more. But at Microsoft I like to think of tests in 3 sets. In some cases it is possible for all the tests in a product to build upon a 'unit test framework'.

Unit: "True" unit tests. Quick, small, efficient, very useful to always run. Independent.

Developer: Unit tests plus tests that exercise simple scenarios and conditions, such as the control framework in WPF, or performing a super simple request. Often these reach outside of the typical "unit test boundaries," but are still quick to run and useful.

Developers are typically required to run all of the developer tests on their machine when preparing a check-in to validate the state of the code.

All tests: The set of all developer tests and everything that the test development team has created. Sometimes reaches into the tens of thousands of scenarios and cases at the end of a product cycle once a matrix of platforms and configurations is taken into consideration.

Developers and testers work together to make sure that all tests are run regularly, and especially when working on approaching a major product milestone, release, or servicing activity.

Why would you use a unit test framework to create more than unit tests?

  • Consistency.
  • Can usually run many integration and scenario tests in the same test runs.
  • Run integration tests on many platforms in an automated fashion using test controllers and harnesses.
  • The definition of a unit test is pretty academic. At the end of the day, I want to get my job done efficiently and without needing to continually learn new systems. So doing more than true "unit testing" is cool with me.
  • Allow your developers and test developers to share experience, test cases, and help each other succeed.

Why does the Silverlight unit testing framework have tests that look just like the full desktop framework's? Are they the same?

When building out the Silverlight unit test system that runs tests inside of your web browser, we made the call to reuse Visual Studio's unit test framework metadata. This is a great win, for a few reasons:

  • Reuse of existing tests: many simple API tests can be run as they are.
  • Many .NET developers already know Visual Studio's unit test framework, or nUnit. No need to waste time learning the basics of another system.
  • If you're new to Microsoft's unit testing framework in general and starting with Silverlight, you'll walk away with the information you need to start unit testing in Visual Studio as well.
  • Why re-invent the wheel?

I like the Silverlight test framework, but how can I make sure that my tests can be ported to a desktop Windows Forms, class library, or WPF project?

Make sure that your unit tests are actually small, independent, simple, and not dependent on the web browser, Silverlight itself, or the more advanced "beyond unit testing" capabilities that are also present in Microsoft.Silverlight.Testing. If you inherit from the "SilverlightTest" base class, you probably have a test that won't port to the desktop .NET framework.

We've worked hard to provide a subset of the full Visual Studio unit test metadata, but here's a quick guide for you:

OK to use in both worlds:

  • All the metadata attributes including TestClass, TestMethod, ClassInitialize, ClassCleanup, TestInitialize, TestCleanup, AssemblyInitialize, AssemblyCleanup, Ignore.
  • StringAssert, Assert, CollectionAssert static types.

Desktop features that will not work with the Silverlight test framework today (or are no-ops):

  • Data-driven tests
  • TestContext
  • File-based assertions and accessing the file system
  • Using WorkItem or TestProperty attributes
  • Web/ASP.NET test functionality.
  • DeploymentItem.

Silverlight test features that will not port to the desktop:

  • Using Silverlight or web browser features.
  • Inheriting from the SilverlightTest base class.
  • Using any of the Silverlight "TestSurface" or TestPanel features exposed by the SilverlightTest base class.
  • Using the "Asynchronous" features in the framework, including the Enqueue and test work item / test task features.
  • Using any of the attributes defined in the Microsoft.Silverlight.Testing namespace, such as AsynchronousAttribute, BugAttribute, and ExclusiveAttribute.
  • Writing your own UnitTestProvider, your own Harness, or your own UnitTestHarness
  • Writing your own test tasks and work items.

Hope this helps get some of you started!