The one where we reverse engineered Microsoft’s C++ Unit Test Framework (Part 3) – Exceptions

Again, if you haven’t already done so I suggest reading through Part 1 and Part 2 of the series. I’ll do a quick TL; DR; recap, but it may not do it justice.

What feels like ages ago, I started on a journey to implement a clone of Microsoft’s C++ Unit Test Framework Test Adapter, but fix some issues I had with it. These included a better reporting mechanism for standard C++ exceptions, and better error messages when libraries were failing to load. Typically this was due to missing dependencies. Part 1 of the series explained Microsoft’s technique to for exposing a dynamic set of classes and functions, for the test framework to call. This is typically solved with reflection, but given that there is no standard mechanism for reflection in C++, some neat tricks in the binary sections were played. Part 2 explores the second step, after discovery — execution. How can we take the metadata information, and actually have it run some code? It’s essentially a plugin system, in a plugin system. Part 2 left off where we were able to actually execute our test methods, but when an Assertion would fail the entire framework would collapse on itself.

I lift my head from my hands, wipe the tears from my eyes, and get real with the situation. I’ve started down this path of re-implementation, why stop now?

Among the package that is shipped with Microsoft’s C++ Unit Test Framework, there is a file called Assert.h. This is the header that you’re to include, in order to make assertions in your tests. Asserting is a critical part of Unit Testing. The three portions of a unit test are essentially:

1. Setup
2. Act
3. Assert

There’s fancy acronyms describing this (AAA), I  prefer mine (SAA). In a general sense, in your test you will do these three things. You setup the test to have your code under test in the state it needs to be. You run the code you want to test. Then you Assert that what was supposed to happen happened, and potentially Assert that other things that weren’t supposed to happen, didn’t. Though this becomes a slippery slope into a rabbit hole. That being said, asserting is a very important part of unit testing, arguably the most important part. So we’ve hit a bit of a conundrum, because negative assertions are the part that tell us when our code isn’t working correctly. Anyone who is experienced with TDD will tell you there is more value in the red lights than the green ones. That means it’s a problem when our framework crashes and burns on a failed Assert.

So, to follow suit with re-implementation. I decided that I would just do away with Microsoft’s Assert library, in favour of my own. Obviously this is the best idea possible. Don’t get me wrong, it’s not like I didn’t try to figure out what was happening with their Assert library. The problem is there is less “public” information in it. Meaning that unlike the CppUnitTest.h file where a lot of the work was in the header because it was template code, most of the assertion code, lived in a compiled binary. The only code in the Assert.h file was template code for comparing generic types. It means I had no real way to figure out what they were doing. All I knew is that whatever they were doing, was crashing my application, and it worked for theirs. So I’ll make one that works with my framework. Now, you might be thinking.

Of what value is re-implementing the Microsoft C++ Unit Test Framework? Is there any actual value in now re-implementing part of their library. 

The answer is probably not, but if you’re curious like me, you like to figure out how things work. The way I do this, is I look at something and implement it myself. If I can do that, I typically run into the problems that the original author ran into, and then I can understand why they solved the problem the way they did. I’ve approached software development this way for my entire professional career, and I’d like to think it has paid dividends.

In all honesty, how hard could writing an assertion library be? Like, basically you only check a few different things. Are these two things equal? Are these two things not equal? Is this thing null? Is this thing not null? If the test passes, don’t do anything. If the test fails, stop executing and barf out some kind of message. If we ponder this for a moment, can we think of something we could use to halt execution and spit out some form of a message? I know! I’ve written this code a lot.

if ( !some_condition )
    throw condition_violated_exception("Violated Condition");

Well, exceptions look like a good place to start for our assertion library. So that’s where we’re going to start. The plan is essentially to have a bunch of Assert calls, that when they fail will throw an exception. Easy right? The code could look something like this.

// MyAssert.h

#include <AssertViolatedException.h>


namespace MyAssert
{
    template <typename T>
    static void AreEqual(const T& expected, const T& actual)
    {
        if( expected != actual )
           throw AssertViolatedException(fmt::format("{0} != {1}", expected, actual));
    }
};

By no means is this complete, it’s really just to illustrate the point that when we have two objects that aren’t equal we generate an exception with an appropriate message.

God it feels good to be a gangster.

Now we can go about our mission, we’ll just use MyAssert.h, and completely disregard Microsoft’s version Assert.h. Given that we’ve implemented so that any escaped standard C++ exception will end up in the framework’s handler. I can guarantee that the assertions will end up there. Right? Given that I didn’t show a snip of AssertViolatedException.h, you can assume that it’s derived from std::exception. If you’re interested in how I actually implemented it, you can find the file here. It has a little bit more complexity, for capturing line information, but largely it’s the same idea.

I’m sure this is EXACTLY how Microsoft would’ve done it.

After we’ve implemented this, we can use it in the same way that you would if you were to use the Assert library included in Microsoft’s framework.

#include <MyAssert.h>
#include "calculator.h"

TEST_CLASS(TestCalculator)
{
public:
    TEST_METHOD(TestAdd)
    {
          calculator c;
          int val = c.add(2, 2);
          MyAssert::AreEqual(4, val);
    }
};

This is great, and it works, for the most part. It works for this case. Can you see where it falls down? If we recall, we’re using a standard C++ exception for our assertion. Unfortunately, the compiler doesn’t care whether or not the exception begins from our MyAssert class or any other place in the application. This means, that any handler prepared to handle a std::exception, will catch our assertion.  Consider this code.

#include <functional>
class calculator
{
   std::function<int(int, int)> on_add_;
public:
     template <typename AddFnT>
     void on_add(const AddFnT &on_add)
     {
           on_add_ = on_add;
     }

     int add(int a, int b)
     {
          try 
          {
                return on_add_(a, b);
          }
          catch(const std::exception &e)
          {
                 return 4;
          }
     }
};

#include <MyAssert.h>
#include "calculator.h"

TEST_CLASS(TestCalculator)
{
public:
    TEST_METHOD(TestAdd)
    {
        calculator c;
        c.on_add([](int a, int b)
        {
               MyAssert::AreEqual(2, a);
               MyAssert::AreEqual(2, b);
               return a + b;
        });
        int val = c.add(2, 22); // see the error here? Tyop.
        MyAssert::AreEqual(4, val);
    }
};

This isn’t by any means good code, nor does it really make sense. Everyone knows developers are notorious for cutting corners to save time, and someone decided 4 was a good error code, and someone made a typo in the test. The unfortunate thing about this, is that it passes. The light is green, but the code is wrong. Now, you’re saying “no one would ever do this in their right mind.” Consider the case where you have a layer between your business logic, and your database. You call the business logic function, it does some work and it passes the values to the store function. A way to test to make sure you’re populating the DB with the right values, is to abstract the database layer and intercept at that level. You also, likely want some error handling there as well. If an exception was to throw from the database. So there you go, at this point our Assert library falls down, hard.

It’s been a long read, and you may feel like you’re getting ripped off at this point, because I haven’t really explained much. Realistically, we actually just learned a whole lot. So I encourage you to keep reading. Microsoft has an Assert library that you can use to make assertions about your program. The generally accepted form of “assertions” within an application is an exception. Microsoft can’t use standard exceptions in their framework, because it could interact with the application under test. I just proved that, by trying it. So what the hell did they do? Well the application crashes, that means something is happening.

Most modern day programmers are familiar with exceptions, most people just see them as the defacto standard for error handling. (I really want to get into talking about return values vs. exceptions for error handling, but that will take me down a rabbit hole.) To keep it short, exceptions in various languages allow for your normal program flow to be separated (for the most part) from your exceptional program flow. Wikipedia, defines exceptions as “anomalous or exceptional conditions requiring special processing”. If you’re familiar exceptions and handling them, you’ve probably seen try/catch before. If you’re familiar with modern C++, you probably at least know of this concept. What you might not know, is that the C++ standard only define what an exception is, not how it is implemented. This means that the behaviour of exceptions and handling them in C++ is standardized, but the way that compiler vendors implement them is free game. Another thing people might not know is languages like C and older languages, don’t have a concept of exceptions.  An exception can come from a custom piece of code, like one above where we throw the exception OR from somewhere deeper down maybe it’s a hardware exception like divide by zero, or out of memory exception. The end result in C++ is the same. Hence why it’s called a “standard” exception. The algorithm is pretty simple. Find an appropriate handler, and unwind the stack until that handler, call the handler. You ever wonder how though?

Well, low level errors like divide by zero, will come from the hardware, generally in the form of an interrupt. So how do we go from a hardware level interrupt to our C++ runtime? On Windows, this is called Structured Exception Handling (SEH). It is Windows concept of exceptions, within the OS itself. There’s a good explanation of this in the book Windows Internals – Part 1 by Mark Russinovich. At a high level, the kernel will trap the exception, if it can’t deal with it itself, it passes it on to user code. The user code, can then either A) deal with the exception and report back stating that, or B) tell the kernel to continue the search. Because this is at the OS level, this is not a language specific thing. So runtimes built on Windows, will use this to implement exceptions within the language. This means that MSVC uses SEH to implement C++ exceptions within the C++ runtime. Essentially the runtime generates a Structured Exception Handler for each frame, and within this the runtime can search for the appropriate handler in the C++, unwind the stack and call the destructor of the objects, then resume execution in the handler. Obviously, these generated Structured Exceptions are well known for the C++ runtime, so it can know how to appropriately deal with the exception.

What if Microsoft was using a Structured Exception for their assertion? The behaviour lines up with that hypothesis, in that something is generated on failed assertion that crashes the application. In SEH, if there isn’t an appropriate handler found the application will be terminated.  How can we prove that? Well it turns out it was easy. Though it’s not recommended. Microsoft recommends if you’re using exceptions in C++ you use standard exceptions, but there is Windows API that you can use in your code to do SEH.

#include "Assert.h"
TEST_METHOD(AssertFail)
{
   __try
   {
       Assert::AreEqual(0,1);
   }
   __except(EXCEPTION_EXECUTE_HANDLER)
   {
       Logger::WriteLine(L"Gotcha!");
   }
}

After we compile and run this code it’s pretty obvious what’s going on. When the Assert::AreEqual fails, we land smack dab in the handler. So I guess that mystery is solved. We just need to figure out how and where to do the exception handling. Now, the __try/__except/__finally APIs are built into C++ on Windows, and allow us to basically install a frame based exception handler. They work very similar to the way you would expect a try/catch to work. After some research I decided this wasn’t exactly what I wanted. I wanted to be able to catch an exception regardless of stack frame. I stumbled upon Vectored Exception Handling. This is an extension to SEH, that allows you to install a handler that gets called regardless of where you are. So you can globally register

The solution then is rather straight-forward. We just need to register an Exception Handler, when the exception throws we can catch it, record the error and continue on our way.  If you read the code in the repository, I had to go through a bunch of layers of indirection to actually get the message to pass to the Test Window Framework. That’s because the architecture of the application has a .NET component, a .NET/CLI component and a pure native component. So for sake of simplicity the way that I went about handling the exception was like this, but not exactly this.

LONG TestModule::OnException(_EXCEPTION_POINTERS *exceptionInfo)
{
    //  if the code is the MS Unit test exception
    if (exceptionInfo->ExceptionRecord->ExceptionCode == 0xe3530001)
    {
        NotifyFailure(reinterpret_cast<const wchar_t*>(exceptionInfo->ExceptionRecord->ExceptionInformation[0]));
    return EXCEPTION_CONTINUE_EXECUTION;
    }
    else
        return EXCEPTION_CONTINUE_SEARCH;
}

auto h = ::AddVectoredExceptionHandler(1, &OnException);

This is where I had to do a bit of guess work. If you recall, this handler will get called for all exceptions. But we only want to do something when it’s an Assert exception. So I had to make the assumption that Assert threw an exception with the code 0xe3530001. Then I did a bit of sleuthing in the memory, to see that a pointer to the message was stored in the first index of the ExceptionRecord ExceptionInformation. With that I could grab the message and fail appropriately.  That being said, I’m not sure if this solution lines up 100% with Microsoft’s functionalities.

To summarize this long journey, I set out to set some things right with the behaviour of Microsoft’s CPP Unit Test Framework. It started out as something fun to investigate and it turned out to be a great learning experience. Out of all the projects that I’ve worked on, I’ve probably learned the most about ingenuity from this one. There are a lot of  neat tricks, cool uses of obscure APIs, and really just overall an interesting view of how Microsoft engineers their tools. You might be wondering to yourself if it was actually worth it. For me, it was about learning, it was about facing the challenges and working to understand them. It wasn’t every really truly about replacing what ships with Visual Studio. So yes, it was worth it. Though I would love it if Microsoft could fix the issues… If I can do it, they most certainly can do it.

Recapping the issues that I ended up solving:

1. Report a better error on standard exceptions [check]
2. Report a better error for binaries that fail to load
3. Support Test Names with Spaces [check]

As you can see, I only solved 2 of the 3 things I set out to solve! The last one is kind of a cop out, because I sort of just lucked into it. When I was messing around with the class attributes, I enhanced my framework to understand a special attribute, to be able to change the test name. Other than just getting it from the method name. So you could specify a test name with a space.

Reporting a better error when binaries fail to load — this is really hard. What makes it hard, is that there isn’t (that I can find) a good API to report the missing dependency. This is something you need to hand roll. Now, there are tools that do it. Specifically Dependency Walker. But, my understanding is that I would need to roll my own dependency walking algorithm. This unfortunately will be a story for another day.

I really hope you enjoyed reading this series. I had quite a bit of fun working on this project, and a lot of fun writing about it.

“What we call the beginning is often the end. And to make an end is to make a beginning. The end is where we start from.” — T.S. Elliot

Happy Coding!

PL

 

References:

MSDN on Structured Exception Handling

https://www.codeproject.com/Articles/2126/How-a-C-compiler-implements-exception-handling