On Leading Yourself

Forgive me, for it has been two months since my last post.

Sometimes life just ends up getting in the way of the things you have desires to do. Your passions and goals get moved to the back burner, as you deal with the daily chaos that is life.  It takes a concerted effort to be able to maintain balance where one is able to dedicate consistent time to their hobbies, interests, and self growth.

In the past couple months, I’ve had a bit of what one could call a career “do-si-do”. It was good and gave some great insight, and different perspectives on careers, life, and balance. It also allowed me to solidify some of the things I had questions about, or the unknowns I wasn’t sure about. I was able to meet a tonne of new people, and gain a whole bunch of different insights into other people’s point of view. When reflecting on everything that transpired in the last 6 months, I couldn’t help but look back on my career. Now, I haven’t had the longest career, I’ve only been a software professional for about 8 years. However, I think my journey has been quite fruitful, and has allowed me to learn a lot about the industry, and learn a lot about myself. Interestingly enough, in my 8 years, I’ve spent more than five, as a “lead”. For me — this felt normal. Isn’t this how every software developers career progresses?

Spoiler alert — it’s not. To me though, it never seemed like I did anything out of ordinary to move myself towards being a technical lead, or even into a team lead position. In fact, when I graduated and started my career, my goal was just to be the best developer. Management? Eff that. I wanted to be the best developer, ever. Not the best developer I could be, not to be the best developer at the company. I wanted to be the best developer — EVER. Lofty goal right? SMART Goal? Wrong, I’ll be the first to say it was a stupid goal. Though it was rooted in good intention, for the most part, I won’t lie and say I wasn’t enticed by the money, women and fame I would get. The reason I wanted it so badly, was that I had never really been the best at anything. Growing up I was exactly average, if you placed me on a bell curve I would be right smack dab in the center of the bell. This being average my whole life lead me to want one thing. I want to be the best at something. I didn’t know what, but I wanted it, bad.

When I got to university, I selected my primary study as business. What 17 year old high-school male doesn’t want to graduate with a business degree, become an investment banker, and then be Batman? I have parents so that tells you something about me becoming Batman. I found out that a degree in business required a lot of reading, something that I wanted no part of. Faced with a daunting path ahead, which consisted of a plethora of reading and likely writing. I did what any 18 year old, pre-business student who hates reading and writing would do. I looked for the easiest out possible. The obvious choice here, was a computing science degree. I had taken Computing Science 101, and received an A (woo not average). So I selected the obvious easiest path forward. Computers were always a thing I was good at, I just hated the thought of being a nerd. Here I was faced with the choice to be a nerd, or suffer having to read books. shudder Flash forward a few years, what was not as easy a path as initially thought, and I graduated. You guessed it, I had an average GPA. To my surprise though, I graduated with a job.

It wasn’t long into my career when I realized that in order to be good at something, I mean above average good, you have to put a substantial amount of effort in. Those people, who are the best at something, anything. Pick something running, swimming, math, music, or writing, you name it. I can tell you one thing about that person, they worked their ass off to get to where they are. It was a cold realization that day, if I wanted to be the best at something, I had to work harder than everyone around me. All the years I spent trying new hobbies, searching for that one thing I would be the best at. I finally realized I would never find it. Because it was right there in front of me. It was a matter of picking something and dedicating myself to it. That meant, to achieve my goal of being the best developer ever. I would need to practice, a lot.

You’re probably thinking at this point, that I’m just telling you my life story. That I’m not actually giving any relevant information on how to actually become a technical lead. Maybe you’re right. Let’s just consider though, if you’re young and you’re looking to be a lead developer, or a team lead, you’re probably not looking to ride the conveyor of time. Sure, that’s one approach to becoming a lead, you wait it out. Eventually, you’ll add enough years of seniority, and enough people in front of you will age out that you’ll get promoted. That is if they don’t hire someone external ahead of you. If you’re fine with that, you can stop reading. If you’re looking to get off the conveyor, you have to be the best, or at least better than the guy in front of you. And that takes work, significant work. You might say “people who are technically strong don’t make the best people leaders”, and you would be right, sometimes. To be a technical lead though, you have to be technically strong. To get strong, you have to train.

So, let’s address that elephant of strong technical leaders not being good people leaders. If we look at why often people who are strong technically, struggle with the interpersonal side. Ask the question, what does it take to be strong technically? Well, with computers especially, you have to be really good at telling them what to do. You have to be very explicit in your instructions. If you’re not, the computer, doing exactly what you told it, will behave different from your expectation. The response is easy. Inspect the source, tell it what you really meant, and have it try again. Machines aren’t like people. After you re-tell what to do, it will execute your instructions with the utmost care and accuracy as before. To be technically strong, you only have to know in depth how and why the computer behaves the way it does. In short, you have to think like a computer. Unfortunately, humans don’t act like computers. So in order to lead people, you have to think, you guessed it — like a human. Which is a hard dichotomy to master. Hence the difficulty between being a technical leader and a people leader.

When I became a Team Lead, I will fully admit, that I wasn’t ready. The guys who had to work with me, would say the same thing. So what did I do that made me stand out? I can’t say for sure, but I think it had something to do with the way I lead myself. Recently I was listing to a Jocko Podcast 170, about 25 minutes in he gives advice to a person who interviewed for a team lead position and didn’t get it, because they didn’t have experience. His quotes is something along the lines of “You’re in charge of something.” He uses examples like machinery or a process. Something. If you can’t think of anything you’re in charge of, think of this. You’re in charge of yourself. The reality is, that if a company can’t see that you’ve got it figured out enough to be in charge of yourself. How can you expect them to feel comfortable letting you be in charge of others? In my case, I didn’t know how to listen to people. I thought being a Team Lead meant barking orders, and expecting perfect results. I didn’t know what I was doing. The silver lining, was I knew how to realistically evaluate the situation, and learn from it. This let me move forward. Moving forward is key. No one is perfect, and no one is the perfect team lead, especially in the technical industry. There’s too much variability, and when people are involved nothing can be perfect. What I have, is a burning desire to do better. I never want to end a day, and be worse at something than I was the day before. There may be setbacks, there may be plateaus, but overall I want an upward direction. So maybe I wasn’t the best at listening, my thought was how can I get better at listening? Maybe I’m not good at communicating. How can I get better at communication? It’s always been about the next step. Taking the next step, and the next step, is how you break away from the pack and lead. This also keeps you humble, because you can realistically see your faults. You have to see them to get better. If you can see your own faults, it will help you lead. A realization that everyone has faults, lets you pick not only yourself up, but others around you as well.

Realistically, if you want to be any sort of lead, in any sort of industry. You have to start by leading yourself. Making the decisions that will ensure that you don’t stagnate. Just because you work 8 hours per day, doesn’t mean you’re getting better. It takes a focused effort, it takes practice, and it takes willpower. You also need the realization that nothing in life is a guarantee. Leading yourself, means taking these steps, not with a short sighted goal, but a long term goal of being better. It means putting in the effort day in, and day out to ensure, you’re better for it. Naturally people will see this, and want to follow it. Eventually, you will be able to switch your focus from supporting and teaching yourself, to supporting and teaching others. That right there, is a whole different story.

“Example is not the main thing in influencing others, it is the only thing.” Albert Schweitzer

[ASPeKT] Oriented Programming

I recently had the pleasure of doing a podcast, with Matthew D. Groves, of Cross Cutting Concerns blog. He essentially “wrote the book”, so to speak, on Aspect Oriented Programming. It’s called AOP in .NET, without pumping his tires too much, I will say that his book is pretty great. I just recently finished reading it, and came to the conclusion that Matthew and I are on the same page regarding a lot of Software Development Fundamentals. Specifically, his take on AOP (Aspect Oriented Programming) and the benefits of it. The ability with AOP to factor out common code that crosses all areas of your code base, and encapsulate it so that you have a single point of change, is a very powerful concept. He illustrates concepts like these, and many others in his book. It also gives a nice overview into the different tools available in the AOP world. Even after writing my own AOP library, I was still able to learn a lot from this book. If you’re interested in AOP or Software Development in general, you should definitely check it out.

To follow up with Matthew’s Podcast, featuring me and the library I created called [ASPeKT] (no relation to ASP, I just liked the way it looked). I wanted to write a post that overviews AOP, and some of the benefits of this powerful programming paradigm. I want to talk about [ASPeKT], the benefits and costs of using a simple AOP library. Then detail some of the challenges I faced as I wrote it, as I work to make it more known, and as I continue to make it better.

Why Aspect Oriented Programming?

AOP is a little known, yet very widely used programming paradigm. How can it be little known, yet very widely used, you ask? Mostly because it’s built into a lot of .NET libraries and frameworks that people use every day, but they just don’t know they’re actually using AOP concepts. Interestingly enough, .NET ASP MVC Authentication, uses AOP patterns. Most people just go about programming and using the Authenticate attribute, without knowing they’re actually hoisting a cross-cutting concern up with a more declarative (cleaner) approach, and letting the framework deal with it.  For me, as a skeptic of AOP at the beginning, it was a huge eye opener to realize that these concepts are actually applied all over in the .NET world. We just don’t realize it. This also brings to light, the power of being able to spot cross-cutting concerns, and be able to encapsulate them, using a library like [ASPeKT]. Thus, removing the need for clunky boilerplate code, and copy-pasta patterns that live in documentation, or worse yet only the minds of your more senior developers.

What is Aspect Oriented Programming, really?

In order to really understand what AOP is, we have to first understand what a “cross-cutting concern” (CCC) is. A CCC is, is the fundamental problem AOP looks to solve. These CCCs typically tend to be the non-functional requirements of your application. They’re code, that spreads across your code base, but they’re not easily factored into their own units. The canonical CCC is logging or auditing. Your application can function, business wise, without it. Yet, if you were to implement the need for “logging” across your application, you would end up with logging code, that pollutes your actual business logic. Things like logging the entry and exit of functions. You end up with code like this.

class Foo 
{
    public void Bar() 
    {
        Console.WriteLine("Entered Bar");
        Console.WriteLine("Pour a drink.");
        Console.WriteLine("Exiting Bar");
    }
};

You can see how this can get tedious, you end up having ‘rules’ or ‘patterns’ that define how and what you log. “This is how we log entry into a function, with it’s parameters. ‘Entering Function – Function. The format lives in a document, stored in SharePoint.” Now imagine when one day that needs to change, and you’re forced to update the 43 million different log lines across your application. Welcome to Hell. Talk about why they call these things ‘concerns’.

Logging isn’t the only concern that spreads and tangles across your code. Things like Authorization, what I call function ‘contracts’ or defensive programming, threading, and many other patterns. These patterns are often overlooked as cross-cutting and don’t always speak to boilerplate, but with a keen eye and some creativity can be teased out into their own re-usable aspects.

Using an AOP tool, allows you to hoist this boilerplate code into its own encapsulated class, where it belongs. Then it allows you to place it declaratively, which makes the code read more explicitly. But also, removes the clutter from the actual business logic of the function. These tools make it so you don’t have to weed through the logging, defensive programming, authorization, etc. Just to find out what the actual intention of the function is. You’ll end up writing code more akin to this.

class LoggedAttribute : Aspect 
{
    public override void OnEntry(MethodArgs args)
    {
        Console.WriteLine($"Entering {args.MethodName}");
    }
    public override void OnExit(MethodArgs args)
    {
        Console.WriteLine($"Exiting {args.MethodName}");
    }
};

class Foo
{
     [Logged]
     public void Bar()
     {
          Console.WriteLine("Pour a drink.");
     }
};

This code effectively functions the same as above. However,  it is substantially cleaner, and the biggest benefit is the ease of change. Now, if I’ve used the LoggedAttribute across my code base and I want to make a change. I only need to make the change in one spot, as opposed to everywhere we copied and pasted the logging lines. AOP allows you to offload the tedious boilerplate code, on the machine. Who is much, much, much faster at typing than humans. The machine also, never makes a typo.

Now that you know what a cross-cutting concern is, I can explain AOP. Effectively AOP is a tool, set of tools, or a paradigm to deal with cross-cutting concerns. In order to prescribe to an AOP model, you need the following 3 things.

  1. A Join Point – this is the point at which our AOP model allows us to interject code. In the case of [ASPeKT], the join-points are method boundaries. i.e. Entering / Exiting of a function. Other libraries like Castle DynamicProxy, and PostSharp allow for actual method interception, so you can say the join-point is this instead of that. This can be useful for things like error handling, retry logic, or threading models.
  2. A Point Cut – this is the way in which you specify, where you want to apply code to the Join Points. Think of this as a sentence describing where you want the code to run. I know my Join Point is Entry/Exit of function. My Point Cut could be “every function that starts with My”, or simply, “every function in the application”. [ASPeKT] uses attribute placement as the Point Cut definition. So where you place the attribute, determines how it will apply the code to the Join Points.
  3. Advice – essentially, the code you want to run at the Join Points. So for [ASPeKT], this is the code you write in OnEntry / OnExit / OnException.

Given these three things, you can start to encapsulate the CCC, which then allows you to start applying that code to other code. You don’t need to copy-pasta, you don’t need to find the document that says how to do logging or auditing. You just apply logging, you apply auditing. It becomes very powerful, because these concerns are now tools in your development toolkit. They’re easily tweaked, they’re modifiable without the daunting overhead of find and replace in every file in the solution. Now, they make sense.

So hopefully now you see the benefit of AOP, and can maybe start to see places where AOP could benefit your projects or workplace. I’ll be honest, it’s not always the easiest sell. So if you’re interested and you want more information, please feel free to reach out. I love talking about these kinds of things.

[ASPeKT] In a Nutshell

My first real intriguing look at AOP, was as I was reading the book Adaptive Code via C# — Gary Maclean Hall. He gave a brief introduction into AOP, mostly to describe its use with logging. Which at the time, I laughed and thought, ‘there’s really no other use than logging’. Later on in the book, he describes a pattern to reduce dependencies on lower layer APIs, by translating API level exceptions into higher level exceptions. This is so higher level components can make decisions based on the exceptions, without being explicitly tied to the lower level library. Consider the use of a library that is encapsulating sending data to a webserver for storage. Something like Office 365 Drive.

You might have code like this.

// MyStorage.dll - our internal wrappers, depends on OfficeWebDrive.dll
interface IStorage 
{
      public void Store(string name, Stream stream);
}


class OfficeDrive : IStorage
{
    public void Store(string name, Stream stream)
    {
         // use OfficeWebDrive.dll third party library
         // make requests to store data,
         // can throw "OfficeWebDrive.StorageFullException"
    }
};
// Application.exe - depends on MyStorage.dll, 
// but should have no dependency on OfficeWebDrive.dll

class Book 
{
    public void AddText(string text)
    {
        // Code to append text.
    }

    public void Save() 
    {
        storage_.Store(Title, Data);
    }
   
    Stream Data { get; }
    string Title { get; }
};

class Application 
{
    public void SomethingHere()
    {
         try
         {
             Book book = new Book("AOP in .NET");
             book.AddText("Lorem Ipsum");
             book.Save();
         }
         catch(OfficeWebDrive.StorageFullException e)
         {
               // Deal with the StorageFullException
         }
    }
};

As you can see, we’ve done our best to program to an interface, using the IStorage interface. Our application should have no dependencies on the actual underlying storage libraries. Except that because we need to deal with the StorageFullException, we need to now explicitly make a dependency between our application and the lower level third party API.

The sentiment then, from Adaptive code, is to wrap the 3rd party library calls, and translate the exception. Like so.

class OfficeDrive : IStorage
{
    public void Store(string name, Stream stream)
    {
        try 
        {
            // use OfficeWebDrive.dll third party library
            // make requests to store data,
            // can throw "OfficeWebDrive.StorageFullException"
        }
        catch(OfficeWebDrive.StorageFullException e)
        {
             // translate the error
             throw new MyStorage.StorageFullException(e.Message);
        }
    }
};

Now, the higher level code can make the same decisions about what to do when the storage is full, but no dependencies on the low level libraries. If we wanted, we could completely change the underlying storage mechanism. Without needing the Application to rebuild.

‘Hey Gary, this is a great spot for AOP’ I thought being clever.

class OfficeDrive : IStorage
{
    [TranslateException(typeof(OfficeWebDrive.StorageFullException), 
     typeof(MyStorage.StorageFullException))]
    public void Store(string name, Stream stream)
    {
        // use OfficeWebDrive.dll third party library
        // make requests to store data,
        // can throw "OfficeWebDrive.StorageFullException"
    }
};

Now, we let the AOP framework handle the boilerplate try/catch code. This also really calls out what is happening, and why.

Where does [ASPeKT] come in?

Well — after I thought about this, I wanted to build it. I guess I could’ve easily used the PostSharp Express version and whipped it out really quickly. But that’s not me. If I’m going to understand something, I’m going to understand it. So I set off to write a small AOP library, where I could solve this problem. It was a rather simple concept, or so I thought.

I didn’t even really know what an AOP library did. That’s where the research started, “open source AOP libraries”, “how does PostSharp work”, etc, etc. Just some of the multitude of search terms I used while I did research into how to build an AOP library.

I had the concept of what I wanted. I wanted something that I could declare an attribute, that would translate an exception. Easy.

Let’s go down the rabbit hole.

At it’s core, AOP is actually allowing a machine to write code for you. You’ve moved the copy and paste from your fingers Ctrl+C / Ctrl-V to the machines apt fingers. Allowing the computer to ‘weave’ the code for you.  You define the boilerplate as an aspect, and you let the machine do the work for you, because that’s what they’re good at (no offense computer).

You’ve got three options for when this can happen.

  1. In the source code, before you compile. Using some type of marked up code in your source, you could run a pre-compiler that will insert code snippets into the source. Before the compiler does it’s work.
  2. After the compiler — though not impossible on native assemblies. It is far easier on languages like C# and Java, which use an intermediate language, between source code and assembly. We can post-process the compiled intermediate language (IL), to apply the code we want.
  3. At runtime. Well this has somewhat of an over-watch pattern, where something is watching the calls as they run through and interpreting whether or not to run our aspects.

Now — knowing this, which one do you choose? My thought process was as follows.

  1. Option 1: Modify the source — I don’t want to write a compiler. Well, I do, but not for this. So that was off the table, at least for now. You also have dependency on language syntax, not that I would expect much of C# syntax to change, but still.
  2. Option 3: At runtime – I don’t want this. I come from a native background, specifically C++. I don’t want to pay overhead where I don’t have to. I didn’t want to have something that is monitoring functions as they run, or building code at runtime. Just wasn’t what I wanted.

So that left option 2, what exactly is that? How does it even work?  I need something that will run, after the binary compiles, and modify it to add in the CCC code.

Let’s go deeper…

To understand post compile weaving, we must first understand how the .NET runtime operates, at a high level. The .NET Runtime is what’s called a “virtual machine”. The draw towards these “virtual machines”, was started with JAVA created by Sun Microsystems. The large benefit to JAVA, was that it was a compile once, run anywhere language. This was hugely powerful, when you consider the alternative of compiling for every architecture, as is the case with C and C++.  The virtual machine, allows you to compile to a well known set of instructions (IL), which then will output machine specific instructions for the hardware when run through the virtual machine. This way, you only need to write a virtual machine for the hardware, and you open yourself up to all the applications available for that runtime. This is one of the reasons JAVA became so hugely popular, because you could write a JVM for your VCR and all of a sudden pow smart VCR, with lots of apps available.

Obviously, Microsoft saw the benefit and flexibility in this and took advantage, so they started shipping JAVA in Visual Studio. They had their own JVM, and JAVA compiler as well. They also saw an advantage to extend this language, for the Windows operating system. Enter J++, Microsoft’s implementation of Java with extensions for the Windows OS. With J++, came a lawsuit from Sun Microsystems. Microsoft had a non-compliant JVM and they were violating the terms of the license agreement (who reads those things anyways?). Wah. Wah. So what does Microsoft do? They do what any billion dollar software development company would do. They eat the lawsuit, and take J++ and turn into what we now know as C#.  They also see the immense power in this .NET Runtime, and see that they can compile a whole multitude of different languages into IL. With the release of .NET Runtime 1.0, there was support for 26 languages (I think). To be completely honest, I’m glad that Sun sued Microsoft, because I hate JAVA as a language, and I love C#. So it was a win, in my opinion.

Anyways — aside from that little lesson in history, we can understand how ‘weaving’ works in a .NET language. Like I said, C# is a language that compiles to IL, aka CIL or MSIL. An intermediate language, is on the fence. Between a language that is “compiled” to actual assembly (hardware instructions). And an interpreted language like JS, that is completely interpreted at runtime. The C# compiler, takes the C# language and outputs a binary form of instructions that complies to the .NET runtime standard. These instructions are then interpreted, and compiled Just-In-Time to output hardware instructions. This means, that after we compile to IL, and before running, we can insert some extra IL instructions, then voila.

How do you weave IL?

Weaving IL is actually pretty straight forward, if you know what you’re doing. Which I didn’t. Ha. At first, I kind of flew by the seat of my pants. I knew I needed to re-write the IL, and I had heard about tools like Roslyn and Mono.Cecil. Roslyn being a compiler, wasn’t exactly what I wanted. I needed at tool to modify the IL, which is exactly what Mono.Cecil is. It not only uses the built in Reflection.Emit, but also adds a lot of ease to manipulating the IL.

The task at hand for me, was to open the binary, find the spot that I had declared my “TranslateExcepton” and then insert the instructions for that, into the method where I declared.  I just decided to make it generic, to work with any Aspects I created. I will spare the gory details, but the high-level was as follows.

  1. Open the compiled .NET assembly
  2. Find functions decorated with Aspekt.Aspects
  3. Write IL to instantiate the Aspect, and make the entry call
  4. Write IL to wrap the existing code in a try/catch
  5. Execute the existing code
  6. Write IL to make the exit call

I could write an entire post on weaving IL, but not today. If you’re interested in that, please drop me a line and let me know. I can describe, in detail, all the pain I felt while I was learning how to do this.

Once, I had figured out how to do this. I had an executable that I could run against my compiled .NET binaries, which would weave aspect code, where I place attributes. Then all I needed to do was write the aspect, to translate the exception. You can actually find this aspect in the [ASPeKT] code. Kind of like an ode to the beginning, for me.

What came next?

Now that I had the start of this framework. I thought to myself, that I had actually built something that could be useful. “People might actually want to use this”. I had also always wanted to author an open source library. So I started reaching out on Reddit, and making the foundation more easily accessible for people to use. I made the core more stable. Then I started writing new features, like [ASPeKT] Contracts. It was a surprising journey. In the past I had written many throw away libraries, and many small tools that have never seen the light of day. Always just to prove to myself, well, that I could. There was just something to this one, it was a niche library and I thought there was something to it.

I guess the reality is that I’m still in the middle of this journey, with [ASPeKT] core having 588 downloads, and Contracts having slightly less at 416. I’m still working towards my goal of 1000 downloads for each. Realistically, a project that uses [ASPeKT] could be the spark that it needs to ignite. Until then though, I will just keep plugging away and making it cooler and easier to use.

Why should you use [ASPeKT]?

Well — if you’re at all curious in the inner workings of an AOP library, the code is public on my GitHub or you can easily get the package on NuGet. The second, is maybe you have a project that could benefit from AOP? [ASPeKT] is pretty lightweight, and easy to get started with. Though, if you’re looking for a robust, feature complete, production ready library — [ASPeKT] isn’t there yet. If you’re looking for a library to contribute to, or a library that can be adapted into something production ready, then shoot me an email!

“Man cannot discover new oceans unless he has the courage to lose sight of the shore.”
― Andre Gide

As always, thanks for reading. Happy Coding!

PL

The one where we reverse engineered Microsoft’s C++ Unit Test Framework (Part 3) – Exceptions

Again, if you haven’t already done so I suggest reading through Part 1 and Part 2 of the series. I’ll do a quick TL; DR; recap, but it may not do it justice.

What feels like ages ago, I started on a journey to implement a clone of Microsoft’s C++ Unit Test Framework Test Adapter, but fix some issues I had with it. These included a better reporting mechanism for standard C++ exceptions, and better error messages when libraries were failing to load. Typically this was due to missing dependencies. Part 1 of the series explained Microsoft’s technique to for exposing a dynamic set of classes and functions, for the test framework to call. This is typically solved with reflection, but given that there is no standard mechanism for reflection in C++, some neat tricks in the binary sections were played. Part 2 explores the second step, after discovery — execution. How can we take the metadata information, and actually have it run some code? It’s essentially a plugin system, in a plugin system. Part 2 left off where we were able to actually execute our test methods, but when an Assertion would fail the entire framework would collapse on itself.

I lift my head from my hands, wipe the tears from my eyes, and get real with the situation. I’ve started down this path of re-implementation, why stop now?

Among the package that is shipped with Microsoft’s C++ Unit Test Framework, there is a file called Assert.h. This is the header that you’re to include, in order to make assertions in your tests. Asserting is a critical part of Unit Testing. The three portions of a unit test are essentially:

  1. Setup
  2. Act
  3. Assert

There’s fancy acronyms describing this (AAA), I  prefer mine (SAA). In a general sense, in your test you will do these three things. You setup the test to have your code under test in the state it needs to be. You run the code you want to test. Then you Assert that what was supposed to happen happened, and potentially Assert that other things that weren’t supposed to happen, didn’t. Though this becomes a slippery slope into a rabbit hole. That being said, asserting is a very important part of unit testing, arguably the most important part. So we’ve hit a bit of a conundrum, because negative assertions are the part that tell us when our code isn’t working correctly. Anyone who is experienced with TDD will tell you there is more value in the red lights than the green ones. That means it’s a problem when our framework crashes and burns on a failed Assert.

So, to follow suit with re-implementation. I decided that I would just do away with Microsoft’s Assert library, in favour of my own. Obviously this is the best idea possible. Don’t get me wrong, it’s not like I didn’t try to figure out what was happening with their Assert library. The problem is there is less “public” information in it. Meaning that unlike the CppUnitTest.h file where a lot of the work was in the header because it was template code, most of the assertion code, lived in a compiled binary. The only code in the Assert.h file was template code for comparing generic types. It means I had no real way to figure out what they were doing. All I knew is that whatever they were doing, was crashing my application, and it worked for theirs. So I’ll make one that works with my framework. Now, you might be thinking.

Of what value is re-implementing the Microsoft C++ Unit Test Framework? Is there any actual value in now re-implementing part of their library. 

The answer is probably not, but if you’re curious like me, you like to figure out how things work. The way I do this, is I look at something and implement it myself. If I can do that, I typically run into the problems that the original author ran into, and then I can understand why they solved the problem the way they did. I’ve approached software development this way for my entire professional career, and I’d like to think it has paid dividends.

In all honesty, how hard could writing an assertion library be? Like, basically you only check a few different things. Are these two things equal? Are these two things not equal? Is this thing null? Is this thing not null? If the test passes, don’t do anything. If the test fails, stop executing and barf out some kind of message. If we ponder this for a moment, can we think of something we could use to halt execution and spit out some form of a message? I know! I’ve written this code a lot.

if ( !some_condition )
    throw condition_violated_exception("Violated Condition");

Well, exceptions look like a good place to start for our assertion library. So that’s where we’re going to start. The plan is essentially to have a bunch of Assert calls, that when they fail will throw an exception. Easy right? The code could look something like this.

// MyAssert.h

#include <AssertViolatedException.h>


namespace MyAssert
{
    template <typename T>
    static void AreEqual(const T& expected, const T& actual)
    {
        if( expected != actual )
           throw AssertViolatedException(fmt::format("{0} != {1}", expected, actual));
    }
};

By no means is this complete, it’s really just to illustrate the point that when we have two objects that aren’t equal we generate an exception with an appropriate message.

God it feels good to be a gangster.

Now we can go about our mission, we’ll just use MyAssert.h, and completely disregard Microsoft’s version Assert.h. Given that we’ve implemented so that any escaped standard C++ exception will end up in the framework’s handler. I can guarantee that the assertions will end up there. Right? Given that I didn’t show a snip of AssertViolatedException.h, you can assume that it’s derived from std::exception. If you’re interested in how I actually implemented it, you can find the file here. It has a little bit more complexity, for capturing line information, but largely it’s the same idea.

I’m sure this is EXACTLY how Microsoft would’ve done it.

After we’ve implemented this, we can use it in the same way that you would if you were to use the Assert library included in Microsoft’s framework.

#include <MyAssert.h>
#include "calculator.h"

TEST_CLASS(TestCalculator)
{
public:
    TEST_METHOD(TestAdd)
    {
          calculator c;
          int val = c.add(2, 2);
          MyAssert::AreEqual(4, val);
    }
};

This is great, and it works, for the most part. It works for this case. Can you see where it falls down? If we recall, we’re using a standard C++ exception for our assertion. Unfortunately, the compiler doesn’t care whether or not the exception begins from our MyAssert class or any other place in the application. This means, that any handler prepared to handle a std::exception, will catch our assertion.  Consider this code.

#include <functional>
class calculator
{
   std::function<int(int, int)> on_add_;
public:
     template <typename AddFnT>
     void on_add(const AddFnT &on_add)
     {
           on_add_ = on_add;
     }

     int add(int a, int b)
     {
          try 
          {
                return on_add_(a, b);
          }
          catch(const std::exception &e)
          {
                 return 4;
          }
     }
};
#include <MyAssert.h>
#include "calculator.h"

TEST_CLASS(TestCalculator)
{
public:
    TEST_METHOD(TestAdd)
    {
        calculator c;
        c.on_add([](int a, int b)
        {
               MyAssert::AreEqual(2, a);
               MyAssert::AreEqual(2, b);
               return a + b;
        });
        int val = c.add(2, 22); // see the error here? Tyop.
        MyAssert::AreEqual(4, val);
    }
};

This isn’t by any means good code, nor does it really make sense. Everyone knows developers are notorious for cutting corners to save time, and someone decided 4 was a good error code, and someone made a typo in the test. The unfortunate thing about this, is that it passes. The light is green, but the code is wrong. Now, you’re saying “no one would ever do this in their right mind.” Consider the case where you have a layer between your business logic, and your database. You call the business logic function, it does some work and it passes the values to the store function. A way to test to make sure you’re populating the DB with the right values, is to abstract the database layer and intercept at that level. You also, likely want some error handling there as well. If an exception was to throw from the database. So there you go, at this point our Assert library falls down, hard.

It’s been a long read, and you may feel like you’re getting ripped off at this point, because I haven’t really explained much. Realistically, we actually just learned a whole lot. So I encourage you to keep reading. Microsoft has an Assert library that you can use to make assertions about your program. The generally accepted form of “assertions” within an application is an exception. Microsoft can’t use standard exceptions in their framework, because it could interact with the application under test. I just proved that, by trying it. So what the hell did they do? Well the application crashes, that means something is happening.

Most modern day programmers are familiar with exceptions, most people just see them as the defacto standard for error handling. (I really want to get into talking about return values vs. exceptions for error handling, but that will take me down a rabbit hole.) To keep it short, exceptions in various languages allow for your normal program flow to be separated (for the most part) from your exceptional program flow. Wikipedia, defines exceptions as “anomalous or exceptional conditions requiring special processing”. If you’re familiar exceptions and handling them, you’ve probably seen try/catch before. If you’re familiar with modern C++, you probably at least know of this concept. What you might not know, is that the C++ standard only define what an exception is, not how it is implemented. This means that the behaviour of exceptions and handling them in C++ is standardized, but the way that compiler vendors implement them is free game. Another thing people might not know is languages like C and older languages, don’t have a concept of exceptions.  An exception can come from a custom piece of code, like one above where we throw the exception OR from somewhere deeper down maybe it’s a hardware exception like divide by zero, or out of memory exception. The end result in C++ is the same. Hence why it’s called a “standard” exception. The algorithm is pretty simple. Find an appropriate handler, and unwind the stack until that handler, call the handler. You ever wonder how though?

Well, low level errors like divide by zero, will come from the hardware, generally in the form of an interrupt. So how do we go from a hardware level interrupt to our C++ runtime? On Windows, this is called Structured Exception Handling (SEH). It is Windows concept of exceptions, within the OS itself. There’s a good explanation of this in the book Windows Internals – Part 1 by Mark Russinovich. At a high level, the kernel will trap the exception, if it can’t deal with it itself, it passes it on to user code. The user code, can then either A) deal with the exception and report back stating that, or B) tell the kernel to continue the search. Because this is at the OS level, this is not a language specific thing. So runtimes built on Windows, will use this to implement exceptions within the language. This means that MSVC uses SEH to implement C++ exceptions within the C++ runtime. Essentially the runtime generates a Structured Exception Handler for each frame, and within this the runtime can search for the appropriate handler in the C++, unwind the stack and call the destructor of the objects, then resume execution in the handler. Obviously, these generated Structured Exceptions are well known for the C++ runtime, so it can know how to appropriately deal with the exception.

What if Microsoft was using a Structured Exception for their assertion? The behaviour lines up with that hypothesis, in that something is generated on failed assertion that crashes the application. In SEH, if there isn’t an appropriate handler found the application will be terminated.  How can we prove that? Well it turns out it was easy. Though it’s not recommended. Microsoft recommends if you’re using exceptions in C++ you use standard exceptions, but there is Windows API that you can use in your code to do SEH.

#include "Assert.h"
TEST_METHOD(AssertFail)
{
   __try
   {
       Assert::AreEqual(0,1);
   }
   __except(EXCEPTION_EXECUTE_HANDLER)
   {
       Logger::WriteLine(L"Gotcha!");
   }
}

After we compile and run this code it’s pretty obvious what’s going on. When the Assert::AreEqual fails, we land smack dab in the handler. So I guess that mystery is solved. We just need to figure out how and where to do the exception handling. Now, the __try/__except/__finally APIs are built into C++ on Windows, and allow us to basically install a frame based exception handler. They work very similar to the way you would expect a try/catch to work. After some research I decided this wasn’t exactly what I wanted. I wanted to be able to catch an exception regardless of stack frame. I stumbled upon Vectored Exception Handling. This is an extension to SEH, that allows you to install a handler that gets called regardless of where you are. So you can globally register

The solution then is rather straight-forward. We just need to register an Exception Handler, when the exception throws we can catch it, record the error and continue on our way.  If you read the code in the repository, I had to go through a bunch of layers of indirection to actually get the message to pass to the Test Window Framework. That’s because the architecture of the application has a .NET component, a .NET/CLI component and a pure native component. So for sake of simplicity the way that I went about handling the exception was like this, but not exactly this.

LONG TestModule::OnException(_EXCEPTION_POINTERS *exceptionInfo)
{
    //  if the code is the MS Unit test exception
    if (exceptionInfo->ExceptionRecord->ExceptionCode == 0xe3530001)
    {
        NotifyFailure(reinterpret_cast<const wchar_t*>(exceptionInfo->ExceptionRecord->ExceptionInformation[0]));
    return EXCEPTION_CONTINUE_EXECUTION;
    }
    else
        return EXCEPTION_CONTINUE_SEARCH;
}

auto h = ::AddVectoredExceptionHandler(1, &OnException);

This is where I had to do a bit of guess work. If you recall, this handler will get called for all exceptions. But we only want to do something when it’s an Assert exception. So I had to make the assumption that Assert threw an exception with the code 0xe3530001. Then I did a bit of sleuthing in the memory, to see that a pointer to the message was stored in the first index of the ExceptionRecord ExceptionInformation. With that I could grab the message and fail appropriately.  That being said, I’m not sure if this solution lines up 100% with Microsoft’s functionalities.

To summarize this long journey, I set out to set some things right with the behaviour of Microsoft’s CPP Unit Test Framework. It started out as something fun to investigate and it turned out to be a great learning experience. Out of all the projects that I’ve worked on, I’ve probably learned the most about ingenuity from this one. There are a lot of  neat tricks, cool uses of obscure APIs, and really just overall an interesting view of how Microsoft engineers their tools. You might be wondering to yourself if it was actually worth it. For me, it was about learning, it was about facing the challenges and working to understand them. It wasn’t every really truly about replacing what ships with Visual Studio. So yes, it was worth it. Though I would love it if Microsoft could fix the issues… If I can do it, they most certainly can do it.

Recapping the issues that I ended up solving:

  1. Report a better error on standard exceptions [check]
  2. Report a better error for binaries that fail to load
  3. Support Test Names with Spaces [check]

As you can see, I only solved 2 of the 3 things I set out to solve! The last one is kind of a cop out, because I sort of just lucked into it. When I was messing around with the class attributes, I enhanced my framework to understand a special attribute, to be able to change the test name. Other than just getting it from the method name. So you could specify a test name with a space.

Reporting a better error when binaries fail to load — this is really hard. What makes it hard, is that there isn’t (that I can find) a good API to report the missing dependency. This is something you need to hand roll. Now, there are tools that do it. Specifically Dependency Walker. But, my understanding is that I would need to roll my own dependency walking algorithm. This unfortunately will be a story for another day.

I really hope you enjoyed reading this series. I had quite a bit of fun working on this project, and a lot of fun writing about it.

“What we call the beginning is often the end. And to make an end is to make a beginning. The end is where we start from.” — T.S. Elliot

Happy Coding!

PL

 

References:

MSDN on Structured Exception Handling

https://www.codeproject.com/Articles/2126/How-a-C-compiler-implements-exception-handling

The one where we reverse engineered Microsoft’s C++ Unit Test Framework (Part 2)

If you haven’t already read Part 1, of this series, then I suggest giving it a skim. If not, I’ll give a quick TL;DR;

A few years ago, I was frustrated with some of the idiosyncrasies of Microsoft’s C++ Unit Test Framework. I set out on a mission to develop a custom test adapter, to solve my problems. It would essentially replace the stock adapter that lives in Visual Studio’s Test Explorer window. There would be no difference to writing tests, the only differences would be in the execution of the tests. Things like capturing standard C++ exceptions and displaying the text, and also understanding why binaries were failing to load. It was a bit of a mission. Part 1 of this series, dives into the mechanisms that Microsoft has developed to expose metadata about the tests. A C++ semi-reflection of sorts, allowing inspection of the binary, without loading it, to discover executable tests. They did this using a set of macros, storing information about the test functions in special sections in the binary. That exercise in discovery, was very fun and exciting for me. But it didn’t stop there, we still need to figure out execution of these tests.

“I have no idea what I’m doing.”  I sat with my head in my hands. I was trying to reason out why the test executor was crashing the engine when test asserts failed. 

Writing a test adapter that plugs into Visual Studio is a relatively simple task. You need a managed (.NET/CLR) library, that publicly exposes two interfaces.

public interface ITestDiscoverer
{
     void DiscoverTests(IEnumerable<string> sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink);
}

public interface ITestExecutor
{
    void Cancel();
    void RunTests(IEnumerable<string> sources, IRunContext runContext, IFrameworkHandle frameworkHandle);
    void RunTests(IEnumerable<TestCase> tests, IRunContext runContext, IFrameworkHandle frameworkHandle);
}

The first interface is used to populate the Test Explorer with the test information. If you’re unfamiliar with the Test Window in Visual Studio. You should get familiar with it, it’s your friend and can be a great teacher if used correctly. You can display it by selecting Test -> Windows -> Test Explorer. It’s come a long way since Visual Studio 2012, and I’m sure Microsoft will be enhancing it in further versions. There’s a reason Microsoft is investing in this Unit Test technology. Unit testing is an important part of software development. With a few C# attributes, sprinkled with some reflection you could easily craft your own managed test executor. Then you describe what types of files your discoverer discovers, and you implement this function. It will get called after a successful build (I’m not sure how live testing affects this.), telling your discoverer a list of files to discover tests in. Your discoverer, should then load the files, look for tests, then send the Test Cases to the Discovery Sink, which will then have those tests displaying in the Test Explorer window. From the last post, you can see how we could implement the ITestDiscoverer interface, in C++/CLI and then use the library we created to walk the binary searching for test cases. So I won’t go into detail on that.

The next actual hurdle, is with execution of the tests, this is done with the ITestExecutor interface. Again, I will leave it up to your imagination, or you can look at my project here to see how this gets tied into the managed world of Visual Studio. I will be describing how we dive into the actual native execution of these tests.

If we step behind the curtains, and think about really what execution of a test is. It’s just a fancy way of executing a function (or method if you prefer) on a class. There is some ‘execution engine’ which is the process in which this will live and occur, and that process will load in your library (or executable for that matter), instantiate one of your test classes, then execute the method or ‘Test Case’ on your class. This is a bit of an over simplification, but for all intents and purposes, that is the ‘magic’ behind the Test Explorer window. Now, if you’re doing shared libraries or DLLs on Windows in C++, there are two ways to call exported functions. The first method, is to use project referencing (or include the header, and add the .lib file path) and allow the compiler and linker to do the work. Another approach is to use ::LoadLibrary and dynamically load the binary yourself. The downside to using the compiler and linker, is that you have to know the classes and libraries at compile time. But it takes care of all the details of loading the library, and linking for you. The benefit to using ::LoadLibrary, is that you’re not tied to binaries at compile time. You could use this, along with some interfaces, to create a plugin system. The downside is there is only implicit contracts between the libraries you’re loading and the application. The compiler cannot enforce that the plugin is implemented correctly.

Our test discoverer and executor, is in essence a plugin which loads plugins. Each of the test libraries, are in fact plugins exposing an interface that we want to call on. So, we need to use a method where we dynamically load the libraries at run-time. When you’re doing dynamic loading of DLLs, it isn’t enough to simply load the DLL into your process. You have to know what and where you want to call. With C++, using classes, this concept gets harder to see. So, as I love to do, we will simplify and go back to an age where these things were more straight-forward. The age of C.

Let’s imagine we were going to write some C application, that would load different plugins to print some text. This is a relatively simple problem, and it will illustrate the point. The design is simple, we could have two components. We would have our executable “Print Application” and a set of “Printer Driver” plugins or DLLs.

// Printer Application 
#include "Windows.h"

// a function pointer, describing the signature of our plugin print function
int (*PrinterFunction)(char*, int);

// our print function
int print(char *driverPath, char * textToPrint, int length)
{
    // We want to load our driver
    HMODULE library = ::LoadLibrary(driver);
    if(library == NULL)
        return 0; // we failed to load, can't do nothing.
    
    // We want to call our drivers print function
    PrinterFunction printFunction = (PrinterFunction)::GetProcAddress(library, "printText");
    if(printFunction == NULL)
        return 0; // no function in the DLL, can't print.

   // finally print.
   return printFunction(textToPrint, length);
    
}

int main(char **argv, int argc)
{
   int printed = print(argv[0], argv[1]);
   printf("You printed %d bytes to %s", printed, argv[0]);
}

 

// Old Brother Printer Driver
#include "OldBrotherPrinter.h"

// The __declspec(dllexport) here, tells the compiler to expose the function
int __declspec(dllexport) printText(char *textToPrint, int length)
{
   PRINTER p = ConnectToOldBrother("COM1");

   return SendBytesToPrinter(p, textToPrint, length);
}

If you were to compile and run this, you would get an .exe and a .dll. One is the application itself and the other is our plugin printer library. When we run our application, we can give it the path to our OldBrotherPrinter.dll and some text, and it should print our text.

There are two very important things to note here. The first is the function pointer that we’ve declared. This means that we know the signature of the function that we want to call. It’s a function that takes a character pointer, and an int as arguments, then returns an int. The second part is that we know the name, “printText”. Now, if the library doesn’t expose a function called “printText” we can’t get the address to it. If it’s not the same signature, we’re going to have a bad time calling it. There are some implicit contracts between the caller and the library implementer. The call to ::LoadLibrary, will load the binary into our memory space. The ::GetProcAddress call, will find the address to that function in our memory space, so that we can make a call on it. We need to cast the return to our known signature, so that we can call it with our arguments. The take-away from this exercise, is that we need to know the name of the function, and the signature, to be able to load and call it on a loaded library.

The reason that I needed to explain this using C, is because it is less complex than C++. As we know, in C++ there are things like classes, and more importantly, function overloading. In plain C, you could see the function name was exported as the name “printText”. This is because in C, you can only have ONE function named “printToText”. In C++, we have the concept of function overloading. Allowing us to do something like.

// Printer Functions.
#include <string>

int printToText(int someInteger);
int printToText(const std::string &someText);
int printToText(char someCharacter);

If you’re asking, well how can that be, they’re all named the same, how can you differentiate them? That’s the right question. This is done by something called ‘name decoration’.  The function names, really look more like this, from MSVC 19, in Visual Studio 2019.

// Decorated function names
?printToText@@YAHH@Z
?printToText@@YAHABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z
?printToText@@YAHD@Z

Now, when you mix in classes, the decoration gets a little bit more complicated. Code like this.

// Printer Class
#include <string>
class PrinterClass
{
    const std::string myData_;
public:
    int __declspec(dllexport) printToText()
    {
        return 0;
    }
};

Will result in a decorated function named something like this.

?printToText@PrinterClass@@QAEHXZ

If you somehow could know the decorated name, then you could load that function by its decorated name. Alright, at this point you’re probably thinking. Are we ever going to get back to talking about C++ Unit Test Framework? Realistically, that’s what you came here to read. However, this is really important background. I hope you can see the foreshadowing. If not, I’ll lay it out on the table.

In order to dynamically load a library, and make a call into it. We need to know three things.

  1. The name of the library
  2. The name of the function we want to call
  3. The signature of the function we want to call

So, knowing we need those things. I hope you’re asking yourself, which ones do we have? Which ones don’t we have, and how do we get them? Well I can tell you, we have 1 and we have 3.  What we are missing, is the all important name of the function we want to call.  The framework will supply us the name of the binary, we know the function signature is ‘void (void)’ so, we just need the name of the function we want to call.

Huh. How the heck are we going to get that? A user can name the function whatever you want to name it. We also have that added pain that the functions live in classes, which the user can also name. Stumped yet? Yeah — me too. When I’m stumped, I go back to the drawing board. In this case, let’s go back to reviewing that “CppUnitTest.h” file. Do you recall back to when we looked at the TEST_METHOD macro? If not, it looks like this.

#define TEST_METHOD(methodName)\
static const EXPORT_METHOD ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo* CALLING_CONVENTION CATNAME(__GetTestMethodInfo_, methodName)()\
{\
    __GetTestClassInfo();\
    __GetTestVersion();\
    ALLOCATE_TESTDATA_SECTION_METHOD\
    static const ::Microsoft::VisualStudio::CppUnitTestFramework::MethodMetadata s_Metadata = {L"TestMethodInfo", L#methodName, reinterpret_cast<const unsigned char*>(__FUNCTION__), reinterpret_cast<const unsigned char*>(__FUNCDNAME__), __WFILE__, __LINE__};\
\
    static ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo s_Info = {::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo::TestMethod, NULL, &s_Metadata};\
    s_Info.method.pVoidMethod = static_cast<::Microsoft::VisualStudio::CppUnitTestFramework::TestClassImpl::__voidFunc>(&methodName);\
    return &s_Info;\
}\
void methodName()

You can see, that one of the macros that is being used is the FUNCTION, and another FUNCDNAME. Well, we know FUNCTION will give us the un-decorated name of the function, maybe FUNCDNAME would be a decorated one? Thank you Microsoft documenation!

‘__FUNCDNAME__ Defined as a string literal that contains the decorated name of the enclosing function. The macro is defined only within a function. The __FUNCDNAME__macro is not expanded if you use the /EP or /P compiler option.

This example uses the __FUNCDNAME____FUNCSIG__, and __FUNCTION__ macros to display function information.’

Well color me stoked, we just made the next tiny step. A decorated function name! But this macro is weird. Do you remember what the memory looked like?

0x07462D94  54 00 65 00 73 00 74 00 4d 00 65 00 74 00 68 00 6f 00 64 00 49 00 6e 00 66 00 6f 00 00 00 00 00 00 00 00 00 44 00  T.e.s.t.M.e.t.h.o.d.I.n.f.o.........D.
0x07462DBA  75 00 6d 00 6d 00 79 00 41 00 73 00 73 00 65 00 72 00 74 00 00 00 00 00 00 00 00 00 00 00 43 50 50 55 6e 69 74 54  u.m.m.y.A.s.s.e.r.t...........CPPUnitT
0x07462DE0  65 73 74 49 6e 76 65 73 74 69 67 61 74 6f 72 54 65 73 74 3a 3a 6e 65 73 74 65 64 3a 3a 44 75 6d 6d 79 43 6c 61 73  estInvestigatorTest::nested::DummyClas
0x07462E06  73 3a 3a 5f 5f 47 65 74 54 65 73 74 4d 65 74 68 6f 64 49 6e 66 6f 5f 44 75 6d 6d 79 41 73 73 65 72 74 00 00 00 00  s::__GetTestMethodInfo_DummyAssert....
0x07462E2C  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3f 5f 5f 47 65 74 54 65 73 74 4d 65 74 68 6f 64 49 6e  ....................?__GetTestMethodIn
0x07462E52  66 6f 5f 44 75 6d 6d 79 41 73 73 65 72 74 40 44 75 6d 6d 79 43 6c 61 73 73 40 6e 65 73 74 65 64 40 43 50 50 55 6e  fo_DummyAssert@DummyClass@nested@CPPUn
0x07462E78  69 74 54 65 73 74 49 6e 76 65 73 74 69 67 61 74 6f 72 54 65 73 74 40 40 53 47 50 42 55 4d 65 6d 62 65 72 4d 65 74  itTestInvestigatorTest@@SGPBUMemberMet
0x07462E9E  68 6f 64 49 6e 66 6f 40 43 70 70 55 6e 69 74 54 65 73 74 46 72 61 6d 65 77 6f 72 6b 40 56 69 73 75 61 6c 53 74 75  hodInfo@CppUnitTestFramework@VisualStu
0x07462EC4  64 69 6f 40 4d 69 63 72 6f 73 6f 66 74 40 40 58 5a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  dio@Microsoft@@XZ

Hmm, the decorated name is.

?__GetTestMethodIn
fo_DummyAssert@DummyClass@nested@CPPUnitTestInvestigatorTest@@SGPBUMemberMet
hodInfo@CppUnitTestFramework@VisualStu
dio@Microsoft@@XZ

It looks, like the function name is __GetTestMethodInfo_DummyAssert(). That’s not the name of our function? Our function was called DummyAssert. Color me confused. Looking back at the macro, now we can see it actually just macros out a metadata function, and then starts our function. So it was never really capturing our method name at all, it was capturing metadata about our function. Shoot! How do we call it now?

Ahhhhh! Time to breathe. Time to rack our brains. Time to dig deep.

Well — what exactly is this MethodMetadata for? They wouldn’t put it in for no reason. So it’s gotta be useful. If we look closely, and kind of expand the macros, removing some non-essentials, that function boils down to this.

static const ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo* __GetTestMethodInfo_DummyAssert()
{
    // removed lines above for simplicity.
    static ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo s_Info = {::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo::TestMethod, NULL, &s_Metadata};
    s_Info.method.pVoidMethod = static_cast<::Microsoft::VisualStudio::CppUnitTestFramework::TestClassImpl::__voidFunc>(&DummyAssert);
    return &s_Info;
}

We can see that they are statically allocating a MemberMethodInfo class, and then, they set something called a pVoidMethod member on that, then the return the address to that. The signature of the GetTestMethodInfo_DummyAssert, function returns a const MemberMethodInfo*. We can see now that we’re getting somewhere, this function captures a pointer to the actual test method. So the algorithm we want is something along the lines of.

1. Use our tool set to scan for the MethodMetadata
2. Load the library into memory
3. Load the __GetTestMethodInfo_X function, by its decorated name in the metadata
4. Call this function, to return us the MemberMethodInfo
5. Make our call on the method.pVoidMethod function pointer

Could it be so simple? Unfortunately, not. If you recall from our simple example, we used free functions. What I mean by free functions, is that they are functions that aren’t associated with any data. I’m sorry what? This has nothing to do with the problem at hand.

Yay! Another history lesson.  If we compare procedural programming vs. object oriented programming. We can look at procedural programming as a set of defined functions where we enter into one, and it calls another, and another so on and so forth. There is procedure to its logic. Where as with object oriented programming, we have this concept of a class or object, that gets instantiated, and has a set of methods that operate on it, they may act on other objects, so on and so fourth. Thus, it can be harder to follow having object A calling object B. etc. etc. However, the two principles aren’t all that different, if you look at it with a different perspective. You can actually model object oriented programming in a procedural language. You do this by creating some data model, and a set of functions that work against it. Consider this C-like code.

struct Person
{
   char *name;
   int age;
};

void construct(Person *p, char *name, int age)
{
     p->name = malloc(strlen(name));
     strcpy(p->name, name);
     p->age = age;
}

void printName(Person *p)
{
    printf("My name is %s", p->name);
}

void destruct(Person *p)
{
    free(p->name);
}

As you can see, this looks a lot like a simple C++ class. You have a constructor, destructor and a simple printName function. You notice that each function operates on a Person*. I didn’t invent this pattern, or discover it. In fact, this was the beginnings of C++. Of course, C++ has come a long way. But still at it’s core, classes or objects are a set of functions, that operate on a chunk of data. Class functions in C++, take a pointer to that data, the instance, as their first argument. When I said that we only worked on free functions, our library example only worked against functions that were not being called on object instances. Our void functions in our test methods, act against the test class instance. Therefor, we can’t just “call” the function outright, or bad things will happen, demons could fly out of our noses. We don’t want that. It has to work with an instance of our data, our class. So that means, we need to actually create an instance of our class first.

So, in order to do that we need to know what our class is. This is really getting complicated. Let’s look at that file again, to see if we can glean some details.

// This is a part of the VSCppUnit C++ Unit Testing Framework.
// Copyright (C) Microsoft Corporation
// All rights reserved.

///////////////////////////////////////////////////////////////////////////////////////////
// Macro to define your test class. 
// Note that you can only define your test class at namespace scope,
// otherwise the compiler will raise an error.
#define TEST_CLASS(className) \
ONLY_USED_AT_NAMESPACE_SCOPE class className : public ::Microsoft::VisualStudio::CppUnitTestFramework::TestClass<className>

...

#pragma pack(push, 8)
    struct TestClassInfo
    {
        TestClassImpl::__newFunc pNewMethod;
        TestClassImpl::__deleteFunc pDeleteMethod;

        const ClassMetadata *metadata;
    };
#pragma pack(pop)

...

template <typename T>
class TestClass : public TestClassImpl
{
    typedef T ThisClass;

public:
    static TestClassImpl* CALLING_CONVENTION __New()
    {
        CrtHandlersSetter setter;
        return new T();
    }

    static void CALLING_CONVENTION __Delete(TestClassImpl *p)
    {
        CrtHandlersSetter setter;
        delete p;
    }


    // assume method matches this pointer
    virtual void __Invoke(__voidFunc method)
    {
        typedef void (ThisClass::*voidFunc2)();
        voidFunc2 method2 = static_cast<voidFunc2>(method);

        CrtHandlersSetter setter;
        (static_cast<ThisClass *>(this)->*method2)();
    }

    static EXPORT_METHOD const ::Microsoft::VisualStudio::CppUnitTestFramework::TestClassInfo* CALLING_CONVENTION __GetTestClassInfo()
    {
        ALLOCATE_TESTDATA_SECTION_CLASS
        static const ::Microsoft::VisualStudio::CppUnitTestFramework::ClassMetadata s_Metadata = {L"TestClassInfo", reinterpret_cast<const unsigned char*>(__FUNCTION__), reinterpret_cast<const unsigned char*>(__FUNCDNAME__)};

        static const ::Microsoft::VisualStudio::CppUnitTestFramework::TestClassInfo s_Info = {&__New, &__Delete, &s_Metadata};
        return &s_Info;
    }

    static EXPORT_METHOD const ::Microsoft::VisualStudio::CppUnitTestFramework::TestDataVersion* CALLING_CONVENTION __GetTestVersion() 
    {
        ALLOCATE_TESTDATA_SECTION_VERSION
        static ::Microsoft::VisualStudio::CppUnitTestFramework::TestDataVersion s_version = { __CPPUNITTEST_VERSION__ };

        return &s_version;
    }
};

Here, we see the same pattern. This method __GetTestClassInfo(), has ClassMetadata which has the decorated name to the __GetTestClassInfo() method. We can load that method and call it. From there this TestClassInfo object, has pointers to a __newFunc and __deleteFunc. This was the key to unlocking our success! We can see the finish line now. The macro, TEST_CLASS, ensures that you derive from the template class TestClass<T>. It uses CRTP to be type aware of our class, and defines two static functions. One that returns a TestImpl* called __New(), which creates a new instance of T (our type) and the other deletes it __Delete(TestImpl*). It also defines a function called __Invoke(__voidFunc) which invokes a void method, against ‘this’. TestImpl is defined as.

// This is a part of the VSCppUnit C++ Unit Testing Framework.
// Copyright (C) Microsoft Corporation
// All rights reserved.

class TestClassImpl
{
public:
    TestClassImpl() {}
#ifdef FEATURE_CORESYSTEM
    virtual ~TestClassImpl() {}
#else
    virtual ~TestClassImpl() noexcept(false) {}
#endif

    typedef TestClassImpl* (CALLING_CONVENTION *__newFunc)();
    typedef void (CALLING_CONVENTION *__deleteFunc)(TestClassImpl *);

    typedef void (TestClassImpl::*__voidFunc)();

    virtual void __Invoke(__voidFunc method) = 0;

protected:
    struct CrtHandlersSetter
    {
    typedef void (__cdecl *INVALID_PARAMETER_HANDLER)(const wchar_t* pExpression, const wchar_t* pFunction, const wchar_t* pFile, 
    unsigned int line, uintptr_t pReserved);

        CrtHandlersSetter()
        {
            if(IsDebuggerAttached())
            {
                debuggerAttached = true;
                return;
            }
            
            debuggerAttached = false;
            // Suppress the assert failure dialog.
            oldReportMode = _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
            oldReportFile = _CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDERR);
            // Set the handler
            oldInvalidParameterHandler = _set_invalid_parameter_handler(reinterpret_cast<INVALID_PARAMETER_HANDLER>(InvalidParameterHandler));
        }
        
        ~CrtHandlersSetter()
        {
            if(debuggerAttached)
            {
                return;
            }
            
            _CrtSetReportMode(_CRT_ASSERT, oldReportMode);
            _CrtSetReportFile(_CRT_ASSERT, oldReportFile);
            _set_invalid_parameter_handler(oldInvalidParameterHandler);
        }

private:
        // Check if a debugger is attached.
        __declspec(dllexport) static bool __stdcall IsDebuggerAttached();
       // The handler for invalid parameters
        __declspec(dllexport) static void __cdecl InvalidParameterHandler(const unsigned short* pExpression, const unsigned short* pFunction, const unsigned short* pFile, 
    unsigned int line, uintptr_t pReserved);
    
private:
        _invalid_parameter_handler oldInvalidParameterHandler;
        int oldReportMode;
        _HFILE oldReportFile;
        bool debuggerAttached;
     };
};

So __Invoke(__voidFunc), is a pure virtual function, that will allow us, using the miracle of polymorphism, to make the call into our class, from just a pointer to TestImpl.

You can see that we have everything we need.

1. Load the binary metadata, and find the test method we want
2. Determine which class it exists in
3. Load the ClassMetadata, to get the decorated name of the __GetClassInfo() function
4. Call __GetClassInfo() to retrieve the class info
5. Use the pointer to __New(), to create an instance of the class
6. Use the decorated GetMethodInfo() name, to call GetMethodInfo()
7. Use the method info, to __Invoke(__voidFunc), on the instance of TestClassImpl we created earlier.
8. Success.

So, I went and did just that. I made the test executor do the steps above. It invoked the method, the test ran. I was over the moon. It was time to fix my first issue. I hurriedly wrapped the __Invoke(__voidFunc) call in a try/catch. Like this

// MsCppUnitTestAdapter.cpp:210 
void VsTestAdapterExecutionContext::ExecuteMethod(System::String ^methodName, TestResult ^r)
{
    ResultRecorder cb(r); 
    static_cast<ResultReporterExceptionHandler*>(handler_)->Reset(&cb);
    try
    {           
        auto className = context_->Info().GetClassNameByMethodName(MarshalString(methodName));
        // load the test class, from the method name, this will load the class into the execution context, but it's name if it doesn't exist, it'll load it.
        TestClass_ *tc = nullptr;
        if (!stdx::find_if(*classes_, tc, FindByClassName(className)))
            tc = context_->CreateClass(className); // if you didn't call load, you won't get the class initialize.

        tc->Reset(); // this will reset the class i.e. create a new instance
        tc->InvokeMethodSetup();
        cb.OnStart();
        tc->InvokeMethod(MarshalString(methodName));
        cb.OnComplete();
        tc->InvokeMethodCleanup();
    }
    catch (const std::exception &e)
    {
        cb.OnError(System::String::Format("Uncaught C++ exception. {0}", gcnew System::String(e.what())));
    }
    catch (...)
    {
        cb.OnError("Unknown C++ Exception");
    }
    static_cast<ResultReporterExceptionHandler*>(handler_)->Reset();
    
}

I didn’t go into any detail about how I ended up getting the class name from the function name. The simple answer is that I parse the class name, from the function name. I also didn’t go into detail, about setup / teardown of the classes and modules. The above snip does some of that, as well does some reporting about the tests. I’ll admit it’s a bit messy, but it works. You can see that I capture std::exceptions and print the error. Now, if a C++ std::exception escapes the test method, my framework will catch it and print the error.

By this point, I was over the moon. I had done something harder than I thought I could and I really pushed my understanding of how these things worked. I had run some tests, and I was getting green lights in the Test Explorer window. I had let some std::exceptions escape, and saw the tests were failing correctly. I made sure the exception information was displayed in the test window. Time to try some negative assertion tests. I setup a test that had a bad assertion, something like.

TEST_METHOD(willAssert)
{
    Assert::AreEqual(3,4, L"They're not equal"); 
}

Each time I ran ‘willAssert’, the test would stay semi-opaque, as if I hadn’t run it at all. When I watched the Task Manager, the test execution engine process would disappear when I ran the test. Oh no. 

I put my head into my hands. I have no idea what I’m doing.

I hope that Part 2 of this series was equally as entertaining as the first part. I actually really loved putting the execution part of this code together. It was such a puzzle. Stay tuned for the next piece of the puzzle, where we explore Structured Exception Handling.

“Magic lies in challenging what seems impossible” — Carol Moseley Braun

Happy Coding!

References

Predefined Macro Definitions

 

 

The one where we reverse engineered Microsoft’s C++ Unit Test Framework (Part 1)

Have you ever really been interested in Microsoft’s C++ Unit Test Framework? I mean really interested? Like where you’d go to great lengths to figure out how it works. Well, I have… The story goes back a few years, maybe 3 or so.

At this point in my career I was deep into C++ development, and I had fallen in love with unit testing. I had fallen in love with Visual Studio’s Test Window, the ease it allowed me to get quick feedback about my development. My defacto standard was the built-in C++ Unit Test Framework. It was accessible. I didn’t need to install anything (past VS), and I didn’t need to download anything special. The project templates came built in, it was the easiest path to unit testing my work. I loved it. However, as with many things in life, there are always the ‘little things’ you have to learn to love.

My main gripe, was that if I wrote a function and a std::exception would escape, for whatever reason, the test would fail with the message “An unexpected C++ exception occurred”. Thanks Microsoft, for this useless information… I put up with it. I would use tactics like wrapping my calls in try/catch, I even wrote a new header that made an extension to the TEST_METHOD macro that would make a function level try/catch. It wasn’t enough for me, I could not believe that this wasn’t built in. For instance, if an Exception escapes in the C# test framework, you get the data about the Exception. This is a novel idea, so why doesn’t it work in C++? My second major stumbling block, was that if you didn’t have the right dependencies in the right directory. You would get an error, on all your tests, something along the lines of “Failed to setup execution context.” Also, a very very helpful error. This was the straw that broke the camels back. The amount of times I had run into this, the amount of times that junior developers scrambled to figure it out. It was too much. Something had to be done. Rather than divorce myself from Microsoft’s framework, and use something like boost::test, like so many had said I should do. I decided to do the sane thing, and just write my own. Not write my own, like re-invent the wheel. Just write my own test executor. I already had all the tests written, I didn’t want to redo that work in some new framework. I wanted to just build my own engine to run the tests I already had. My thought was that if someone at Microsoft could build it, so could I. They’re humans too — I think. Armed with only naive curiosity, my trusty Visual Studio, and the internet. I set out to do just that.

Where do I even start? Let’s just start at discovering the tests. How can we discover what tests are available in the binary? Well, if you’re familiar with the C# Unit Test framework, defining test classes and methods is done with attributes, similar to the C++ macros. My thought is that the C# Test Discoverer, must use reflection, look for the attributes, discovering the test classes and methods. I don’t know this for sure, but I would bet that is the case. Cool. Well, apart from some third party libraries, there’s no built in reflection in C++. So that can’t be the case for the C++ tests, can it? Maybe they load the assembly and have it tell the discoverer what tests are available? That’s what I would do if I engineered this.

Stop for a minute. Let’s reflect.

I said that my second problem with the framework, was that when you tried to run the tests and the dependencies couldn’t be loaded, you would get the error “Failed to load execution context”. Now — let’s think about this. If you’re able to see all the tests, yet the assembly can’t be loaded due to missing dependencies. How are we able to see what tests are in the binary? Magic! That’s how. Just kidding — I don’t believe in magic. It means that they’re not “loading” the library, which means that information about the tests, lives somewhere in metadata in the binary… Reflection… Could it be???

Well, the magic was right there in front of us the whole time, if you’re using the framework. The magic lies in the ‘ CppUnitTest.h’ header file. It took me a few beers, and a few hours to figure out just exactly WTF they were doing in there. It was essentially like trying to decipher Cuneiform .

If you’re unfamiliar, a typical TEST_CLASS and TEST_METHOD(s) looks like this.

#include "CppUnitTest.h"

TEST_CLASS(DummyClass)
{
    TEST_METHOD(DummyAssert)
    {
        /// My Code Under Test Here
    }
};

If you build and discover this, you’ll end up with a test class named DummyClass and a test in your Test Window, that says DummyAssert. So the magic lives in that TEST_METHOD macro. We will ignore the TEST_CLASS for now. Let’s look at TEST_METHOD. This is the macro, pulled directly from ‘CppUnitTest.h’

///////////////////////////////////////////////////////////////////////////////////////////
//Macro for creating test methods.
#define TEST_METHOD(methodName)\
    static const EXPORT_METHOD ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo* CALLING_CONVENTION CATNAME(__GetTestMethodInfo_, methodName)()\
    {\
        __GetTestClassInfo();\
        __GetTestVersion();\
        ALLOCATE_TESTDATA_SECTION_METHOD\
        static const ::Microsoft::VisualStudio::CppUnitTestFramework::MethodMetadata s_Metadata = {L"TestMethodInfo", L#methodName, reinterpret_cast<const unsigned char*>(__FUNCTION__), reinterpret_cast<const unsigned char*>(__FUNCDNAME__), __WFILE__, __LINE__};\
\
        static ::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo s_Info = {::Microsoft::VisualStudio::CppUnitTestFramework::MemberMethodInfo::TestMethod, NULL, &s_Metadata};\
        s_Info.method.pVoidMethod = static_cast<::Microsoft::VisualStudio::CppUnitTestFramework::TestClassImpl::__voidFunc>(&methodName);\
        return &s_Info;\
    }\
    void methodName()

Okay — so humour me and let’s ignore the __GetTestClassInfo(); and __GetTestVersion(); calls and look to the line ALLOCATE_TESTDATA_SECTION_METHOD, which if we scan a little higher in the file is found here.

///////////////////////////////////////////////////////////////////////////////////////////
//Macros for creating sections in the binary file.
#pragma section("testvers$", read, shared)
#pragma section("testdata$_A_class", read, shared)
#pragma section("testdata$_B_method", read, shared)
#pragma section("testdata$_C_attribute", read, shared)

#define ALLOCATE_TESTDATA_SECTION_VERSION __declspec(allocate("testvers$"))
#define ALLOCATE_TESTDATA_SECTION_CLASS __declspec(allocate("testdata$_A_class"))
#define ALLOCATE_TESTDATA_SECTION_METHOD __declspec(allocate("testdata$_B_method"))
#define ALLOCATE_TESTDATA_SECTION_ATTRIBUTE __declspec(allocate("testdata$_C_attribute"))

But what does it all mean Basil? Well, without diving into too much history, we need to at least know about Windows’ binary formats. If you didn’t already know, every form of executable “binary” on the Windows platform is in the format of a Portable Executable (which was an extension of the COFF format). This is what allows the operating system to load and run executables, dynamic libraries, etc. It’s a well defined format, see the Wiki link above if you don’t believe me. A PE file looks like this.

Portable_Executable_32_bit_Structure_in_SVG_fixed

I’m not going to explain everything in this image, only the relevant information. If you look down just passed the DOS STUB, on the very right (your right) you’ll see a 2 byte number called #NumberOfSections, this tells us the count of sections in the binary. That’s something we care about. I know this, because I know they’ve made sections where the data lives. I know this, because of the

#pragma section("testdata$_B_method", read, shared)

and the

#define ALLOCATE_TESTDATA_SECTION_METHOD__declspec(allocate("testdata$_B_method"))

Then, if you look at the bottom, you’ll see the ‘Section Table’. It means, that from the COFF Header, the offset of the Optional Header, there lives N sections in the Sections Table. In there, we will find our “testdata$_B_method” section, and in there, we will find SOMETHING! Are you bored yet? Because when I got this far, you couldn’t pull me away. I was like a 13 year old watching my first R rated movie. What did they store in there? What was it used for? The only thing I could do, is dive a little deeper. My best bet, was that these MethodMetadata were stored in that section.

ALLOCATE_TESTDATA_SECTION_METHOD\
static const ::Microsoft::VisualStudio::CppUnitTestFramework::MethodMetadata s_Metadata = {L"TestMethodInfo", L#methodName, reinterpret_cast<const unsigned char*>(__FUNCTION__), reinterpret_cast<const unsigned char*>(__FUNCDNAME__), __WFILE__, __LINE__};

It would be a block of data, that would contain a bunch of strings. The first being a wide character string, “TestMethodInfo”, the next a wide character string of the ‘methodName’ defined in the macro, the next a character string of the __FUNCTION__, next the string of __FUNCDNAME__, a wide character string of the filename __WFILE__ , and lastly the __LINE__. (If you’re interested in a list of Predefined Macros there you go.)

This was my assumption, but I couldn’t know for sure unless I saw it with my own two eyes. But how do I do that? Well there are a few third-party tools that will dump the PE (I’ll let you figure out what to search…), but I needed to write my own tool anyways so I just jumped in feet first. A few quick Bing searches (just kidding I used Google), and I found out I needed to open the binary as a flat file, and then map that file into memory. From there, I could get a pointer to the start of the file and use some macros, structures and functions in Windows.h to move about this file.  The “pseudo” algorithm is as follows

1) Open the binary as a flat file
2) Map the binary into memory
3) Obtain a view of the map (a pointer to the map)
4) Navigate the PE to understand file type, and number of sections
5) Iterate through each section definition until we find the correct section
6) Using the mapping offset, along with the section definition, find our data

There we go, simple as that. Let’s try it. The memory when we do that, and we point to the section table, looks like this.

0x07440210  2e 74 65 78 74 62 73 73 00 00 01 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0 00 00 e0 2e 74 65 78 74 00 00 00 82 15 02 00 00 10 01 00 00 16 02 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 60 2e 72  .textbss............................ ..à.text............................... ..`.r
0x07440262  64 61 74 61 00 00 c3 bd 00 00 00 30 03 00 00 be 00 00 00 1a 02 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 40 2e 64 61 74 61 00 00 00 cc 0e 00 00 00 f0 03 00 00 0c 00 00 00 d8 02 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 c0 2e 69 64 61  data..Ã....0......................@..@.data...Ì....ð.......Ø..............@..À.ida
0x074402B4  74 61 00 00 86 29 00 00 00 00 04 00 00 2a 00 00 00 e4 02 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 40 2e 6d 73 76 63 6a 6d 63 79 01 00 00 00 30 04 00 00 02 00 00 00 0e 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 c0 74 65 73 74 64 61  ta...).......*...ä..............@..@.msvcjmcy....0......................@..Àtestda
0x07440306  74 61 49 06 00 00 00 40 04 00 00 08 00 00 00 10 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 50 74 65 73 74 76 65 72 73 09 01 00 00 00 50 04 00 00 02 00 00 00 18 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 50 2e 30 30 63 66 67 00 00  taI....@......................@..Ptestvers.....P......................@..P.00cfg..
0x07440358  04 01 00 00 00 60 04 00 00 02 00 00 00 1a 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 40 2e 72 73 72 63 00 00 00 3c 04 00 00 00 70 04 00 00 06 00 00 00 1c 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 40 2e 72 65 6c 6f 63 00 00 69 1a  .....`......................@..@.rsrc...<....p......................@..@.reloc..i.
0x074403AA  00 00 00 80 04 00 00 1c 00 00 00 22 03 00 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 42 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ...€......."..............@..B....................................................
0x074403FC  00 00 00 00 cc cc cc cc cc e9 86 36 01 00 e9 41 78 01 00 e9 9c 33 00 00 e9 b7 93 01 00 e9 b3 73 01 00 e9 7d b4 00 00 e9 68 a8 00 00 e9 13 62 00 00 e9 7e 76 00 00 e9 09 6f 01 00 e9 a4 59 00 00 e9 8f e4 00 00 e9 aa 9d 01 00 e9 98 73 01 00 e9 ce ab

So what gives??? I don’t see any section called “testdata$_B_method”.  I can however see a ‘testdata’ section. At this point no amount of research other than this anecdotal evidence, leads me to believe the ‘$’ is some kind of delimiter on the section name. I guess we have to assume. We assume that the “testdata” section will contain our test method metadata. The problem is now, there are other things that sit in this section. There’s class, method, and attribute metadata. So, if it’s all lined up in a single section, how do we decipher what is what? Meaning, if we’re just trying to use pointers to walk around, how will we ever know what type we’re pointing to?

Did anything strike you as odd about the MethodMetadata structure? Maybe, if I show you the structure definitions of all the metadata objects, you might see something.

struct ClassMetadata
{
    const wchar_t *tag;
    const unsigned char *helpMethodName;
    const unsigned char *helpMethodDecoratedName;
};

struct MethodMetadata
{
    const wchar_t *tag;
    const wchar_t *methodName;
    const unsigned char *helpMethodName;
    const unsigned char *helpMethodDecoratedName;
    const wchar_t *sourceFile;
    int lineNo;
};

struct ModuleAttributeMetadata
{
    enum AttributeType { MODULE_ATTRIBUTE };
    const wchar_t *tag;
    const wchar_t *attributeName;
    const wchar_t *attributeValue;
    AttributeType type;
};

struct ClassAttributeMetadata
{
    enum AttributeType { CLASS_ATTRIBUTE };
    const wchar_t *tag;
    const wchar_t *attributeName;
    const void *attributeValue;
    AttributeType type;
};

struct MethodAttributeMetadata
{
    enum AttributeType { METHOD_ATTRIBUTE };
    const wchar_t *tag;
    const wchar_t *attributeName;
    const void *attributeValue;
    AttributeType type;
};

Huh. If you look carefully, the first actual member of these structures is a wchar_t* called tag. Then if we go to our use of it.

static const ::Microsoft::VisualStudio::CppUnitTestFramework::MethodMetadata s_Metadata = {L"TestMethodInfo", L#methodName, reinterpret_cast<const unsigned char*>(__FUNCTION__), reinterpret_cast<const unsigned char*>(__FUNCDNAME__), __WFILE__, __LINE__};

You might notice, that there’s a L”TestMethodInfo” set as the tag, so one could deduce, dear Watson that that is how we can decipher our different metadata components. By their tag! Let’s readjust our rudders, and fly! If we get our ‘testdata’ section, then move to the area with the data, we should see a bunch of nicely spaced wide strings in memory, right? Wrong!

0x07471120  94 43 03 10 b8 43 03 10 d8 43 03 10 40 44 03 10 f8 44 03 10 67 00 00 00 00 00 00 00 94 43 03 10 6c 46 03 10 98 46  ”C..¸C..ØC..@D..øD..g.......”C..lF..˜F
0x07471146  03 10 08 47 03 10 f8 44 03 10 78 00 00 00 00 00 00 00 94 43 03 10 cc 47 03 10 08 48 03 10 80 48 03 10 f8 44 03 10  ...G..øD..x.......”C..ÌG...H..€H..øD..
0x0747116C  7e 00 00 00 00 00 00 00 94 43 03 10 28 4b 03 10 68 4b 03 10 e0 4b 03 10 f8 44 03 10 8b 00 00 00 00 00 00 00 94 43  ~.......”C..(K..hK..àK..øD..........”C
0x07471192  03 10 c8 4c 03 10 08 4d 03 10 80 4d 03 10 f8 44 03 10 95 00 00 00 00 00 00 00 94 43 03 10 4c 4e 03 10 78 4e 03 10  ..ÈL...M..€M..øD..........”C..LN..xN..
0x074711B8  e8 4e 03 10 f8 44 03 10 9d 00 00 00 00 00 00 00 94 43 03 10 dc 4f 03 10 20 50 03 10 a0 50 03 10 f8 44 03 10 a7 00  èN..øD..........”C..ÜO.. P.. P..øD..§.
0x074711DE  00 00 00 00 00 00 94 43 03 10 70 51 03 10 98 51 03 10 08 52 03 10 f8 44 03 10 af 00 00 00 00 00 00 00 94 43 03 10  ......”C..pQ..˜Q...R..øD..¯.......”C..
0x07471204  78 53 03 10 b0 53 03 10 28 54 03 10 f8 44 03 10 bb 00 00 00 00 00 00 00 94 43 03 10 0c 55 03 10 48 55 03 10 c0 55  xS..°S..(T..øD..».......”C...U..HU..ÀU
0x0747122A  03 10 f8 44 03 10 c3 00 00 00 00 00 00 00 94 43 03 10 dc 56 03 10 08 57 03 10 78 57 03 10 f8 44 03 10 cc 00 00 00  ..øD..Ã.......”C..ÜV...W..xW..øD..Ì...
0x07471250  00 00 00 00 94 43 03 10 6c 58 03 10 b8 58 03 10 38 59 03 10 f8 44 03 10 d2 00 00 00 00 00 00 00 94 43 03 10 30 5a  ....”C..lX..¸X..8Y..øD..Ò.......”C..0Z
0x07471276  03 10 60 5a 03 10 d0 5a 03 10 f8 44 03 10 de 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ..`Z..ÐZ..øD..Þ
0x0747110A  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

What the hell? I was 100% expecting there to be a string sitting pretty right there in my section. Back to the drawing board I guess… Let’s recap, the first member of that struct is a wchar_t *. Huh. The first member of the struct, is a wchar_t *. What does that even mean? It means, that it is a ‘pointer to a wchar_t’. A ‘pointer’ to a wchar_t! Oh right! A pointer! I remember from school that a pointer is just an address! So where we were expecting there to be some text sitting pretty, was garbage, Or so we thought. Wrong again. It is a POINTER! That means sitting in that location is an address! An address to where though? It should be an address to my string, but where on Earth (quite literally) would that address be? It has to be somewhere that is constant, right?

If we study the sections of a PE fie, there’s a section called ‘.rdata’. Microsoft defines this section as “Read-only initialized data”. Thinking back a moment these are magic strings, (heaven forbid), aka ‘static strings’, static is read-only. If I was to hazard a guess, it probably means that they live somewhere in that section, because the compiler has to put magic string somewhere… So that garbage number, is probably a pointer to a string somewhere in that ‘.rdata’ section. So, if we take that address, and adjust it for where the section data lies, we can find the “TestMethodInfo” string.

0x07462D94  54 00 65 00 73 00 74 00 4d 00 65 00 74 00 68 00 6f 00 64 00 49 00 6e 00 66 00 6f 00 00 00 00 00 00 00 00 00 44 00  T.e.s.t.M.e.t.h.o.d.I.n.f.o.........D.
0x07462DBA  75 00 6d 00 6d 00 79 00 41 00 73 00 73 00 65 00 72 00 74 00 00 00 00 00 00 00 00 00 00 00 43 50 50 55 6e 69 74 54  u.m.m.y.A.s.s.e.r.t...........CPPUnitT
0x07462DE0  65 73 74 49 6e 76 65 73 74 69 67 61 74 6f 72 54 65 73 74 3a 3a 6e 65 73 74 65 64 3a 3a 44 75 6d 6d 79 43 6c 61 73  estInvestigatorTest::nested::DummyClas
0x07462E06  73 3a 3a 5f 5f 47 65 74 54 65 73 74 4d 65 74 68 6f 64 49 6e 66 6f 5f 44 75 6d 6d 79 41 73 73 65 72 74 00 00 00 00  s::__GetTestMethodInfo_DummyAssert....
0x07462E2C  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3f 5f 5f 47 65 74 54 65 73 74 4d 65 74 68 6f 64 49 6e  ....................?__GetTestMethodIn
0x07462E52  66 6f 5f 44 75 6d 6d 79 41 73 73 65 72 74 40 44 75 6d 6d 79 43 6c 61 73 73 40 6e 65 73 74 65 64 40 43 50 50 55 6e  fo_DummyAssert@DummyClass@nested@CPPUn
0x07462E78  69 74 54 65 73 74 49 6e 76 65 73 74 69 67 61 74 6f 72 54 65 73 74 40 40 53 47 50 42 55 4d 65 6d 62 65 72 4d 65 74  itTestInvestigatorTest@@SGPBUMemberMet
0x07462E9E  68 6f 64 49 6e 66 6f 40 43 70 70 55 6e 69 74 54 65 73 74 46 72 61 6d 65 77 6f 72 6b 40 56 69 73 75 61 6c 53 74 75  hodInfo@CppUnitTestFramework@VisualStu
0x07462EC4  64 69 6f 40 4d 69 63 72 6f 73 6f 66 74 40 40 58 5a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  dio@Microsoft@@XZ

Voila! At last we finally found it. We found a ‘TestMethodInfo’ (if you’re wondering why it’s T.e.s.t.M.e.t.h.o.d.I.n.f.o…, it’s because it’s a wide character string so each character is 2 bytes. Unicode amirite?).

To recap, we’ve loaded the DLL into memory, mapped it and walked to the section. We taken the tag pointer, adjusted the address to find it in the ‘.rdata’ section, now we know that we’re looking at the MethodMetadata structure. So, we can take the original pointer, and cast that to a MethodMetadata object.

struct MethodMetadata
{
    const wchar_t *tag;
    const wchar_t *methodName;
    const unsigned char *helpMethodName;
    const unsigned char *helpMethodDecoratedName;
    const wchar_t *sourceFile;
    int lineNo;
};

Then, for each of the other members, which are pointers to strings in the .rdata section we can just adjust and capture the same way we did the tag. In fact, the compiler did us a favour, and laid them out so you can see them in the memory dump above. Next, we just advance our section pointer the distance of the size of a MethodMetadata, and we are able to get the next test name!!! (This is mostly true, I’ve glossed over some details of the other things that can be in the section)

I really hope you’re thinking “Damn! That was so much fun!” Because I definitely was! You can see now, that the steps to tie this into a test adapter aren’t too far away. I won’t get into that until a later post, but as for this post, we’ve uncovered how to discover what tests lie in a Microsoft C++ Unit Test Framework created test DLL. Wasn’t that just a blast digging into that?

I hope you will join me for the next part, where we figure out how to actually load and execute the tests from this metadata.

If you’re interested in seeing this code, my cobbled together project lives here. I apologize in advance for the code. I had a really bad habit of putting as many classes as I could in a single file. I don’t do this anymore. It was out of laziness.  The PeUtils.h/.cpp and CppUnitTestInvestigator.h/.cpp, in the CppUnitTestInvestigator project have the code for loading the DLL metadata.

“It’s still magic, even if you know how it’s done” — Terry Pratchett

Happy Coding!

 

** As a disclaimer, I don’t own any of the code snips above, they are directly pulled from the VsCppUnit C++ Unit Testing Framework, which is Copyright (C) Microsoft Corporation **

 

References:
PE Format
Section Specifications
DUMPBIN

Template<T> vs. Generic<T>

The other day I was discussing the differences between C++ templates and C# generics. In my opinion, C++ templates reigns supreme. Not because I’m one of those guys with a “My compiler compiles your compiler.” shirt. But because in general abstractions in C++ tend to ‘cost’ lest than the same abstractions in C#. For instance, the foreach exists in both languages, but the iterator concept is far lighter weight then the IEnumerable concept. Both of these concepts make use of generics to be able to support iterating a collection of “things”.  Having one single ‘concept’, that works across a multitude of different containers, allowing for traversal. The iterator concept is a bit more in-depth, in that it allows for more than forward traversal unlike IEnumerable.

The discussion came down to “why?”, why I thought that the template system in C++ was better than that of C# generics. Why is right. At the time, my statement was rather off the cuff, and somewhat tongue-in-cheek. Granted I had my reasons, but not backed by research. Only backed by my overview knowledge of the operations of the two systems. So now that I’ve had time to mull it over, and solidify my arguments. I’ll explain my point-of-view, grounded in my initial thoughts.

Let me ask you something? When was the last time someone wrote Tetris “in” C# generics? No, I don’t mean “with” C# generics, I mean “in” C# generics.

Spoiler: the answer is never.

Why? Let’s explore reason numero uno, which admittedly wasn’t the first reason to my head at the time. The C++ template system is Turing complete. That’s right, it’s Turing complete. Wait what does that even mean? Well, the definition for Turing complete is along the lines of  “A system of data-manipulation, is said to be Turing complete, when it can simulate a Turing machine.” Thank you Wikipedia for that ever so enlightening statement. Basically, what this means is that the meta-language built into the C++ template system, is actually a full blown programming language. One where if you’re brave enough, or crazy enough, you can even implement Tetris. Remember, that the C++ template engine runs at compile time, so you guessed it — it’s compile time Tetris.

You don’t need to use the template system only for silly things like Tetris. You can also use it for actually useful things as well. Things like specific implementations of your generic algorithms based on traits about a type. Or how about even finding traits about a type?  You can use the template system, to choose different implementations based on the type you’re using, or rather traits about the group of types you’re using. As well as do computations at compile time.

template <int N>
struct sum
{
    static const int value = N + sum<N-1>::value;
};

template<>
struct sum<0>
{
    static const int value = 0;
};

int main()
{
    std::cout << sum<5>::value << "!\n";
}

Figure 1 – A simple example of a summation recursive template meta-program

All of this for “free” at compile time. When I say free, I mean run-time free. The C++ template engine does its work at compile time, and will make heavy use of the type system. Don’t get me wrong, it’s not all roses. If you make heavy heavy heavy use of the template system, you can have really long compile times and even bloat your binaries. I’ve seen code generation times upwards of 30 minutes. The code actually only took about 5 or so minutes to compile, but the code generation was ~30 minutes. Imagine writing all those classes, TGFCPPT (Thank God for CPP Templates).  So at this point, C++ templates have delivered a knock-out punch to C# generics. KO! Generics down for the count. However, I am still going to proceed with examining my initial reasoning, because I like to “beat dead horses” so to speak. I think it can be said that generics are like the template systems little brother. They look similar, but one is about half as useful and twice as fat.

So the second reason, and the first that came to my head (not surprisingly) was overhead. The C++ template system doesn’t have run-time costs, because it’s a compile time operation. Whereas the generic system must have some run-time overhead, just due to its nature.  I wouldn’t say I was stabbing in the dark, but I definitely didn’t have research to back up my claim. I had read about the generics system in a few blogs, I understood how it worked, sort of. Anyways, the claim was contended. Fair enough, I didn’t have the research in my back pocket to prove it. For most, it would’ve ended there. However, on my never ending quest to remove misinformation and search down the truth, I couldn’t let it go. So really, what is the cost of the C# generics vs. the C++ template engine?

First, lets look at how the C++ template engine actually works. To be honest, I couldn’t find all that much real information on how C++ templates work. So I will have to share my understanding, which admittedly is naive, but get’s the point across . When you have some code like the following.

template <typename T>
void print(T value)
{
    std::cout << value;
}

print(5);
print("Hello World");

The compiler will for all intents and purposes, generate two functions for you.

void print(int value)
{
    std::cout << value;
}

void print(const char *value)
{
   std::cout << value;
}

Now, there’s little value in this example. But as you can see, the compiler is basically saving you the keyboard time. The compiler will only generate those functions that are needed, so it doesn’t just go out and create functions for every type. Only the ones that need to be created. Given this happens compile time, by the time you’re running those template definitions are a thing of the past, and you’re running the code that was generated by the compiler. These functions each have their own unique addresses, and their own ability to be optimized.

Now, compare this to how C# generics work. Consider List<T>, you could have instantiation for List<int>, List<string>, even List<Object>. How does this work?

Well, for a minute we can brainstorm a couple of different ways this could be achieved.

  • We could do something similar to the C++ template engine, and for every unique instantiation we could generate IL. Meaning that List<int>, List<string>, and List<object> would each get their own set of instructions.  This could be referred to as code specialization.

Let’s be honest with ourselves, this isn’t a good approach here. It would lead to HUGE binaries, and would be hugely inefficient. Given that C# objects are references, the code would be duplicated for a large portion of instantiations.

  •  We could ignore the type of the target. Effectively erasing it from our implementation. I don’t care if it looks like a duck, or quacks like a duck, it’s a duck. Then we only need to maintain ONE version of the code, and at run-time, we just ignore the type so the List<int>, List<string>, and List<Object> all use the same instructions. They’re just ignorant to the type.

This has issues too, because since we’ve erased the type, we’ve lost that information about the objects. Then we need to do extra work, to make sure that List<Object> != List<string>. As a note, this is the approach JAVA takes to its generics. The benefit to this approach, is that we only have one master set of IL for the implementation. There’s not duplication, like the approach above. It also means, that you need to treat POD types, that are stack based, as heap values.

  • We could use one shared implementation like JAVA, but at the same time use run-time techniques to manage our types. So this way we’re respecting the types of the objects. So List<int>, List<string>, and List<Object> all would use the same code, except there would be some overhead to store the information about the types.

This last approach is the approach that C# takes for their generics. Now, this isn’t exactly 100% true, because I believe that C# generics use a mixture of code specialization, as well as code sharing. Because when we talk about “objects” vs. “primitives”, in C# one is heap allocated and referenced, and the other lives on the stack. Therefore, if you treated something like an int the same as a string, you would need to do what is called “boxing” and allocate that int on the heap. Which is mega overhead and not cheap. So there is a combination of code specialization and code sharing going on.

One of the goals of the design of generics, was that it should be as efficient as code that is hand specialized. Using these two techniques, the implementation is equally as efficient as hand specialized code, and much much more efficient that type erasure. This is due to the boxing thing I mentioned earlier. That’s a lofty goal, and I’ll say an admirable one, taking from the spirit of C++. However! There are definitely other costs that are paid, the cost of memory overhead for storing the vtables, and other RTTI data. Making generics, not exactly free. Clever techniques for loading this information saves execution time, but we still have to pay for the memory used. Contrary to popular belief, memory isn’t always free. Although, at the C# level, it can pretty much be assumed to be. Like C++ code generation, C# generics can also basically explode when being evaluated. In one blog that I read, the author mentions that a single instance of List<T> will explode out into 28 different types, where 5 unique List types could cost upwards of 70K in the working set.

It’s not to say that C# generics don’t have some neat features. Specifically type constraints, and the ability to constrain types to interfaces. This is something that can be done with C++ templates (slightly easier post C++11), but the reality is that it’s not for the faint of heart. The second upside of this, is that the compiler errors you get when you violate a type constraint on a generic, they’re readable by a human. One of the downsides of C++ template usage and definitely of template meta-programming, is that the compile errors are pretty much non-coherent for anyone who isn’t a compiler writer or demigod.

In conclusion, I still think that C++ templates are far superior to C# generics. That said, they both have their benefits and costs. It is like anything, you need to understand the cost of something before you use it, and you should always pick the right tool for the job.

“Price is what you pay. Value is what you get.” — Warren Buffet

As always — keep calm and code on.

Happy Coding!

 

References:

[1] On generics and (some of) the associated overheads 

[2] Design and Implementation of Generics for the .NET Common Language Runtime

[3] Turing Completeness – Wikipedia

 

The Zero Cost Abstraction of Iteration

I’ve been hooked on C++ for about 10 years now. Since CMPUT101, I’ve been attracted to the syntax of the language, the simplicity of use, and later the driving ideals behind the language. C++’s author Bjarne Stroustrup, stands alone when it comes to preaching language ideals and pushing the language forward all while respecting these ideals. This is no easy task, especially when the language is designed by a committee. I’ve always been convinced one of the key reasons C++ is such a large success, and has remained so timeless; is the fundamental guiding principles behind the language. The principle that I believe reigns supreme is the ‘Zero Cost Abstraction’.

What is the ‘Zero Cost Abstraction’? Also called a ‘Zero Overhead Abstraction’. Well — it’s an abstraction, that costs nothing. Not like monetary currency; or a favour. Instead resource or runtime cost. Bjarne’s definition is “an abstraction that imposes no space or time overhead in excess of  what would be imposed by careful hand coding of a particular example of the abstraction.” So in other words, an abstraction that doesn’t use any more resources than if you carefully crafted the assembly yourself.

Well then, what is an ‘abstraction’? Well an abstraction is a concept that you can work with, that ignores the details of operation. A good metaphor for this is a steering wheel on a car. You turn the wheel left and right, to turn left and right, but you’re not concerned with the details of actually moving the wheels of the car on the road. In fact, you don’t want to be concerned with those details at the level of driving.  You want to be more focused on the bigger task at hand — driving the car.

So abstractions in computing allow us to work with concepts, craft algorithms and structure code, so that we don’t have to worry about the details. Programming languages themselves are abstractions, they allow us to work with logical operations, classes, English words, and common symbols to create instructions that cause computer hardware to do things. Languages like C and C++, are considered ‘close to the metal’, this means that they only thinly abstract the assembly that is run on the processor, only thinly abstract the management of memory (either directly to the MMU or through the OS), and only thinly abstract the other pieces of hardware. Languages like C# and JAVA (shudder), are higher level languages, meaning they have many layers between the code and the actual instructions that execute on the CPU. These higher level abstractions give way for more flexibility, but often have a cost. In the case of JAVA, the flexibility of portability, at the cost of run-time and space overhead. Garbage collection adds the flexibility of not having to worry about memory management, at the cost of run-time, and space overhead. Not to mention the cost to developers, as the become brainless in the areas of memory lifetime, and ownership rules.

[Coming Soon – a post on Virtual Machines]

So what does any of this have to do with the ‘Zero Overhead Abstraction’ principle? Well, obviously I have a story to go along with it. So some time ago I was working on my library [ASPeKT] and I was soliciting the help of the internet for advice on how to work better with the IL I was crafting. The comments I got, were, well, unexpected to say the least. The majority of the criticism I received was my use of direct for loops for iteration. As opposed to the cooler C# foreach loop or even better LINQ-to-everything!! Now, I can take critiquing of my code, after all that’s how you learn. So I pondered this for quite some time. I reviewed some of the pull requests which had transformed some of my boring for-loops, into beautifully crafted LINQ queries. I wasn’t buying it. I just couldn’t. It hid some of the intent of what I was trying to do, and call me old school, but I found it harder to read.

Enter foreach, this one I liked. C++11 introduced new semantics for it and I liked the clarity that it gave. When you’re iterating a collection using an indexer, the first thing you do is get the item at the index. The foreach, just abstracts this and your boundary checking in a nice syntax package. Color me smitten. Now, I didn’t actually refactor much in my library to use foreach. Honestly, I like it but I wrote the code already and it worked. However I started using foreach, pretty much everywhere I could, just like I did in C++ now. I know that C# probably had this before, and I obviously adopted it backwards, but I do a lot of shit backwards. So sue me. 

Fast forward a couple of months, I’m reading “Writing High-Performance .NET Code” by Ben Watson. He’s got a little section on loops for iteration. The section I think was on comparing C++ to C# for performance. When we’re talking speed of execution, less instructions is more. The less instructions you have to run, the faster your program will be. The fastest program you can write, is one that does nothing.

There was a snip on comparing a loop in C# to a loop in C++, and the generated ASM instructions. He was making the statement that C# doesn’t ALWAYS have much overhead over a language like C++. He showed how old skool for loop iteration, was virtually the same instructions for C++ and C#. But the foreach, it was something else. I was floored, I had never really thought about it, but at that moment I did. What does a foreach in C# actually do?

His example was something like this

[C++]
int sum2()
{
    int range[4] = { 1, 2 , 3 , 4};
    int sum=0;
    for(auto i=range; i<range+4; ++i)
    {
        sum+=*i;
    }
    return sum;
}

[ASM]
//-O2 Disabled so it wouldn't optimize the loop out
sum2(): # @sum2()
  push rbp
  mov rbp, rsp
  lea rax, [rbp - 16]
  mov rcx, qword ptr [.L_ZZ4sumvE5range]
  mov qword ptr [rbp - 16], rcx
  mov rcx, qword ptr [.L_ZZ4sumvE5range+8]
  mov qword ptr [rbp - 8], rcx
  mov dword ptr [rbp - 20], 0
  mov qword ptr [rbp - 32], rax
.LBB1_1: # =>This Inner Loop Header: Depth=1
  lea rax, [rbp - 16]
  mov rcx, qword ptr [rbp - 32]
  add rax, 16
  cmp rcx, rax
  jae .LBB1_4
  mov rax, qword ptr [rbp - 32]
  mov ecx, dword ptr [rax]
  add ecx, dword ptr [rbp - 20]
  mov dword ptr [rbp - 20], ecx
  mov rax, qword ptr [rbp - 32]
  add rax, 4
  mov qword ptr [rbp - 32], rax
  jmp .LBB1_1
.LBB1_4:
  mov eax, dword ptr [rbp - 20]
  pop rbp
  ret
.L_ZZ3sumvE5range:
  .long 1 # 0x1
  .long 2 # 0x2
  .long 3 # 0x3
  .long 4 # 0x4
[C#]
static class Program
{
     static int sum()
    {
          int[] range = new int[]{ 1, 2 , 3 , 4};
          int sum=0;
          for(var i=0; i<4; ++i)
          {
                 sum+=range[i];
           }
          return sum;
   }

}

[JIT ASM]
Program.sum()
    L0000: push ebp
    L0001: mov ebp, esp
    L0003: push edi
    L0004: push esi
    L0005: mov ecx, 0x717168e2
    L000a: mov edx, 0x4
    L000f: call 0x58f3200
    L0014: lea edi, [eax+0x8]
    L0017: mov esi, 0x251c084c
    L001c: movq xmm0, [esi]
    L0020: movq [edi], xmm0
    L0024: movq xmm0, [esi+0x8]
    L0029: movq [edi+0x8], xmm0
    L002e: xor esi, esi
    L0030: xor edx, edx
    L0032: mov ecx, [eax+0x4]
    L0035: cmp edx, ecx
    L0037: jae L0049
    L0039: add esi, [eax+edx*4+0x8]
    L003d: inc edx
    L003e: cmp edx, 0x4
    L0041: jl L0035
    L0043: mov eax, esi
    L0045: pop esi
    L0046: pop edi
    L0047: pop ebp
    L0048: ret
    L0049: call 0x73a52b10
    L004e: int3

As you can see 25 instructions in C++ vs.  30 instructions for C#. So your standard for loop in C++ and C# are running essentially the same instructions.

The foreach was a different story — he wouldn’t even show the instructions. He only showed the generated IL, and let me tell you, it was a lot.

In my own research, the ASM generator that I used, generated 150+ instructions, which didn’t seem like a lot. However, I did notice is that it didn’t generate ASM for the actually calls to MoveNext(), it just used the syntax ‘call’ MoveNext(). Which likely has a lot more instructions under the hood.

Let’s compare that to the pure C++, range foreach.

[C++]
int sum()
{
    int range[4] = { 1, 2 , 3 , 4};
    int sum=0;
    for(auto i : range)
    {
        sum+=i;
    }
    return sum;
}

[ASM]
sum(): # @sum()
  push rbp
  mov rbp, rsp
  mov rax, qword ptr [.L_ZZ3sumvE5range]
  mov qword ptr [rbp - 16], rax
  mov rax, qword ptr [.L_ZZ3sumvE5range+8]
  mov qword ptr [rbp - 8], rax
  mov dword ptr [rbp - 20], 0
  lea rax, [rbp - 16]
  mov qword ptr [rbp - 32], rax
  mov rax, qword ptr [rbp - 32]
  mov qword ptr [rbp - 40], rax
  mov rax, qword ptr [rbp - 32]
  add rax, 16
  mov qword ptr [rbp - 48], rax
.LBB0_1: # =>This Inner Loop Header: Depth=1
  mov rax, qword ptr [rbp - 40]
  cmp rax, qword ptr [rbp - 48]
  je .LBB0_4
  mov rax, qword ptr [rbp - 40]
  mov ecx, dword ptr [rax]
  mov dword ptr [rbp - 52], ecx
  mov ecx, dword ptr [rbp - 52]
  add ecx, dword ptr [rbp - 20]
  mov dword ptr [rbp - 20], ecx
  mov rax, qword ptr [rbp - 40]
  add rax, 4
  mov qword ptr [rbp - 40], rax
  jmp .LBB0_1
.LBB0_4:
  mov eax, dword ptr [rbp - 20]
  pop rbp
  ret

Hmmm…. 30 instructions. So as you can see, this is an illustration of a ‘Zero Cost Abstraction’. The code has been shrunk by quite a few characters. It becomes easier to read and reads with more intent, yet it doesn’t carry any overhead over the original loop.

So, what’s the development cost of writing your own foreach enumerable collection in C#. Implementing the IEnumerable concept.

Well — first you need to implement the IEnumerable interface, which returns an IEnumerator. The IEnumerator class is really the meat and potatoes. It contains your iteration state and a reference to your collection. You need to implement the MoveNext() call to increment the enumerator, which returns true or false based on whether it incremented. Then you need to implement Reset(), which resets the enumerator. You need to implement the ‘Current’ field to return the current object.

Vs.

What’s the development cost of making something enumerable in C++. Implementing the iterator concept.

You need to implement the iterator concept. Iterator concepts are described here, the minimum you need is the ForwardIterator. Which you need to be able to read and increment with multiple passes. The iterator concept models the pointer, so you need to implement operator++() and operator*(), and operator==(). Like the IEnumerator the iterator is the brains of iterator (fancy that), and contains some reference to the collection as well as iteration state.

In order for your collection to work with C++ foreach, you need to implement a begin() and end() method, which return an iterator.

begin() => returns an iterator to the first object
end() => returns an iterator of one past the last object

It’s as easy as that.

The moral of this story, just because an abstraction makes the code cleaner, doesn’t mean you’re not paying for it, someway. Of course, the flip-side is, if there is careful thought put into the abstraction — you can eat your cake and have it too.

“Small leaks sink big ships” – Benjamin Franklin

Happy Coding!

PL