Moving a big software project to a new compiler can be a lot of work, and few projects are bigger than Chromium. In addition to the main Chromium repository, which includes all of Blink, there are over a hundred other open-source projects which Chromium incorporates by reference, totaling more than 48,000 C/C++ files and 40,000 header files. Porting Chromium on Windows to VC++ 2015 requires getting all of these projects to build and run.

As of March 11th 2016 Chromium for Windows has switched from VC++ 2013 to VC++ 2015, and it doesn't look like it's switching back. This will give us more C++ 11 features, new security options, much faster LTCG/PGO builds, and other advantages.

The tracking bug for this project currently has over 330 comments on it, with contributions from dozens of developers. Writing about all of those changes would require an entire book. So I'm going to focus on my favorite part of this project – compiler bugs. In particular, bugs where the compiler silently generates incorrect code.

Update: another code-gen bug in VS 2015 was found on March 31st, and worked around about two weeks later. A VC++ bug was filed. And another silent-bad-code-gen bug was found in September 2016, and worked around a few days later.

If you throw a diverse enough set of code at a compiler, build it in lots of different ways, run lots of tests, and ship the code to customers, you're going to find some bugs.

Because Chromium is open source I will include links to Chromium bugs and fixes where possible, in addition to links to the VC++ bug reports.

I didn't find all of these bugs, but it was my job to investigate them, come up with a minimal repro, and report them to Microsoft. And I have to say that the Microsoft team was amazing. They were very supportive, and helpful, and it was clear that they really wanted VC++ 2015 to be as good as possible, and that includes building Chromium. We had an excellent symbiotic relationship: I fed them high quality bug reports, and they fixed them incredibly quickly.

Anyone can file VC++ bugs at the connect website. Getting your bugs noticed is easier if you work for Google, but anyone who files quality bugs can get help. The VC++ MVP program lets ordinary developers who participate in the VC++ community get privileged access to the VC++ team. Recommended – I enjoyed being a VC++ MVP before joining Google.

Chromium is open source so Microsoft can theoretically reproduce any of these bugs if I just tell them what source file to compile and where to look. But that can be a slow process. If the repro steps for your bug includes "Download 90,000 source files and our custom build system and compile for hours" then you can't expect priority service.

This is especially true because most reports of bugs in compilers are erroneous – most 'compiler bugs' are actually bugs in the code being compiled, such as invoking undefined behavior. So, I always had to prove to myself that these were compiler bugs before reporting them, which usually meant simplifying the repro case to make it trivially obvious that the compiler was at fault. I've discussed techniques for doing this before.

Failed Chromium

After getting all of Chromium building the next step was to start running Chromium. It would run fine once, but on subsequent runs I'd hit an assert. I suspected a code-gen bug but it was not the case. The problem ultimately turned out to be this code:

HandleWrapper h(CreateMutex(…));

if (h.HandleIsInvalid() && GetLastError() == SOME_ERROR_CODE) …

It turns out that the HandleWrapper class allocates memory, and the debug version of the VC++ 2015 CRT allocations functions call the FlsGetValue function, and FlsGetValue calls SetLastError(0) because it is so incredibly proud that it succeeded. SetLastError(0) destroys the error code from CreateMutex, and madness ensues. I filed a bug and tweeted about this and it turned into a fun little twitter discussion with opinions about evenly split between "if memory allocation affects LastError then Armageddon is imminent" and "if you assume memory allocation won't affect LastError then you are Satan".

This way of using HandleWrapper is very convenient and is pervasive in Chromium so fixing all of the uses seemed impractical so instead the HandleWrapper constructor now preserves and then restores LastError. Some other misuses of GetLastError were also fixed.

This was not a compiler code-gen bug, but I wanted to include it because the twitter storm was wonderful.

Microsoft decided to fix the bug which I assumes means that they are choosing Satan over Armageddon.

Failed test

