Adventures cross-compiling a Windows game engine

As part of my game development major at DAE, I have to work on several projects which were not made with support for my platform of choice (Linux). Thankfully, most of these have been simple frameworks wrapping around SDL and OpenGL, so my job was limited to rewriting the build system from Visual Studio's .sln project file to a cross-platform CMake project (and fixing some bugs along the way). Not too bad. I'd miss the beginning of the first class, but was up and going shortly after. Among these were the first two semesters of Programming. Here's a list of school engines I have ported so far:

  1. Programming 1 “SDL Framework”: https://git.allpurposem.at/mat/SDL-Framework
  2. Programming 2 “GameDevEngine2”: https://git.allpurposem.at/mat/GameDevEngine2
  3. Graphics Programming “RayTracer”: https://git.allpurposem.at/mat/GraphicsProg1
  4. Gameplay Programming “_FRAMEWORK”: https://git.allpurposem.at/mat/GameplayProg

The versatility of having a cross-platform project allowed me to add tons of niceties for some of these. The one I'm most happy with is the “GameDevEngine2” framework from Programming 2, to which I added web support and ended up using it for my and 2FoamBoards's entry in the 2023 GMTK game jam.

Programming 3

I'd been having it easy. A couple nonstandard Microsoft Visual C++ (MSVC) bits of syntax here, a couple win32 API calls (functions that are specific to Windows) there... I wasn't expecting what arrived in my downloads folder today. I applied my usual CMake boilerplate, with SDL support, hit run to see the perhaps 50-100 errors... and instead was greeted with a simple but effective singular error.

apm@apg ~/S/Prog3 (main)> clang++ source/GameWinMain.cpp 
In file included from source/GameWinMain.cpp:9:
source/GameWinMain.h:12:10: fatal error: 'windows.h' file not found
#include <windows.h>
         ^~~~~~~~~~~
1 error generated.

Oh, no

There's no SDL. There's no OpenGL. No GLFW, Qt, or GTK. It's all bare Windows API calls. I think I was in some form of state of disbelief, as I spent the next 30 minutes slowly creating #defines and typedefs to patch in all the types. Maybe, just maybe, I could patch around the types and it would magically open a window and I could get started with my classwork. No such thing happened.

Options

So: what are my options? Is this salvageable, without having to boot the dreaded virtual machine? Let's see... I could:

Obviously the first two options would be preferable, as they don't come with a hard dependency on the unfamiliar world of WINE. However, they sadly also take the most time. I have not yet discarded the second option (the author of the engine gave me the green light to rewrite it for native Linux, and even use it in exams (that's a first!!)), but as I have to follow the class from the start, I think I'll be going with WINE.

aur/msvc-wine-git

Of course, I'm not the first person to want to build a .sln project from Linux. This appears to be a solved problem, with the polished-looking msvc-wine toolchain available as a native package for my distro. So I went ahead and installed it:

apm@apg ~/S/Prog3 (main)> gimme msvc-wine-git
[sudo] password for apm: 
:: Resolving dependencies...
:: Calculating conflicts...
:: Calculating inner conflicts...

Aur (1) msvc-wine-git-17.7.r4-2

:: Proceed to review? [Y/n]:

It diligently fetched MSVC, the Windows 11 SDK, and all the necessary components from Microsoft's servers, while I had time to read the documentation. I happened upon the CMake instructions, which is how I've managed all my school-related projects so far, and it didn't stick in my brain. I don't intend to criticize the writing, but something about it being all the way in the bottom in a FAQ, with no code blocks or example commands, or having a class going on around me while I was doing this prevented me from understanding how I'm supposed to use it. The only time I've ever used a separate toolchain was Emscripten; it provides a nice little emcmake wrapper for CMake which takes care of a lot of the details for you. I gave it a few tries, but seeing I was getting nowhere, and every second was lost class time, I decided to move on to my last option.

LLVM

I knew a little about LLVM before this, from having used clangd as my language server for C++ projects. As I understand it, it's a group of compilers designed in such a way that the “frontends” (which read the text code and output an intermediate language) and “backends” (read intermediate language and output the final binary) are swappable and interchangeable. This means you can use the same backend to compile both C++ and Rust code, while still getting equally well-optimized machine code out the other side. I enlisted the help of @JohnyTheCarrot@toot.community, who I knew has worked with clang before. He told me about the concept of an “LLVM triple”, which is a setting for LLVM compilers that tells it what sort of machine you want it to output code for. Crucially, you can specify a triplet for a completely different system than your own, and it should still work. I tried the following command:

clang++ -target x86_64-w64-mingw32 source/*.cpp -o game

This currently outputs 227 linker errors. I know there were many syntax-related compiler errors which I've since fixed, but it does get us past the dreaded #include windows.h! All of the linker errors take the following form:

/usr/bin/x86_64-w64-mingw32-ld: /tmp/GameEngine-ac27d8.o:GameEngine.cpp:(.text+0xc95f): undefined reference to `__imp_DeleteObject

Fun with the linker

Each of these is related to a call of a Windows-related function. It looks like we're missing the libraries! Adding the -mwindows flag tells Clang it's compiling & linking a GUI Windows app, instead of a command line one. This causes linking against a lot of win32 GUI-related functions, reducing the linker errors to a mere 9. There's two kinds:

At first, I assumed I'd have to get these from a copy of Windows. However, I remembered WINE has a lot of open source reimplementations of these DLLs (Windows's version of .so shared libraries), and sure enough locate msimg32.dll (note the lowercase: I wasted some time with this because Linux is case sensitive, while Windows is not!) pointed me straight to a DLL I could yoink. I added it to the list of files to compile, and the msimg32-related linker errors were gone. Hooray!

...or so I thought. I excitedly copied in winmm.dll and tried to compile...

clang-16: error: unable to execute command: Segmentation fault (core dumped)
clang-16: error: linker command failed due to signal (use -v to see invocation)

Excuse me?? The linker is segfaulting?? To be honest, I have no idea whether this is an actual bug in LLVM's linker, but it sure did stump me for a while. I thought maybe my copy of winmm.dll was corrupt, or WINE did something weird with it. I went as far as downloading Microsoft's version of the DLL, but was met with the same sad message. What could I be possibly doing wrong?

Oh. I'm not supposed to be copying the DLLs into here, am I? The last time I used a linker without going through CMake, I was passing libraries to it was -l<libname>. But it can't be that easy for this... can it? It'd have to go to my default WINE prefix to fetch them, which sounds plain weird. Libraries come from system paths, not user-specific folders. Well, might be worth a try anyways...

apm@apg ~/S/P/build (main)> clang++ -mwindows -target x86_64-w64-mingw32 ../source/*.cpp -o game -lmsimg32 -lwinmm
In file included from ../source/GameWinMain.cpp:10:
../source/GameEngine.h:19:9: warning: '_WIN32_WINNT' macro redefined [-Wmacro-redefined]
#define _WIN32_WINNT 0x0A00                             // Windows 10
        ^
/usr/x86_64-w64-mingw32/include/_mingw.h:239:9: note: previous definition is here
#define _WIN32_WINNT 0xa00
        ^
1 warning generated.
Warning: corrupt .drectve at end of def file
Warning: corrupt .drectve at end of def file
Warning: corrupt .drectve at end of def file
apm@apg ~/S/P/build (main)> ls
game.exe*

wait. That built?? HUH???? There's no way it—

apm@apg ~/S/P/build (main)> ./game.exe
-snip-
0130:err:module:import_dll Library libgcc_s_seh-1.dll (which is needed by L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe") not found
0130:err:module:import_dll Library libstdc++-6.dll (which is needed by L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe") not found
0130:err:module:LdrInitializeThunk Importing dlls for L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe" failed, status c0000135

Right. Not so fast, heh. Still, this is great news! I don't know how or why this works, but we're linking to the DLLs somehow somewhere. WINE can't find some mingw32 libraries which were pulled in by -mwindows, but we can easily point it to them with export WINEPATH="/usr/x86_64-w64-mingw32/bin"

And that's it! Here's the engine in all its glory, with audio support and all! It's beautiful...

A screenshot of a completely black window with many lines of warnings from WINE behind it

Right, there's nothing built on it yet. It's just a blank canvas. But hey, it doesn't crash!

What's next?

Having this run through WINE does come with a few limitations:

Long-term, depending on the course workload and how complex the engine functions end up being, I think I will rewrite it in SDL. This will have the added bonus of enabling, like with my other engine ports, web support (see my Programming 2 end project here and a game jam game made in the same engine here). However, I think this will take longer than I think is reasonable to spend while procrastinating on other classes, so I'm leaving it here. I wrote down my process while it was still fresh in my mind, so I hope this was an interesting read! As always, any and all constructive feedback is welcome directed to me: @mat@mastodon.gamedev.place .

I am considering writing up my general porting process in a separate blog post, so perhaps expect that next!

Addendum

After doing some additional research, and asking around in the very helpful WineHQ IRC room, I found a way to get debugging working! The first step is adding the -g flag to the clang++ invocation, which tells clang we want it to generate debug information (namely source maps, so the debugger can show which line of code we're at). Then I simply have to run winedbg --gdb game.exe, and I am presented with a (nearly) full-featured gdb prompt!

A screenshot of a gdb interface showing source code of a WinMain function which runs the game engine

I'm unsure how to hook this up to neovim (maybe I can look into the Debug Adapter Protocol for this?), but for now just having a gdb environment is awesome enough. Unto more adventures!

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at