[bq]: the tablet was made by BQ, a defunct Spanish company which, coincidentally, was the first to launch a Ubuntu Touch device: https://www.zdnet.com/article/first-ubuntu-smartphone-aquaris-e4-5-launches-into-cluttered-mobile-market/

Why I'm leaving my comfort zone

So: what's Google done this time to drive me off of Android? Maybe it's the recent lack of punctuality of AOSP source code releases, being several months late and thus preventing alternative Android distributions from staying up to date (and now as of writing this post, they will reduce the AOSP source code update frequency from quarterly to only twice per year?? this does not bode well, yikes). Maybe it's their (for now slightly less bad) attempt to require passport verification & payment to distribute apps even outside of their Play Store? Perhaps it's their repeated sabotage of open source apps' ability to function?

Yes to all, but I will blame this little green boi: In case you don't know (so lucky), this is a feature introduced by Google for Android 12 which helpfully pops up to tell you when a (non-Google Play Services) app accesses your location. While on the surface the location indicator is a good idea, there is no user-facing whitelist and thus the Home Assistant or weather apps, both of which are FOSS and I thus fully trust, would cause the green dot to pop up with all its animations for a few seconds, every five minutes. It literally drove me crazy, and was the last straw for me to finally consider an alternative.

The distro

Now, there's a few distributions of Linux for phones, but with my goal of moving away from Google I did not want to settle for something that still depends on Android, otherwise we'd have the Chromium problem. This means that all the distros that depend on Halium to run a Linux userspace don't qualify, despite being the most likely to have hardware working given they use the Android drivers and kernel. That leaves me basically with only one option, that being the Alpine-based postmarketOS. I got to try a device running it at last year's FrOSCon and found its performance very impressive, though I got some fair warnings about missing drivers for stuff like the fingerprint reader. I also read @neil@mastodon.neilzone.co.uk's excellent blogposts about his experience trying it on a OnePlus 6 phone. I asked around the @postmarketOS@treehouse.systems Matrix rooms, and my awesome local Aachen hackspace @CCCAC@chaos.social where I spotted someone with postmarketOS stickers on their laptop (if you're reading this, hi!), and finally settled on buying a Fairphone 5 to run pmOS on. The reason I did not install it on my existing Pixel 3a is twofold:

1. It has very little RAM. You can run pmOS on 3GB, but it doesn't feel very future-proof given the state of the web (also, I don't know how heavy Waydroid is)

2. I don't want to sacrifice my existing Android install. What if I need to do something that only works on Android (foreshadowing)? What if I delete important files while flashing pmOS?

I definitely think Mobile NixOS is worth a look for its immutability & reproducibility, which makes a lot of sense on a smartphone. They don't list my device on their Devices List, but I'm sure I could get it working like pmOS given enough tinkering. NixOS gives me this safety cushion of being able to easily rollback my entire system if something goes wrong. Meanwhile, every time I touch a system file on postmarketOS I feel like I'm committing a crime, and every update I do feels like a gamble on whether my phone will keep booting or not. For now, I'm waiting for postmarketOS Duranium to become usable, and then make a decision whether to distro-hop based on the project state, since I'll have to reinstall anyways.

Installing postmarketOS

I received my Fairphone 5 from a non-Amazon online store, skipped thru its Android setup, and directly accessed the hidden developer settings to unlock my bootloader. Fairphone, for whatever reason, has a convoluted step of inputting device data on their website to get a “Bootloader Unlocking Code”. I guess they want to track how many of their devices run unlocked… kind of uncool, as the phone's unlockability relies on them keeping this web tool up and running. I then went to follow the postmarketOS install instructions, but found there were multiple options. For example, if you want full-disk encryption (FDE), the “pre-built image” option does not provide it, and you must use the pmbootstrap CLI tool. Thankfully, the Fairphone 5 pmOS wiki page has instructions for this which I followed without issue…

…or so I thought! After the install, my phone booted showing the postmarketOS bootanimation (with Linux console output periodically eating away at the anim… looks kind-of broken but should be fixed once pmOS adopts Plymouth), asked for my FDE password, aaaand (:drums:)proceeded to get stuck on a black screen. Thankfully, I did this at CCCAC, and quickly got help troubleshooting. In the end, it turns out the “Plasma Mobile” package was broken, so I picked the other name that sounded like it'd have a usable UI: “GNOME Mobile”.

[!NOTE] This is the part where I want to be really really clear that I don't intend to create any negativity toward Linux mobile projects. I will be complaining about things quite extensively, but only because I think it is really important to highlight how the experience is as an end-user (and was asked to share it!). I have massive respect for the people that write the code that makes Linux on phones possible, and I hope to help make all this a reality. I cannot possibly be more excited about Linux Mobile right now, so much that I'm fully dedicated to using it day-to-day.

GNOME Mobile

OK, so I installed GNOME Mobile instead of just going with Phosh. I expected the UI would be somewhat similar to the Phosh screenshots I'd seen, but I was keen to explore different ways of interacting with a phone, and its name sounded more “upstream” than Phosh. GNOME is a high-quality desktop (if the defaults suit you), and in general I expect the defaults on a phone to be reasonable.

After booting and entering my FDE password, I am first greeted by the full “desktop” version of GNOME Display Manager (GDM), asking me which user I want to log in as. Once I select myself (the only option), a full QWERTY on-screen keyboard (OSK) pops up and I get to type my user password, which is supposed to be a PIN, so should just have a numberpad…

After inputting my PIN, it looks like GDM crashes to a black screen, but after a few seconds a familiar GNOME interface pops up! …and proceeds to (only sometimes) ask for my password again, twice (with different-looking modals), to unlock the GNOME Keyring. Thankfully, these two extra password prompts have gone away recently, so I can only assume the bug is fixed. There's a nice “welcome” program that explains some of the basics, and tells me it's meant for enthusiasts. Well, here I am :)

GNOME Mobile presents the usual GNOME Desktop app grid, with the ability to swipe between multiple pages. The usual quick settings tiles can be accessed by swiping down from the top, and notifications awkwardly pile up under it, with a very small scrolling area. When an app is open, the homescreen will show that window and move the search bar out of the way. It took a little getting used to, but I find that I quite like this interaction model!

The frame drops are entirely due to software encoding, normally it is snappier. Also the double Do Not Disturb button might be an extension messing with things, my bad!

What I like less is how the icons in the app grid behave…

Sometimes you can move icons, sometimes not. Making a folder is a nigh-impossible task (I must have tried a hundred times, and only succeeded three times in total). Also: perhaps I am imagining it, but sometimes I boot my phone and the icons have rearranged themselves, and other times some are just missing. I definitely lose apps every once in a while, but thankfully the search bar has my back to find them again.

When I installed the OS, there were two settings apps, one being GNOME's own Settings and the other being “postmarketOS Tweaks” which let me change some things like the hostname. The latter app has since disappeared, I think moved to the Phosh settings app in this merged MR. I have not yet looked into restoring it as I quite like the hostname I picked: hermes, messenger of the gods and himself the god of trickery, two things that represent this phone quite well.

The lockscreen is very usable. It shows a blurred version of my wallpaper, the time, and notifications I received (most without content… and no way to enable it). I can swipe up to show a PIN entry, which absolutely has to be 6 numbers. Sometimes it will eat some of the numbers I input and I have to try again but, when it works, it works well. The lockscreen does not allow poweroff or reboot. only suspend is available, which I could not figure out how to fix. Notifications on the lockscreen often say “Just now” regardless of when they were received.

Something really cool inherited from desktop GNOME is the “GNOME Online Accounts” feature, which allows signing into many different online services and integrate them to the OS. I added my self-hosted Nextcloud account and was happy to see it import everything. Tapping the date on the top-left shows my upcoming events and, when my modem is working, I can start SMS conversations in Chatty with my contacts imported from Nextcloud.

It's a little bit funny that all GTK apps seem to insist on showing me their keyboard shortcuts in menus, despite not having a keyboard attached. This makes dropdown menus much bigger than they need to be, but isn't a super big deal.

I found a couple ways to crash the shell, like closing an app while a popup or side menu is open, and apps themselves can crash when they try to do desktop things like lock the mouse cursor. However, when not doing weird stuff, the experience is pretty stable.

Hapticsn't

The first thing I noticed after interacting a little bit with the UI, is that there is zero haptic feedback. On Android, when you do certain actions such as typing or switching apps, the motor inside the phone makes a nice “buzz” as feedback. I didn't realize how much I came to rely on this until it was taken away from me. I was expecting it to work since the pmOS wiki page for my device says it works:

After asking in the GNOME Mobile Matrix room, I learned that there is a service called feedbackd, which other interfaces talk to but GNOME Mobile does not. I have to imagine then that haptics would work on Phosh or other UIs, but (this will be a recurring theme) GNOME Mobile wants to do it a different way, and thus… hasn't done it yet. I plan to work on this at some point, especially after some productive discussion with other GNOME Mobile devs, but I haven't gotten much further than asking for input from the XDG Portals folks. I aim to take the time to work on it further, especially talking with feedbackd developers, but other matters were more pressing thus far. I at least was able to verify that the vibration motor can work by talking to it directly with the kernel's force feedback API.

Auto-brightness

Normally this was going to go in the next section, but fortunately while writing this blogpost it started working! Unfortunately however, now I want it back off, and I can't find a way to remove it.

Anyways, auto-brightness is supported in GNOME upstream, but never showed up in my power settings as the article says it should. I asked for help in the Matrix room and we verified that net.hadess.SensorProxy detects and exposes my phone's light sensor, so I guess the gnome-settings-daemon was failing to pick this up. Not having auto-brightness means that I have to fumble to find the brightness bar every time I go outside, as I can't see what's on my screen otherwise. Now that it works, it adjusts extremely quickly to any small changes in ambient light, which unfortunately overrides my own settings. The toggle still isn't there, so I can't just turn it off. The biggest issue with auto-brightness, now that it's here, is that it animates this transition, which causes issues with my OLED display's driver: these manifest as brief flashes of horizontal bands of garbage pixels, or the color grading of the entire display changing, or (admittedly this one's pretty cool) the display showing several copies of GNOME in a grid, Andy Warhol-style.

Of course, while doing my final editing, auto-brightness is gone again, and I don't have to deal with the screen glitching anymore! I opened pma!4274 to track the display artifacts, so I know when I can try enabling auto brightness again :)

Smaller missing bits in the shell

Flashlight

There's no way to toggle the flashlight. I found a merge request to the shell that would add it, but with no activity on that repository in the last 8 months I'm not holding my breath for it to be merged. In the meantime I am running a GNOME Extension that adds this feature, which I had to manually install using the terminal as it does not appear to be packaged anywhere. I'll count this as the first required use of the terminal.

OLED support

GNOME Mobile ships a dark theme, but no option for a “pure black” background. Back on Android I distro hopped to LineageOS just for this feature. The GNOME “dark” theme is a light-grey which might make sense on the usual LCDs that desktops or laptops have, looks really bad on an OLED display, which most phones nowadays have. I tried researching how to theme GTK or “libadwaita” apps, since that seems to be mainly what my install came with, and only found references to a defunct app called “Gradience”. I was able to install a Flatpak of a fork that was updated slightly more recently, and navigated its desktop-only interface very awkwardly to set the background color to black. This worked for most apps thankfully, but I did not find a way to theme the actual GNOME shell, which unfortunately is still stuck as this ugly grey, and I've no idea where to begin to try fixing this on my device. There's a “User theme” extension that can apply CSS to the gnome-shell, but no indication on what the CSS would look like to fix the background color.

Qt apps functionality

I get that it's the GNOME shell, so I should be using GTK apps, but on the already limited Linux mobile app ecosystem, I need to be able to use apps written in other frameworks, even if they may look slightly different. I have two Qt apps installed at the moment: KDE Connect (integrate phone with desktop), and Kitinerary (find public transport routes). Both of them open with a blinding white background, with a very broken-looking interface. Tapping a text field does not bring up the keyboard (thankfully I can manually double-tap the gesture bar to get it). There is a window bar at the top with minimize, maximize, and close buttons. I haven't found a solution for these issues, but I did install “Kvantum” and “qt6ct” to at least change the background color. However, these apps still look & feel extremely broken and it makes me sad that my experience of their developers' hard work is ruined by the way they get shipped.

Copy & paste

It's not missing, but it might as well be, because the UI is so inconsistent and often buggy that I resort to manually typing over my crazy-long Bitwarden passwords. Though the Wayland clipboard works fine, each app seems to be expected to implement its own way to select text, copy, and paste. Libadwaita (GTK4) apps, which normally have pretty good UX, make it near-impossible to get the popup for copy/paste, and then the popup uses icons with no description, often leaving me guessing at their function (also it has a weird black outline). The “Text Editor” app has an especially tricky to use selection/popup interaction, probably on the account of allowing text input and those interactions conflicting with selection (it took me over 30s to get it to pop up for the screenshot):

On-screen keyboard customization

The default keyboard gets the job done for typing basic text, but (as far as I know) there is no option to change the keys available or how it works. On Android, I used and loved Unexpected Keyboard, which lets you type special characters via quickly “flicking” in many configurable directions from any given key. It also features a Ctrl key, which would entirely solve the copy-paste UI issue from above. It even binds a lot of desktop-like keys, like Escape, which is extremely useful when using e.g Vim for text editing. I am aware of a similar project for Linux mobile called Unfettered Keyboard, but unfortunately (I think) it cannot run on GNOME Mobile due to GNOME Mobile's keyboard being part of the shell, rather than a separate program. If I do stick with GNOME Mobile, I will probably learn to write extensions and see if I can write some sort of shim that lets you plug in other keyboard programs like Unfettered Keyboard.

Customization of the quick settings & statusbar

When I enable location support, there is a constant location icon in the statusbar reminiscent of that green dot which finally drove me off Android. The quick settings tiles also seem to randomly change what's available, such as auto-rotate appearing and disappearing between reboots, and a mysterious “Wired” connection that's always on taking up the first slot. On a big screen, I wouldn't mind as much since I have space to spare, but on a phone I need to save space by not showing useless icons, and I especially need things to stay where they are to build up any kind of muscle memory. Thankfully there is the amazing Quick Settings Tweaks shell extension that lets me hide the irrelevant toggles & icons, though it can't fix the rotation appearing and disappearing. Speaking of screen rotation…

Screen rotation

When auto-rotate is off, this really behaves like a desktop. You can go in the system settings and manually select “Portrait”, “Landscape Left”, “Landscape Right”, or “Portrait (Flipped)”, then click “Apply” and finally confirm “Keep changes”. Android did this thing where it still reads the sensor, and shows a button for a second when you physically rotate the phone which you can tap to actually do it. I always ran my Android phone like this to prevent accidental rotation, and I would love to see this on GNOME Mobile. Maybe that's a good first contribution if I decide to dive into UI stuff.

Rarely, I get lucky and an “Auto-rotate” toggle is in the quick settings. When it is enabled, rotation is instant the moment I tilt my phone. While I like the lack of animation, I do wish it were a little less sensitive as it's very easy to accidentally rotate. Thus, even in the rare event the feature is available, I keep it turned off. I think this issue is related to iio-sensor-proxy, as sometimes sudo systemctl restart iio-sensor-proxy brings it back, but other times this command gets stuck, so I am not sure.

Battery life

Normally this'd have gotten its own h2, but unfortunately the otherwise excellent battery life (which I estimate would last well over a day) gets completely trounced by two issues in (presumably) GNOME Mobile:

1. Since suspending on mobile is not really a thing, battery life relies on apps using as little power as possible at all times, and suspending features when not used. I'm not sure what part of the stack is responsible for this, but for example I would expect 3D rendering to not use power when the screen is off, or the camera app to suspend the camera hardware when unfocused. This is not the case right now, and for example I have had my phone die while going out because I forgot to swipe away the “Camera” app window, which caused it to stay on for several hours until the battery gave up. I'd love to learn more about this!

2. The gnome-shell process will randomly start consuming 100% of one CPU core, and not stop until restarted. I have to constantly feel my phone in my pocket in case it starts getting warm, and if so log out & back into the shell to prevent it eating the entire battery life. This is tracked by this issue, and two weeks ago I spent quite some time trying to figure out the root cause and collecting profiler data. Unfortunately I don't see a fix on the horizon, and if it keeps happening I might have to switch off of GNOME Mobile entirely.

Of note: my phone came with “suspend” enabled, which until recently would cause a kernel panic (I was quite confused why pressing the off button would cause a reboot!), so I disabled it in the GNOME Settings. Suspend is now fixed, but I don't think I can use it as it prevents all network-based apps from receiving notifications. Supposedly SMS & calls can still wakeup the device, but 95% of my communications go thru Matrix, and the remaining 5% are Signal, neither of which work while suspended.

Something strange I encountered is that the phone will not charge over a USB-A cable. On Android it would charge slowly, so I could leave the phone plugged into my desktop's USB port while developing apps and test on it, however postmarketOS requires having the phone plugged into a proper C-to-C cable connected to a USB-C power delivery capable power supply. The phone will also always show a notification prompt asking whether I want to transfer files via MTP (but doesn't actually do anything), “developer” (what?), or just charge, even if the cable is power-only.

I usually leave my phone charging next to my bed, and most of the time this works, however sometimes I will find the phone really hot and displaying the full-disk encryption password prompt, which I guess means it kernel panicked. This also happened a few times in my pocket, and the password prompt keeping the screen on likely contributed to quite some power loss.

Ok but what about the phone stuff

Yes yes, I'm getting there. I just have a lot to say about it even before inserting a SIM card!

The Fairphone 5 requires removing the battery to access the SIM slot, which at least they make very easy, but does force me to reboot to switch SIMs. These days I only use one though, so it shouldn't be a problem. pmOS detected my SIM and it showed up as “Mobile Network” in the Settings app, with the usual toggles expected from other OSes. I had to manually select the correct Access Point Name (APN) for my French cell operator, which I have experience with as I had to do the same on LineageOS (which actually was harder than on pmOS, since Lineage had me manually input all the settings!). Once that was set up, I saw for the first time a nice 5G icon in my statusbar, indicating mobile data works! It's also the first time I get to use the 5G I pay for, since my previous phone only supported 4G.

I can't seem to change the “Network Mode” option from its default setting of preferring 4G to one that prefers 5G: it lets me select it, but doesn't apply and shows me a popup that says Failed: reloaded modes (allowed '2g,... (and trails off). I can probably find out more by using journalctl, but it has not bothered me enough to do so.

What HAS bothered me is mobile data randomly becoming unavailable. One time it was caused by an update (that's what I get for running postmarketOS “edge”), which I reported and it got fixed in record time. Other times however it seems to happen quite randomly, like the auto-rotation disappearing (though less often thankfully!). This is usually remedied by a reboot, but is quite annoying. I'd expect a notification saying my network connection failed, and I do get these, but only when I leave the range of a wifi network (which is very common given I carry the phone around!) and some other random times it decides to send this, not when mobile data disappears.

Socials

The primary reason for carrying my phone around is being reachable, and reaching people when needed.

SMS

The install came with this very nice app simply called “Chats” (but the process name reveals it is actually Chatty) which allows me to send & receive SMS. It claims to support MMS, but it fails when I try to send one. The app also (!!!) let me log into my Matrix account and read messages in unencrypted rooms, sadly this last bit is not useful to me as most of my communications are encrypted. If you don't encrypt your messages however, like in SMS, this is an awesome app that runs well and does what it says. I did have some trouble sending SMS to new numbers, one of my family members did not receive a rather time-sensitive message, but I was having some similar-ish troubles on my Android phone so it might be a carrier thing.

Matrix

On the Matrix side, I first tried Fractal to have the GTK experience, however it unfortunately seems to run into similar crashing & freezing issues as what I reported when I tried Fractal on desktop. Thankfully I was delighted to see that the Matrix client I used on Android is packaged on Flathub! FluffyChat runs great and basic messaging worked without any issues.

There's what is likely a packaging bug that prevents it from loading an emoji font, so I can't see most emojis. There's a permanent bar at the top of the app that just says “FluffyChat”, and I think is the GTK app trying to draw client-side decorations (probably GNOME Mobile should tell apps to, uh, not do that). Unfortunately, the Flutter code behaves like the desktop version, and lacks several rather important features. A non-exhaustive list:

• can't play or record voice messages, have to manually download them then open in an audio player
• can't play videos, same deal
• no option to take a photo to send
• no notifications while the app is closed
• notifications for the room you're looking at get filtered even if the screen is locked or the app window is unfocused
• it makes every picture I send *extremely green* (flipping endianness I think):

I have fixed audio in a merge request, and I think video should be fairly simple to enable as well. I fear that taking photos will require some new XDG portal, so I'll be leaving that one for last. The notifications issue requires a UnifiedPush integration, which the Flutter package supports, but needs some work. I have a test version working on my desktop, but it is lacking a lot of logic for how to handle them. I hope they are merged quickly though, as I don't want to have to start to rebase a bunch of branches in a fork…

Signal

Signal is what I give to people who don't want to invest an hour in picking a Matrix server, client, figuring out encryption, and then not saving their recovery key. This means some of my extended family and contacts from work. Thankfully there is a Signal client for mobile Linux called Flare which can send and receive messages including images (though it strips EXIF metadata, which means some photos get sent sideways or upside-down). It can't handle calling, but I usually make Signal calls on my desktop anyways, so it's not a big deal.

Fediverse

I use Mastodon to access the Fediverse, and was very happy to discover @Tuba@floss.social. It implements basically everything I could want out of a Mastodon client, and looks pretty good while doing so. Just missing an OLED background, but I'm pretty sure that's on me for making such a messy GTK theme. I'd like to fix the background color at some point, though.

Sadly, Tuba frequently triggers some bug in the Vulkan driver that causes it to print “LOST_DEVICE”, and the app gets totally frozen midway through sliding a view out. I don't know where to report this, but it means I can't navigate the Fediverse for very long before I get stopped in my tracks. Another freeze which might be related occurs when I write a too-long post or attach a picture, it probably triggers some re-layout that hits a GPU bug and freezes. I unfortunately have lost several surely-banger-posts to this specific freeze. It also suffers from quite poor scrolling performance sometimes, potentially related to running out of Vulkan memory (I see that log message a lot).

E-Mail

I installed Thunderbird, which brought along an extension called mobile-config-thunderbird, and promises to make the UI more usable on phones. Unfortunately, something goes terribly wrong and it doesn't render my inbox at all, so it's not particularly useful as an email client right now. It does send me notifications though (as long as the app window is open!!!), so at least I can tap on one to read the email, since that does render.

On the topic of notifications

Yeah, it's quite important to be able to see when one of these apps wants my attention! Thankfully everything I'm running is FOSS, so there's no dark patterns to worry about here.

Push notifications

I'm not super qualified to explain this, but my surface-level understanding is that on both Android and iOS, there is a central server that the OS stays permanently connected to, and services you have apps for can “push” to that server, which then tells the OS to wake up the app so it can show you its notification. This heavily reduces power usage, and saves each app from implementing its own background service. On Android, this is implemented through Google Firebase Cloud Messaging, but thankfully an alternative exists in the form of UnifiedPush, which let me self-host my own push server that supporting services (Matrix and Mastodon, in my case) could use instead. This meant that Android apps like FluffyChat and Tusky didn't have to run in the background, but still showed me reliable notifications piped through my very own server, which my phone was always connected to.

On postmarketOS, I was very pleased to find a UnifiedPush wiki page, but was a little worried to see only a KDE-specific implementation, with just a single app listed as supported. Thankfully I was able to install kunifiedpush on GNOME Mobile and write a config file to make it connect to my self-hosted Ntfy server. It was all a little manual (and required terminal usage #2, probably due to me running it outside of its native KDE), but it means apps can now register to it and it actually delivers notifications, nice! I am able to receive notifications from the Fediverse via Tuba, which supports UnifiedPush, and as stated earlier I began work on FluffyChat support for UnifiedPush on its Linux builds.

Flare (Signal client) has an optional background service that keeps a connection to their servers, which is unfortunately required as Signal does not support UnifiedPush. SMS works fine as well.

Actually seeing the notifications

Man, I really really hoped this would work! Unfortunately I have some experience with upstream GNOME not really showing me all notifications, so I should have expected this. Even for apps that do consistently send notifications for messages, like FluffyChat and Flare, I will usually only see the first notification in a conversation, subsequent messages get “grouped” (which is a nice feature UI-side! but) which means I get no sound or pop-up for subsequent messages. GNOME also doesn't show me any notifications while fullscreen which, while I can understand the rationale, is not how I want it to work. This means that if I am watching a video fullscreen, I won't find out that my cooking timer has gone off until the video ends and I exit fullscreen!

Oftentimes “old” notifications get stuck, and also display wrong times. This happens with FluffyChat notifications quite frequently, where I open my phone and it says I received a message from my dad “Just now” or claims it came recently, when I actually had a full conversation hours ago.

Additionally, as explained earlier when talking about the lockscreen, it by default doesn't show the notification content for privacy reasons. I can enable showing content per-app in the GNOME settings, which would be great except it does not show every app, especially FluffyChat which is the one I actually need to be able to read quickly.

Pebble

Thankfully, I wear a Pebble smartwatch, and an amazing developer who goes by Muhammad maintains a Pebble connector app that can buzz my watch when I get a message (even when GNOME unwisely decides to hide the notification), like my watch used to do back on Android! Rockwork is an unofficial Pebble client for Ubuntu Touch, and with some work I was able to rebase an experimental non-Ubuntu-Touch backend for it written by Xela Geo. I abstracted some of the buildsystem further to make it usable as an Alpine package, and have been happily running Rockwork on my postmarketOS phone, with almost everything working. I opened a merge request to upstream, and if/once it is merged I hope to contribute my first package to Alpine.

I can control my music and read notifications on the watch, while opening RockWork lets me switch watchfaces and view historical step counter & sleep data. I cannot exaggerate how awesome this is. Of course though, it's not all perfect, though most of it can probably be blamed on my porting work (the app seems to work fine on its native Ubuntu Touch platform). I still need to get the app store and calendar sync to work, and there's a big problem with some apps using the XDG portal Notifications API, which GNOME implements privately and thus Rockwork can't eavesdrop on to forward to the watch. I don't know how I will solve this last one, and it currently means I don't get any SMS notifications.

Using the camera

One of the things that made me pick the Fairphone 5 over other similar devices is the “Partial” status of the Camera (rather than “Broken”). When I got the phone, I was excited to try out the camera, as I usually take lots of pictures of my cat different places I go. I didn't expect much given the rating, but I am mostly positively impressed at how well it works given the level of support. Using the built-in “Snapshot” camera app (which is the only one I got working), there is no way to change the focus or the zoom level, but you can take pictures and videos, as well as scan QR codes. The focus appears to be stuck at a fixed setting and does not auto-adjust. Only the wide-angle rear camera or the front selfie cam are supported by postmarketOS at the moment, probably due to a missing driver for the normal one. They both seem to have similar picture quality, so I won't test them separately (but all the pictures shown are taken with the wide-angle). By default, pictures are very dark and green-tinted, especially indoors. However, if I cover the sensor with my hand (or point the camera at a bright light) for a bit, then when I uncover it the colors will briefly be a bit brighter and less green and I can take my picture, which ends up a lot better (but still dark):

It doesn't deal well with shooting when a light is in shot, as it seems to get overexposed.

Trying to take a video used to freeze the phone for a few seconds, then reboot it (kernel panic?), however as of last week it no longer does this, though recording is extremely laggy. I don't know, but I think the GPU drivers might not (yet?) support hardware video encoding or decoding (except the wiki says it does, assuming it is “Venus”), so the result is not very usable yet. Here's a recording of where I'm writing this blogpost (I promise my lights are on):

The camera app allows viewing recent photos & videos, but no way to zoom into them or rotate media after-the-fact. There is also no standalone gallery app I could find, so viewing media is unfortunately quite awkward. Maybe once I finally set up Immich, the website can stand in for a gallery app.

Either way, once I find a way around the FluffyChat image endianness bug, I will feel quite happy sending some of these pictures to family & friends.

Audio

Oh, yeah, I haven't talked about this one yet. The pmOS wiki page lists Audio as “Broken” for my device, and indeed this was the case when I first installed pmOS. However, I saw that a lot of work was being done in this area and felt that I could trust these amazing folks to get it working. My PineBuds (bluetooth earbuds) paired fine and allowed me to listen to a couple YouTube videos in the meantime. Lo and behold, a few weeks into daily driving this phone I got to start enjoying the speakers on my Fairphone 5 via pma!7700, which I installed on my device thanks to Mr. Test. As I'm writing this, the MR is now merged and should be built soon!

![NOTE] It's called mrtest, a tool for testing Merge Requests, but I keep reading it like Mister Test and so I will make you read it that way at least once >:)

The speakers don't sound quite right, and sometimes go wonky until I do a suspend-resume cycle, but it's already extremely impressive work by everyone involved. It was very exciting to follow the discussion and see the first few demos from the devs, featuring classics like Rick Astley singing his one and only hit single through the speakers. I'm told microphone support is coming soon, which will allow me to start doing VoIP calls like Jitsi or MatrixRTC with friends!

Calls

Note how I specified VoIP… yeah, calls are their own thing. Even with speaker & mic working, more work will need to be done for call audio (which is a separate issue because of weird modem reasons). I can confirm that making phone calls works, as in, I can make someone's phone buzz, and they can make my phone (not buzz because GNOME doesn't implement haptics but) show a call notification—if nothing is fullscreen of course—that I can use to pick up.

I believe there is some extra complexity in Germany with VoLTE support being required, but I'll find out for sure once the call audio stuff is in place. Let's just hope I don't need to take any important calls anytime soon!

Web browsing

I only tried Firefox, as it was installed by default on my phone (with the mobile-config-firefox configuration by default).

Interface

The UI in portrait mode is reminiscent of Firefox for Android with the URL bar at the bottom, except tabs are always displayed. I would like for the tabs to auto-hide or, even better, browse them in a grid like the Android version provides, but this is perfectly usable. A right-click action can be simulated by long-tapping, which will also select the word you long-pressed. If you tap on the word again, the right-click menu closes and you can drag selection handles to select more/less text, then long-tap again to act on it.

In landscape mode, the UI moves to the top of the window and permanently takes up about one-third of the screen, given both the URL bar and the tabs are always visible and neither can be collapsed. This makes the landscape mode functionally useless, as there is not enough space to interact with page content. The only time I use it is when I want to fullscreen a video, which thankfully can be easily done by double-tapping on the media. The “popout player” is also activatable, though unfortunately GNOME Mobile does not allow floating windows to overlay other apps, so it's not useful like it is on desktop.

The HTML <select> tag (used for dropdown selections) works exactly like on desktop, with very small touch targets, and is not scrollable, making only a few entries near the top of the list selectable.

When interacting with the popup menus that appear when one of the many permanently-visible buttons at the bottom bar, I found that there is no intuitive way to close them. Tapping outside of the menu does nothing, and clicking the button that opened it simply flickers it off-then-back-on. Thankfully, I found that by tapping the URL bar, it pops up the OSK, which I can then dismiss to get back to the page. This is quite awkward to do, but lets me use most of the browser features.

Extensions

This being the full version of Firefox, all extensions are available to install, and I was really happy to get my favorites uBlock Origin, Dark Reader, LibRedirect+Indie Wiki Buddy, Stylus, and Constent-O-Matic synced from my Firefox Account. I set Dark Reader to force every page to use an OLED-black background, set up my AI-blocking stuff on Stylus, and configured LibRedirect to point to my favorite frontends for websites I do not wish to send traffic to. I did have to make sure to disable settings sync in the Dark Reader preferences, as otherwise the OLED preference got automatically copied to all my desktops!

Unfortunately extensions suffer from the same “popup menu” behavior described in the previous section, and have the extra issue of only part of the menu being rendered (however, the entire menu is interactive, so if you know your way around you can still blindly navigate):

Thankfully, the Bitwarden extension has a “pop out” mode that puts it in its own window (and that window does get fully rendered!). The button to trigger this is always in the same spot, so I can reliably blindly tap it. However, the popout window replaces Firefox in GNOME Mobile, so if I have Bitwarden open I cannot see Firefox. It also triggers the app overview a couple seconds after opening, which is probably a GNOME Mobile bug, which often interrupts me typing my master password and causes me to accidentally launch whatever app appeared where the OSK key I was aiming for was. There is a native Bitwarden client called BitRitter, but the last commit was over a year ago so I fear it may suffer a similar fate to Goldwarden. There is also to my knowledge no system-wide “autofill” API for Linux, that would allow a password manager to fill login details into non-web apps.

Surfing the web

Websites themselves render and feel great as despite this technically being “Firefox Desktop”, meaning they are correctly detecting by other means that this is a phone. I did get locked out of Google.com (something about my browser being unsupported), but it served as a good slap on the wrist, reminding me to instead use an alternative frontend to Google Search, such as Startpage. I checked again while writing and it seems they now “support” my browser. I read news articles, browsed blogs, and used Piped to access YouTube videos without too many issues. It seems that Firefox does not unload tabs very readily as a few times my phone ran out of RAM, and the entire browser got killed by the OOM daemon, so I've been careful to keep my tab count low.

Terminal

Of course, we can't talk about a Linux distro without mentioning the terminal. My GNOME Mobile install came with a terminal emulator it calls “Console”, but I was able to determine by inspecting the running processes it is actually kgx (I see this project explicitly bans LLM contributions, and I applaud that!). The interface is well-adapted to my display, and has a nice feature that shows you a preview of all your open sessions/tabs in a grid. When using the terminal, the OSK gains some extra buttons for Tab, Ctrl, Alt, and the arrow keys.

The default shell is Alpine's own default ash, with no colored prompt, tab completion, or some features I am used to like this s{imple,yntax} from Bash, the latter of which I especially miss when having to type commands via a touchscreen and every saved keystroke counts. Although fish is available on the Alpine repos, kgx offers no way I could find to launch fish instead of ash. I don't want to set my system default shell to fish as it's not POSIX-compliant, and would much prefer to only have interactive sessions (kgx and possibly ssh) launch with it.

Although the interface adapts well to mobile, it does not have any mobile-specific features which would be very welcome, mainly:

1. Ability to select & copy text. It's very much mouse controls here, you can double-tap to select a word and triple-tap to select a line, but no way to grow/shrink the selection. When I am asked to share log output, I have to triple-tap each line, use the OSK to hit Ctrl+Shift+C (which is only possible thanks to the extra keys that appear), and one-by-one paste them into FluffyChat.

2. Following the system theme, or allowing a custom theme to be set. An off-grey is used as the background color instead of my configured pure-black background, which is important on OLED displays. The only setting I could find is a toggle in the hamburger menu for switching between light and dark mode. There is an open issue for custom color themes, with a linked merge request which sadly has had no response from the maintainer since four years ago when a change was requested, which has long since been implemented. I really hope it moves forward…

3. Pinch-to-resize. This one's more of a nitpick, but I run into it quite often so I'm putting it here. Termux did this on Android, and it meant that any TUI app that wanted more space could be very quickly and easily dialed in. On kgx, I have to use the hamburger menu to access a + and – button, and it only allows going down to half of the default size, which is not always enough to display e.g btop.

Other than that though, it serves its purpose in a crutch (mostly restarting iio-sensor-proxy every once in a while) and with a couple small-ish changes I'd be very happy to use it.

Some other apps worth mentioning

GNOME Software is an app store that comes with the install and helped me discover a lot of awesome mobile-friendly apps from Flathub. I also believe there is a bunch of software on aports+pmaports, but sadly it is not at all discoverable via GUI, you have to use the terminal and already know the package name. This is the case of polycule for example, a very functional Matrix client built using the same SDK as FluffyChat, which I tried out but could not get the UI working very well on GNOME Mobile. GNOME Software does seem to have some support for Alpine packages as it notifies me every day about “System Updates” (despite having explicitly configured it to NOT check updates automatically) which when tapped list some Alpine packages. However, when I accept these updates, it doesn't always finish, and even if it does, running sudo apk update && sudo apk upgrade in a terminal gets even more updates. So: terminal requirement #3 is for updating the system. Flatpak updates work fine, however. I did notice that after installing updates it will show a message saying “Last checked: X days ago”, so I don't know where it gets these from.

KDE Connect is installable and, after manually enabling the firewall rules with the terminal (required use #4), was able to connect to my desktops. This lets me use a keyboard & mouse without plugging them in, control media, and do a few other things, though I did not get the file sending feature to work. The theme looks really bad because my Qt themes are broken as discussed earlier.

My favorite desktop calculator, called Qalculate! (yes with the exclamation mark!), is available but not mobile-friendly, however it can be made usable by navigating to File→Minimal window, and all the crazy unit conversions and math features are there. It does hide the history though, so I guess a mobile-native client/mode would still make sense.

There's a YouTube client with quite nice UI called Pipeline (fka. Tubefeeder), but I only got it to play a video once. It uses something called “Clapper enhancements” to play YouTube videos, and this doesn't seem to pull an up-to-date version of yt-dlp, as I get error messages about formats missing. I also tried the Flatpak, but that one complains about missing video decoding codecs on my phone, so it does not bode well. Something nice about Tubefeeder is it lets me select a Piped instance, but I did not yet find a way to have it sync my subscriptions like LibreTube does on Android. It also suffers from a quite similar freezing bug to Tuba, so I assume they both trigger the same Vulkan “lost device” codepath.

I used an RSS reader called Pulp for a bit until I touched a setting that makes it crash on startup. I'm now using Newsflash, which nicely syncs from my Nextcloud News, but only displays the article content in a narrow centered column that does not follow my libadwaita theme, so I don't use it very often. I opened an issue to track this.

Android apps

As much as I'd like to use Linux exclusively, there are some cases where being able to fall back to Android is very useful. This is where Waydroid comes in, running an entire LineageOS image with optional Google Play Services inside a container. I installed it from GNOME Software, selected my desired Android image in a dialog it presented me, and (after debugging a lot in the terminal due to silent crashing, but I don't remember what it was so I don't have a cool story to share sadly) I now have a working Android system I can boot into!

Besides the Waydroid app itself which contains the Android interface in a window, there's the really cool feature of running Android apps as their own windows. This means that, once Waydroid has booted, the apps really are quite seamlessly integrated into my shell! They have their own launcher icons, native windows, and can be closed by swiping them away. I can even use Android's back gesture (only for Waydroid apps), if I enable it inside Android's own settings app! There is noticeable input lag when interacting with these apps, which is a shame, but not a dealbreaker for me as they are only fallbacks.

I was very excited to run some of the apps I really miss from Android, and can happily report Öffi from F-Droid works great for public transport planning. I also tried to install the excellent OpenStreetMap client CoMaps (community fork of Organic Maps), which ran great, but I discovered that GPS is not bridged to Waydroid, so it's not actually able to give me directions to places. There's an issue tracking this in the Waydroid repository and some workarounds shared via a debugging “mock GPS” feature, but I didn't manage to get any working, and I would much rather have this as a hardware bridge that the Android system sees as physical hardware, much like how network is bridged through a fake wired connection.

I also tried to get my bank app running, as it is one of the only two things still tethering me to my old Android phone. The bank app is required to sign into the bank website from a new device, and to do sensitive operations like sending money to an account, so I unfortunately cannot “just use the website.” The app allows signing into it in two ways:

1. Taking a photo of my ID and doing a “live selfie.” This is where I discovered Waydroid does not bridge the camera, so that was a dead end.

2. Proving physical proximity to my old phone, and accepting prompts on the logged in app there. This is where I discovered Waydroid does not bridge Bluetooth devices.

So, I won't be getting rid of my old phone just yet. I have to charge it every few weeks when a bank thing pops up, and I guess Signal will bother me about my “Primary Device” eventually as well (the Android app is on my old phone, though Flare seems to have experimental support for being the primary device, I do not yet fully trust it to not lose my data and likely won't use it until it's considered stable). Thankfully these are rather rare occasions, and I can somewhat safely only carry around my postmarketOS phone!

There's an alternative project to Waydroid called Android Translation Layer which takes the WINE approach of “natively” running programs made for other OSes, foregoing the container approach entirely. In theory, this should let apps integrate even better, and potentially even pass the hardware to the apps that Waydroid so sorely lacks. There's a super impressive NewPipe port on Flathub using this. Unfortunately, I was unsuccessful in using the binary to run any of the apps that their own compatibility list says are supported, which I guess is likely a packaging issue on pmOS's side. I'm keeping my eye on this project though!

Conclusion

Despite me complaining so much (sorry!), I am extremely impressed with the state of Linux on mobile, and every doubt I had that I would regret moving to it is mostly erased. Most of the problems I list are minor papercuts and should be relatively easy to solve. They should make for easy targets when any of them annoys me enough that, instead of writing about it, I actually set up a dev environment and try fixing it. The community has been incredible, responding to all sorts of questions and often helping me live-debug issues with whatever crazy thing I'm trying to get working. I especially want to re-shout-out the folks at CCCAC without whom I don't think I would have taken the plunge and actually spent nearly 500€ on a device exclusively to run their software.

This blog has been dormant for a while, but with my recent adventures I am sure I will have plenty to write about, so perhaps expect some more writings once I land my first contribution to the OS!

While Android is “free,” we all pay for its development by submitting control (and data) to Google, further strengthening its control over half of the mobile OS duopoly. Since I stopped paying Google, I have now set up a recurring monthly donation to the postmarketOS team, and will look into supporting individual projects that I use every day on my phone to ensure development can continue and the amazing volunteers keeping this dream alive are remunerated for their efforts. A huge THANK YOU to everyone involved in #LinuxMobile for making the computer in my pocket possible!

No LLM was used to write this. As always, feel free to direct any corrections or feedback to my fediverse account @mat@allpurposem.at.

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

To give you this experience, there's online fantasy consoles such as PICO-8 (nonfree) and TIC80 which make it super accessible to prototype and finish small experiences. There's also hardware like the Playdate (nonfree) that further plays with input methods and form factors to really constrain your playground. Finally, there's the thriving homebrew communities around consoles such as the SNES and the N64 (check out this awesome demake of Portal!).

I've personally always had a soft spot for the Wii. Partially because I grew up with its incredible games such as Super Mario Galaxy 2 but also because Wii game modding gave me a peek at what would later be my career: game development. Although I've dabbled with Wii development in the past, I never felt I really understood what I was doing. A couple months ago, I set out to fix this. Armed with my finished DirectX assignment for a university Graphics Programming course, and the open door of “you can add extra features to raise your grades, but those are not mandatory,” I thought of this: what if I show up to the exams with my Wii, and do the presentation on it?

DirectX on the Wii (jk)

As excited as I was to enact this idea, I knew that I wasn't just going to compile my DirectX shaders and code for the Wii's CPU and call it a day. DirectX is, uh, not very portable or compatible with the Wii. The Wii is equipped with a GPU codenamed “Hollywood,” which has a whopping 24MB of video RAM as well as featuring no hardware support for any sort of shader. It really makes you appreciate some of the amazing scenes crafted on this console.

> Click here to explore Slimy Spring Galaxy on noclip.website

So, we must speak Hollywood's own API (called GX) to coax it into rendering a mesh with textures and transparency (as required by the assignment).

NOTE: In the final project, I've created a GX folder to hold all GX-specific code, and isolated the DirectX stuff into a separate folder called SDL. This way, I can control which platform-specific code is used via a simple CMake option. If you'd like to follow along, you can find everything here.

libogc

To access this API from C++, there's a library maintained by the folks at @devkitPro@mastodon.gamedev.place called libogc. This library, combined with the PowerPC toolchain, allows one to build programs targeting the Wii (and the GameCube, since they're so similar!).

NOTE: whenever I refer to the Wii from now on, it (mostly) also applies to the GameCube.

Although devkitPro themselves do not have a CMake toolchain file available, I was able to find an MIT licensed one courtesy of the rehover homebrew game. Passing this toolchain file to CMake automatically sets it up to build for the Wii. Cool stuff!

NOTE: The rest of this post is accurate to the best of my understanding, but it is likely I got some things wrong! If you need accurate info, I suggest you take a look at libogc's gx.h with all the functions and comments as well as the official devkitPro GX examples rather than following my own code. Comments, questions, and corrections are as always welcome at my fedi handle @mat@mastodon.gamedev.place !

Video setup

I won't dwell on the init too long, as most of it is just taken from a libogc example it's not too thrilling. What is cool though is how, to do v-sync, we create two “framebuffers” which are merely integer arrays... on the CPU? This is where one of the big differences in the Wii's hardware design compared to a modern computer comes in: both the CPU and GPU have access to 24MB of shared RAM. Meanwhile on a modern PC, the GPU will exclusively have its own dedicated RAM which the CPU cannot touch directly.

This shared RAM is where we store these framebuffers arrays, named by the Wii hacking scene “eXternal Frame Buffers” or XFBs (source. Because access to this so-called “main”)). Because having the GPU work on the XFB directly would be slow, the GPU has its own bit of actually private RAM which stores what's officially called the “Embedded Frame Buffer” (EFB). GX draw commands work on the super-fast EFB, and when our frame is ready we can copy the EFB into the XFB, for the Video Interface to read and finally display to the screen. This buffer copy is loosely equivalent to “presenting” the frame as is done in the APIs we're used to.

                 ┌─────────┐                           
                 │         │                           
       ┌─────────┤   CPU   ├───────────────────┐           
       │         │         │                   │           
       │         └─────────┘                   ▼           
       │                                  GX drawcalls      
       │                                       │           
       ▼                          ┌────────────▼──────────┐
Create XFB arrays                 │                       │
       │                          │ GPU private RAM (EFB) │
       │                          │                       │
       │                          └────────────┬──────────┘
       │                                       ▼           
       │                           Copy EFB to current XFB 
       │                                       │           
       │      ┌───────────────────────┐        │           
       │      │                       │        │           
       └──────► Shared MEM1 RAM (24MB) ◄───────┘           
              │                       │                    
              └─────────┬─────────────┘                    
                        ▼                              
                  Display frame                        
               ┌────────▼────────┐                     
               │                 │                     
               │ Video Interface │                     
               │                 │                     
               └─────────────────┘                     

The following code specifically handles finishing the frame and displaying it:

void GraphicsContext::Swap()
{
    GX_DrawDone();

    GX_SetZMode(GX_TRUE, GX_LEQUAL, GX_TRUE);
    GX_SetColorUpdate(GX_TRUE);
    GX_CopyDisp(g_Xfb[g_WhichFB], GX_TRUE);

    VIDEO_SetNextFramebuffer(g_Xfb[g_WhichFB]);
    VIDEO_Flush();
    VIDEO_WaitVSync();
    
    g_WhichFB ^= 1; // flip framebuffer
}

Every time a frame is done, we tell the GPU we're done via GX_DrawDone() and then do our EFB –> XFB copy via GX_CopyDisp(g_Xfb[g_WhichFB], GX_TRUE) (where g_Xfb is our two XFBs and g_WhichFB is a single bit we flip every frame). Then we notify the Video Interface of the framebuffer it should display with a call to VIDEO_SetNextFramebuffer(g_Xfb[g_WhichFB]). Finally, VIDEO_Flush() and Video_WaitVSync() ensure we don't start rendering the next frame before this one is displayed.

Drawing a mesh

Now that we know how framebuffers work on the Wii, let's get to drawing our mesh!

Vertex attribute setup

Before we can start pushing triangles via GX, we must tell it what kind of data to expect. This is done in two steps:

1. First, we tell GX that we will be giving it our vertex data directly every frame, rather than having it fetch it from an array via indices.
   GX_SetVtxDesc(GX_VA_POS, GX_DIRECT);
   
2. We then access the vertex format table at index 0 (GX_VTXFMT0), and set its position attribute (GX_VA_POS) as follows:
   • The data consists of three values for XYZ coords (GX_POS_XYZ)
   • Each value is a 32-bit floating point number (GX_F32)
   • I'm not sure what the last argument is for, but zero worked fine for me.

GX_SetVtxAttrFmt(GX_VTXFMT0, GX_VA_POS, GX_POS_XYZ, GX_F32, 0);

Both of those functions are then repeated for normals and texture data, if needed.

NOTE: The Wii's GPU supports indexed drawing, where vertex data is stored in an array and drawn using indices into that array. This allows fewer vertices to be defined while reusing them.
I didn't know about this until I finished this project, so we'll be sticking with non-indexed drawing. The concept is quite similar, but you'd set the vertex desc to GX_INDEX8 and bind an array before calling GX_Begin. You'd then pass indices rather than vertex data inside the begin/end block.

Drawcalls

Each frame, we must queue up some commands in the GPU's first-in-first-out buffer. We can tell GX it's time to draw some primitives via the GX_Begin function, passing along the type of primitives (triangles!), the index in the vertex format table we filled in earlier, and the number of vertices we'll be drawing. Afterward, we can give it the data in order by calling the respective function for each attribute we configured. Finally, we cap it off with a GX_End (which libogc just defines as an empty function, so I guess it may just be syntax/API sugar).

GX_Begin(GX_TRIANGLES, GX_VTXFMT0, m_Vertices.size());
for(uint32_t index : m_Indices)
{
    // NOTE: really wish I used GX's indexing support...
    const Vertex& vert = m_Vertices[index]; 

    GX_Position3f32(vert.pos.x, vert.pos.y, vert.pos.z);
    GX_Normal3f32(vert.normal.x, vert.normal.y, vert.normal.z);
    GX_TexCoord2f32(vert.uv.x, vert.uv.y);
}
GX_End();

Transformations

NOTE: This section will assume you're familiar with matrix transformations. If you don't know what this is, here's a link to the first of two pages in the OpenGL tutorial discussing this, which is the explanation that finally made it click for me.

The first important matrix is the model matrix. This matrix's job is to convert model-space vertices into world-space. This is useful when we want to rotate, scale, or translate an object in the world.

To look around in our scene, we need to set up a view matrix, which takes care of translating world-space into view-space. Finally, we'll need a projection matrix that turns the given view-space into clip-space, at which point the GPU takes over and handles stuff like culling and converting to non-homogeneous coordinates.

In normal graphics, we tend to pair view and projection together, and leave model on its own for transforming normals and other data to worldspace in the shader. The Wii however takes a different approach: load the combined modelView matrix, and separately handle the projection.

The reason for this is quite interesting: GX instead expects you to give it light information in view space rather than the usual worldspace. We'll cover simple lighting in a later section.

So, we must only set these two matrices to handle all of our transformation needs:

    GX_LoadProjectionMtx(projectionMat, GX_PERSPECTIVE);

    // use the same matrix for positions and normals
    GX_LoadPosMtxImm(modelViewMat, GX_PNMTX0);
    GX_LoadNrmMtxImm(modelViewMat, GX_PNMTX0);

Textures

Textures are actually really easy! We can directly bind a byte array as a texture, since the CPU and GPU have that 24MB of shared RAM.

I initially tried to use the Wii's native format (TPL), which has some really cool features such as the CMPR compressed texture encoding, which has the GPU decompress the texture live when it needs the data, at (seemingly) no performance cost. Awesome!

Sadly, I couldn't get it working...

Even using basic TPL, there were some gnarly artifacts:

I finally caved and decided to just use PNG and decode it to a raw RGBA8 byte array, bypassing TPL entirely. This got rid of the artifacts, so I guess we'll never know why they happened!

GX_InitTexObj(&m_Texture, decodedData, width, height, GX_TF_RGBA8, GX_CLAMP, GX_CLAMP, GX_FALSE);

To use the texture, we can simply bind the texture object that we got during init to the index we want to sample from:

GX_LoadTexObj(const_cast<GXTexObj*>(&m_Texture), GX_TEXMAP0);

By default, GX reads from GX_TEXMAP0 when it draws triangles, so this is actually all we needed to do!

Transparent textures

We can set up blending with the alpha channel like so:

GX_SetBlendMode(GX_BM_BLEND, GX_BL_SRCALPHA, GX_BL_INVSRCALPHA, GX_LO_OR);

This tells GX that, when blending two transparent samples, it should take the previous pixel's alpha value (GX_BL_SRCALPHA) and the inverse of the new one's alpha (GX_BL_INVSRCALPHA). I'm not sure what the GX_LO_OR is for, but blending sure does seem to work so I'm keeping it. There's a good explanation of this exact blend function over at LearnOpenGL.

Although on a first glance transparency seems to work, there's a pretty big issue that appears if you look at the fire effect from close up (I don't have a screenshot from the Wii build, so this one's from DirectX, however the same effect is visible)!

One of the triangles that makes up the effect is getting drawn before another triangle that should render behind it... the first one writes to the Z-buffer, causing the second triangle to get discarded. This is usually good, because it skips drawing pixels that are fully occluded, and makes sure stuff that's behind a model doesn't end up getting drawn over it. In the case of translucent images however, we get artifacts like the one above.

This image was rendered with the Z buffer entirely disabled, which shows why we need it:

The solution is thankfully quite simple:

if(m_UseZBuffer)
{
    GX_SetZMode(GX_TRUE, GX_LEQUAL, GX_TRUE);
}
else
{
    GX_SetZMode(GX_TRUE, GX_LEQUAL, GX_FALSE);
}

Set m_UseZBuffer to false for models using transparent textures, and that last GX_FALSE in the GX_SetZMode disables writing to the Z buffer. Note that we still want reading (the first GX_TRUE), as otherwise the fire effect would end up rendering over our vehicle mesh!

”““Shaders”“”

Unlike modern APIs, the Wii's GPU is not programmable with arbitrary shaders. Instead, we can play with something quite powerful called texture evaluation (TEV) stages. We've got a whopping 16 TEV stages to play with, which Nintendo graciously calls a “flexible fixed-pipeline.” Each stage is essentially a configurable linear interpolation (lerp) between two values A and B by a factor of C. Finally, a fourth value D is added to the result.

u8 TEV_stage(u8 a, u8 b, u8 c, u8 d)
{
    return d + (a * (1.0 - c) + b * c);
}

NOTE: There's also optional negation, scale, bias, and clamping. I'm skipping over them here because I didn't end up using them. There's more complete documentation available here.

The source of A, B, C, and D can all be configured per stage. You could, for example, have it lerp between your texture's color and the light color based on the amount of specular lighting it receives. I tried to set this up with lots of help from Jasper (thanks again!) but ultimately it didn't work. I'd like to try again sometime in the future!

Diffuse lighting

The Wii's GPU features built-in per-vertex lighting. This means that you can (optionally) tell it to calculate how much light each vertex receives from up to eight light sources, which can be either distance-attenuated (like a lamp) or angle-attenuated (like a spotlight).

GX provides a type GXLightObj that we can load and then set up with all our parameters. For the renderer I was making, I needed to set up a “sun” light, which is a very far away point light with (practically) no attenuation.

NOTE: normally in graphics programming, this is be done with a simple directional light. However the way I got it to work on the Wii was by simulating this attenuation-free point light model, so I went with that.

This is the bit of code that initializes it every frame:

GX_SetChanAmbColor(GX_COLOR0, ambientColor);
GX_SetChanMatColor(GX_COLOR0, materialColor);

GX_SetChanCtrl(
        GX_COLOR0, GX_ENABLE,
        GX_SRC_REG, GX_SRC_REG,
        GX_LIGHT0, GX_DF_CLAMP, GX_AF_NONE);

guVector lightPos = { -lightDirWorld.x * 100.f, -lightDirWorld.y * 100.f, -lightDirWorld.z * 100.f };
guVecMultiply(viewMatNoTrans, &lightPos, &lightPos);

GXLightObj lightObj;
GX_InitLightPos(&lightObj, lightPos.x, lightPos.y, lightPos.z);
GX_InitLightColor(&lightObj, lightColor);

GX_LoadLightObj(&lightObj, GX_LIGHT0);

Let's go over each step.

Color registers

First, we tell GX what ambient and material colors we'll use. The ambient color is used for lighting all vertices, no matter of received light. This makes sure the back of our mesh is not just pure black. The material color will tint your whole model (it's like a global vertex color), so I keep it as white.

Channel setup

GX_SetChanCtrl configures the lighting channel we'll use. We want the light to affect GX_COLOR0, which is where our texture will be. We tell it to get the ambient and material color from the registers we set just before (GX_SRC_REG). We set GX_LIGHT0 as a light that affects this channel, with the default diffuse function GX_DF_CLAMP. Finally, we disable attenuation by passing GX_AF_NONE, meaning our light can be infinitely far away but still light our model as if it were right next to it.

Position transformation

We then calculate the light position, which is very far away opposite to the direction it'll shine. Note that we multiply it with the view matrix (with the translation part stripped out) as light stuff is in view space!

Light object creation

Finally we create our GXLightObj, giving it its position and color, and load it into the GX_LIGHT0 channel. Make sure to disable lighting on the fire (it makes its own light, wouldn't make sense to be in shadow) and wham! There's our sun!

You can find all my lighting and TEV code in Effect.cpp. The filename is unfortunate, but as this was initially a DirectX project, I was stuck with that name from the header.

We're done!

I quickly built a GameCube version the night before the due date, and submitted the required .exe alongside my sneaky .dol binaries with no further elaboration. I wanted to keep the surprise. I showed up the next day to campus with a very full backpack, and when it was time pulled out the Wii to present my “extra features to raise your grades, but those are not mandatory.” It seemed to make quite a splash! Looks like I'm not the only one who grew up with the Wii :)

You can download a build here. Wiimote and GameCube controls are supported!

Tags for fedi: #homebrew #wii #gamecube #gcn #devkitpro #directx #linux #graphics #gamedev #foss #retrocomputing

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

#include "pythonstart.h"

def greet(name):
    print("hello, " + name + "!")
    return

def greet2(name):
    print("how are you, " + name + "?")
    return

def bye():
    print("ok bye!")
    return

#include "pythonmid.h"

username = "Mat"

print("Hello from \"Python\"!")

greet(username)
greet2(username)
print("getting ready to say bye...")
bye()

#include "pythonend.h"

The first code review comes in, and it seems your contribution may be in jeopardy:

That's just Python code! It won't work with the rest of our C++ codebase!

Before they can reject your code, you sharply interject:

Hey! Did you even test my code?

With skepticism in the air, one of your brave teammates steps up and runs the code through a C++ compiler. To everyone's amazement, the result is identical to that of running it with Python! The code not only speaks Python, but fluently converses in C++:

$ g++ Python.cpp && ./a.out
Hello from "Python"!
hello, Mat!
how are you, Mat?
getting ready to say bye...
ok bye!

The commit is eventually merged, and your unconventional approach not only saves your job but earns you a place in the annals of the team's most memorable code submissions. You are also banned from touching that codebase again.

How it works: unraveling the enigma

This kind of program is termed as “polyglot,” which literally means “written in multiple languages.” The entire idea of writing one of these relies on finding intersections between the two (or more!) languages' syntax. In Python, a # signifies a comment, while in C++ (and C), it denotes a preprocessor directive. These lines are the key to the program. We'll see how the preprocessor works (and how we can abuse this!) in a little bit.

You'll notice the only preprocessor directives in our code are #include statements. Unlike other languages, C and C++ opt for a simple but effective solution to calling external library functions: copy-paste. Seriously.

When I write #include <iostream> at the top of a C++ file, what actually happens is the entire file called “iostream” (installed system-wide as part of the C++ standard library) gets pasted by the preprocessor, residing now where that #include statement once was. You don't technically have to use the #include directive to get a C++ program calling library functions: you can get the same behavior by just copying the file's contents manually at the top of your code (but that's a terrible idea!).

For example, here are two C++ header files:

preamble.h:

int main()
{
    int retVal = 0;

postlude.h:

}

And our beautifully readable code:

#include "preamble.h"

for(int i = 0; i < 4; i++)
{
    retVal += 1;
}
return retVal;

#include "postlude.h"

Let's run our code through the standalone C/C++ Preprocessor cpp:

$ cpp code.cpp
# 1 "code.cpp"
# 1 "preamble.h" 1
int main()
{
    int retVal = 0;
# 2 "code.cpp" 2

for(int i = 0; i < 4; i++)
{
    retVal += 1;
}
return retVal;

# 1 "postlude.h" 1
}
# 10 "code.cpp" 2

You can see the preprocessor outputs a bunch of lines starting with #. These are a kind of comment meant for us puny humans to understand exactly what the preprocessor did. The first number indicates the line number, and the string in quotes is the filename. The optional number at the end of the line represents a flag, where 1 means it's the start of an include, and 2 means we are returning to a file after an include is done. You can find the full docs here. You can see we start at line 1 in code.cpp, which then includes preamble.h. The contents of preamble.h follow, and afterwards we return back to code.cpp. So on and so forth, finally copy-pasting together an amalgamate program that consists of a simple main function that returns 4.

The preprocessor is a very powerful tool, and as long as the final text that is passed to the compiler is valid, anything goes!

Let's break down the polyglot program from the start of the post:

Functions

In Python, functions are defined as follows:

def greet(name):
    print("hello, " + name + "!")

Somehow, we need to translate this into working C++ just through the preprocessor. Because Python allows declaring functions anywhere, but C++ does not, we can use a function pointer instead. C++ has a neat trick here called a lambda, which allows us to define unnamed functions inline, and it's perfect to have our pointer point to.

Armed with this knowledge, we can use #define to create a macro that will turn def into auto (a special C++ keyword that deduces the type of a variable based on what's assigned to it), and another macro that turns greet(name) into our lambda definition:

#define def auto 
#define greet(arg) greet = [](std::string arg) {

Applying this to our Python function from above gets us some of the way there

auto greet = [](std::string arg) {:
    print("hello, " + name + "!")

We still have to handle that pesky : that Python requires at the end of function declarations. Now, where does C++ have a :... aha! The revered ternary operator, that everybody totally loves! Its syntax is as follows: condition ? truthy : falsy. We don't care about the logic here, we just want that sweet : character, so we can add the most cursed ternary expression I've ever written to the end of the greet macro:

#define greet(arg) greet = [](std::string arg) { false?false

Running the preprocessor through our function, we get the following:

auto greet = [](std::string name) { false?false:
    print("hello, " + name + "!")

That's some good progress! There's three main issues left: – the ternary operator is left hanging there. We need a “falsy” value for this thrilling and definitely-very-useful comparison to compile. – there's no print function in C++ (this project was conceived before std::print was added to C++23). – we need to close that dangling curly bracket and add a semicolon at the end of our lambda, somehow.

Implementing the print function can be done with a simple function-style macro that just plops the argument into std::cout. This only works for simple prints, but I'm not going for anything more here :)

Additionally, we can knock the unfinished ternary issue out by adding a stray false; at the beginning. Usually this will just do nothing as it just gets discarded, but in the case that a print occurs right after a function definition, it will complete the ternary operator. Hooray!

#define print(a) false;std::cout << (a) << std::endl; 

Now for closing the function... there are no keywords left we can use here. I haven't found a way to make this work consistently without polluting the print macro with closing brackets that would cause it to break if used more than once or outside of a function. Thankfully, Python has a return keyword we can add without changing the behavior of the function:

def greet(name):
    print("hello, " + name + "!")
    return

Then on the C++ side, we can redefine it to close our lambda!

#define return return; };

Finally, our simple function now preprocesses to this valid albeit cursed C++ code:

auto greet = [](std::string name) { false?false:
    false;std::cout << ("hello, " + name + "!") << std::endl;
    return; };

“int main”

Here's the next bit we have to tackle, after the function definitions:

username = "Mat"

print("Hello from \"Python\"!")

greet(username)
greet2(username)
print("getting ready to say bye...")
bye()

This code was given in one of my university courses to showcase the basics of Python. In it, we create a variable, call a couple functions, and call it a day.

Python allows writing code willy-nilly outside of any function, but in C++ this is not exactly the case, especially if we need to call library functions. Our print statements must reside inside the main function. We can have our initial header (the one with all the function macros) also start the main function by adding a lone int main() { at the end of it. We also need a header at the end with the sole purpose of closing that opening bracket:

pythonstart.h:

#define greet(arg) greet = [](std::string arg) { false?false
#define print(a) false;std::cout << (a) << std::endl; 
#define return return; };

// start the main function (will be closed by pythonend.h)
int main() {

pythonend.h (thrilling):

}

The code

Looking at the first lines of the actual code, a lot of stuff is missing for it to work in C++:

username = "Mat"
print("Hello from \"Python\"!")

The first obvious issue is that C++ requires types, while Python does not. We will need a pythonmid.h header to plop a std::string in there and so tell username its type:

pythonmid.h:

#define username std::string username

Then, oh no! Our print macro-function inserts a stray false right after my string literal, causing a compile error! We must redefine print to remove the false prefix, but keep the lone semicolon as it can serve to punctuate the username declaration:

#undef print
#define print(a) ;std::cout << (a) << std::endl;

Finally, the function calls:

greet(username)
greet2(username)
print("getting ready to say bye...")
bye()

In short, every function must be redefined to expand into a call rather than a declaration, like so:

#undef greet
#define greet(name) greet(name);

That's it!

And there we go! Here's the full “Python” file from the start of this post, put through the C++ preprocessor:

// -snip- the entire contents of the <iostream> and <string> C++ headers
# 2 "pythonstart.h" 2
# 15 "pythonstart.h"

# 15 "pythonstart.h"
int main() {
# 4 "Python.cpp" 2


auto greet = [](std::string name) { false?false:
    false;std::cout << ("hello, " + name + "!") << std::endl;
    return; };

auto greet2 = [](std::string name) { false?false:
    false;std::cout << ("how are you, " + name + "?") << std::endl;
    return; };

auto bye = []() { false?false:
    false;std::cout << ("ok bye!") << std::endl;
    return; };

# 1 "pythonmid.h" 1
# 25 "pythonmid.h"
std::string
# 20 "Python.cpp" 2

username = "Mat"

;std::cout << ("Running \"Python\"!") << std::endl;

greet(username);
greet2(username);
;std::cout << ("getting ready to say bye...") << std::endl;
bye();

# 1 "pythonend.h" 1
}
# 31 "Python.cpp" 2

You can find the full sources on my Gitea.

I hope this was a fun introduction to polyglot programming! It's usually filled with crazy hacks like these, and thus can be very fun whilst being immensely impractical, but believe me: it has its uses!

While researching for a different project in 2020, I came across this perfect example: Cosmopolitan is a project that allows C programs to build to an “actually portable executable”: a file that runs simultaneously on Linux, MacOS, Windows, FreeBSD, OpenBSD, NetBSD, and can also directly boot from the BIOS. I recommend Justine's blog post for a fascinating read!

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

You: DirectX 11? At this time of year? At this time of day? In this part of the country? Localized entirely within your Linux system? Me: Yes. You: May I see it? Me: No. Yes!

Yes, it's true! Set up all nice and fast, with no WINE. Just a native executable, with full diagnostics for any IDE, compatibility with debuggers, nice and (usually) helpful output, and maintaining full Windows build support at the same time.

One of my favorite courses this year at DAE has to be Graphics Programming 1. In it, we go through the logic and techniques to create a software raytracer (laggy web build), then a rasterizer. I've really enjoyed the course so far, and it's been painless to follow on Linux... with one small looming issue: the last few weeks of the semester involve learning the DirectX 11 API, and porting the rasterizer to it such that you can seamlessly switch between hardware and software rendering. Really cool assignment, but this won't work on Linux!

Software rasterizer

DirectX is a proprietary closed standard made by Microsoft, and is thus likely never to come to Linux and other platforms.

Wait, but how come I can play DirectX Windows games on Linux?

Aha! I'm glad you asked, even if by proxy of me prewriting that question as the section title. As part of Valve's Proton compatibility layer for playing Windows games seamlessly on Linux, some really smart folks maintain a wonderful project called DXVK. DXVK works together with WINE to translate DirectX 9-11 API calls into Vulkan calls, enabling compatibility with any system that supports a recent enough Vulkan spec. When a game kindly asks DirectX to draw a triangle, DXVK will handle this function and make the equivalent Vulkan API calls for the GPU's Vulkan driver to handle. I'm just now learning about these APIs (I have exclusively used OpenGL in my previous projects), so I can't explain how this works much further. It's magic to me, and I don't cease to be impressed when I can just launch a Windows-only DirectX-based game and it “just works” on Linux, often even with better performance than if it were running on the native Windows DirectX drivers.

Anyways, this is all great, but that covers running Windows games through WINE and having DXVK translate DirectX calls to Vulkan. Having to build .exe files, and especially the tooling for handling them on Linux (WineDbg, profiling...) sucks. It works, and I wrote a blog post about doing exactly this, but I'd still much rather avoid having to do this. So what's the trick?

The trick

I asked in a few chat rooms whether there was another way, and by pure chance someone at the ASUS Linux community happened to mention I'd have to use “dxvk native.” Yes, DXVK was designed to work together with WINE. Nowhere in its wiki does it mention any other use cases (that I could find). However, checking its recent releases we can see a file named dxvk-native-X.X-steamrt-sniper.tar.gz. Native? As in, doesn't depend on WINE? As in I can link against it and produce a normal Linux ELF binary that can be debugged and poked at just like I'd do an OpenGL program?

I immediately got to work, and found shockingly little documentation online about using this. The best resource I found is the build script of the Deus Ex: Human Revolution decompilation, which enables Linux support through DXVK. I whipped up a small CMake script, and ended up with this snippet:

if(UNIX)
    include(ExternalProject)

    ExternalProject_Add(dxvk-native
        GIT_REPOSITORY https://github.com/doitsujin/dxvk.git
        GIT_TAG v2.3
        CONFIGURE_COMMAND meson setup <SOURCE_DIR>
        BUILD_COMMAND ninja src/d3d11/libdxvk_d3d11.so src/dxgi/libdxvk_dxgi.so
        INSTALL_COMMAND "")

    ExternalProject_Get_property(dxvk-native SOURCE_DIR BINARY_DIR)
    set(DXVK_SOURCE_DIR ${SOURCE_DIR})
    set(DXVK_BINARY_DIR ${BINARY_DIR}) 
    unset(SOURCE_DIR)
    unset(BINARY_DIR)
    add_dependencies(${PROJECT_NAME} dxvk-native)

    include_directories(SYSTEM
        ${DXVK_SOURCE_DIR}/include/native/directx
        ${DXVK_SOURCE_DIR}/include/native/windows)
    target_link_directories(${PROJECT_NAME} PRIVATE
        ${DXVK_BINARY_DIR}/src/d3d11
        ${DXVK_BINARY_DIR}/src/dxgi
    )

    target_link_libraries(
        ${PROJECT_NAME} PUBLIC dxvk_d3d11 dxvk_dxgi
    )
endif()

Drop this into your CMake project, and you should be able to use and interface with DirectX just like you would on Windows! Just make sure that, instead of a HWND, you give DirectX (well, DXVK) a pointer to your SDL_Window. I added this bit of code to handle that:

#ifdef WIN32
        // Get the handle (HWND) from the SDL window
        SDL_SysWMinfo sysWMInfo{};
        SDL_GetVersion(&sysWMInfo.version);
        SDL_GetWindowWMInfo(m_pWindow, &sysWMInfo);
        swapChainDesc.OutputWindow = sysWMInfo.info.win.window;
#else
        swapChainDesc.OutputWindow = m_pWindow;
#endif

After a pretty quick build, here it is! A pure DirectX window running on Linux, in all its glory!

Well, it's not drawing anything interesting, but that's because I still need to port my rasterizer! Seeing the default blue background is already amazing given how far from an officially supported DirectX platform I'm running this on.

Full source code for the project is available on my Gitea instance here.

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

Ah, C++ standard containers. So delightfully intuitive to work with. The most versatile has to be std::vector, whose job is to wrap a dynamic “C-style” array and manage its capacity for us as we grow and shrink the vector's size. We can simply call push_back on the vector to add as many elements as we want, and the vector will grow its capacity when needed to fit our new elements.

If you understand how a std::vector works, feel free to skip to the code.

But is it that simple?

Resizing the vector's internal array is not cheap! It incurs allocating a whole new (bigger) block of memory, copying all the elements to it, and finally freeing the old block (note that this copy may be a move, see here). Because we add elements one by one, this would trigger a lot of resizes, as the vector keeps having to guess how many elements we plan to add and reallocating a bigger and bigger internal array every time we push_back past its capacity! So, a conforming std::vector implementation will usually try to get ahead of us and secretly allocate a bigger block when it sees we start pushing to it, and then it can just keep track of the size of the vector (how many elements we've pushed to it) separately from its capacity (how many elements it can grow to before it needs to resize the internal array again).

std::vector kindly exposes this internal functionality to us through some functions. For example, the capacity() function returns the current capacity of the vector's internal array. If we know the size it will grow up to ahead of time, we can use the reserve(size_type capacity) function to have it pre-allocate this capacity for us. This avoids reallocating a lot when doing a bunch of push_backs, which can let us gain a precious bit of performance (see the example here for some actual numbers).

The code

Now that we understand std::vector::reserve, let's take a look at some C++:

std::vector<int> myVec{}; // create a vector of size 0
myVec.reserve(1); // reserve a capacity of 1
myVec[0] = 42; // write 42 to the first element of our empty(!!) vector
std::cout << myVec[0];

When run, the above prints 42. I hope I'm not the only one who's surprised this works! I'm overwriting the value of the first element in a vector... which has no elements. This is an out of bounds write, and should definitely not work. Not only that, but on my machine I can replace index 0 with up to index 15187 and it still works fine! Index 15188 segfaults, though, so at least that's sane behavior (so long as I get far enough away from the start of the vector...). So what the peck is going on??

The peck (it's going on)

Okay, okay, I'll say the thing. We've found what in C++ is called “undefined behavior” (UB). This is a magical realm where anything could happen. Your computer might replace every window title with your username, or your program might send an order to all pizza restaurants in a 5km radius. If you're lucky, your program will just crash. More likely though, your code will do exactly what you intended it to do, and either subtly break something later on, or never signal anything on your machine... and break on someone else's.

Why is this undefined behavior, you ask? We told our vector to reserve a size of 1, so 0 is a perfectly valid index in the its internal array. However, the C++ standard never states that vector should have an internal array! It only asks for vector implementations to be able to grow and shrink, and for reserve() to “ensure a capacity” up to which no reallocations need to happen.

NOTE: after lots of research (and asking the smart folks of the #include C++ community), I've been unable to find an implementation where this does break. That doesn't mean it's okay to rely on this behavior! It's still UB!

Why it works for us

Despite this being undefined behavior, it works consistently in my program. Why is this? When we run the line myVec[0] = 42, the std::vector::operator[] function is called with an argument of 0, to return a reference to the location in memory at index 0 for this vector. Let's look at the source code for this function in GCC's libstdc++ (which I used for my testing, though the same issue applies on clang and MSVC):

/**
 *  @brief  Subscript access to the data contained in the %vector.
 *  @param __n The index of the element for which data should be
 *  accessed.
 *  @return  Read/write reference to data.
 *
 *  This operator allows for easy, array-style, data access.
 *  Note that data access with this operator is unchecked and
 *  out_of_range lookups are not defined. (For checked lookups
 *  see at().)
 */
_GLIBCXX_NODISCARD _GLIBCXX20_CONSTEXPR
reference
operator[](size_type __n) _GLIBCXX_NOEXCEPT
{
    __glibcxx_requires_subscript(__n);
    return *(this->_M_impl._M_start + __n);
}

Looking past all the macros (the subscript thing expands to an empty line by default, we'll look into it later), this simply takes the pointer to the start of the internal array (_M_impl._M_start), adds our argument __n, and returns it as a reference. As long as _M_start points to some valid allocated address, we should be fine accessing it within bounds of the array (note, of course, that this is only true for this implementation of libstdc++! Other implementations may do different things; we're in UB-land here). This explains why our index outside of the vector's size worked: we're indexing the internal array, not the vector! As long as we call reserve on the vector first, and our index is within that reserved array's size the data should be perfectly okay being written to and read from an out-of-bounds-but-within-capacity index of a vector (on this specific version of GCC's libstdc++). If we remove the myVec.reserve(1) line, the program does crash as expected, since _M_impl._M_start is not initialized and thus points to invalid memory.

Array out of bounds

The reason why accessing an index higher than the array's size works is covered here, but a tl;dr is that you are indeed overwriting memory you shouldn't be, and by chance nothing bad is happening. If we run it through the valgrind memory error detector, it indeed detects our error for any index outside the array. Here's the log for a write at index 1, after a call to reserve(1):

Invalid write of size 4
   at 0x1091FC: main (ub.cpp:8)
 Address 0x4e21084 is 0 bytes after a block of size 4 alloc'd
   at 0x4841F11: operator new(unsigned long) (vg_replace_malloc.c:434)
   by 0x109825: std::__new_allocator<int>::allocate(unsigned long, void const*) (new_allocator.h:147)
   by 0x109604: allocate (alloc_traits.h:482)
   by 0x109604: std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) (stl_vector.h:378)
   by 0x1093FF: std::vector<int, std::allocator<int> >::reserve(unsigned long) (vector.tcc:79)
   by 0x1091EA: main (ub.cpp:6)

Let's dissect this output: 1. The first line indicates that we wrote 4 bytes somewhere that's “invalid.” That's the size of a 64-bit int, which is the type we're writing into index 1. 2. The big call stack tells us where the array that we're accessing out of bounds was allocated. The penultimate line points us to that std::vector::reserve call we made, which creates a “block of size 4” (the vector's internal array, with the capacity for a single 4-byte int).

This indicates that we are indeed accessing the internal array out of bounds, and that it is a memory error that will cause UB even on this implementation of std::vector. So that answers that!

Speed at the cost of safety

Although on my GCC install, using this as actual storage “works” “fine,” it has... issues. When we try to do a range-based loop, it will never get the elements we wrote out of bounds. If the vector gets copied, it will only bring over the data within its size, and leave behind everything else. These kinds of issues would be super hard to diagnose had I not spotted the UB here!

Shouldn't std::vector::operator[] warn us that we're accessing an element outside of the vector's size? Let's check the C++ standard on vector functions.

Only at() performs range checking. If the index is out of range, at() throws an out_of_range exception. All other functions do not check.

- The C++ Standard Library: A Tutorial and Reference by Nicolai M. Josuttis (2012), pages 274-275

Well, darn. I can understand why, though. When writing code in C++, we expect to have the lowest possible performance overhead, yet still get to use all these nice abstractions. Performing bounds checks, even if cheap, can really add up if we have to do it for every vector access. Changing it to at(0) does indeed print a (relatively) helpful crash message:

terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 1) >= this->size() (which is 0)

As I was writing this, an excellent relevant post by @saagar@saagarjha.com graced my Mastodon timeline:

Original source.

That's not all, though! Remember that curious __glibcxx_requires_subscript(__n); macro in the GCC implementation of operator[], which I said we'd look at later? Now is before's later, so let's take a look at the definition:

#ifndef _GLIBCXX_ASSERTIONS
  # define __glibcxx_requires_subscript(_N)
#else
  # define __glibcxx_requires_subscript(_N)	\
  __glibcxx_assert(_N < this->size())
#endif

So it does do something! You just have to have _GLIBCXX_ASSERTIONS defined. Indeed, if we define that macro with the -D_GLIBCXX_ASSERTIONS compiler flag, we get this wonderful totally-readable error when the code tries to index out of bounds:

/usr/include/c++/13.2.1/bits/stl_vector.h:1125: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = int; _Alloc = std::allocator<int>; reference = int&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.

Okay, it's no “you're accessing this vector out of bounds, please stop,” but it certainly is better than dealing with the potential mess of undefined behavior that awaits otherwise. I guess I'll be adding this flag to all my debug builds from now on!

If you're curious, this is my original code where I found the issue.

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

1. Programming 1 “SDL Framework”: https://git.allpurposem.at/mat/SDL-Framework
2. Programming 2 “GameDevEngine2”: https://git.allpurposem.at/mat/GameDevEngine2
3. Graphics Programming “RayTracer”: https://git.allpurposem.at/mat/GraphicsProg1
4. Gameplay Programming “_FRAMEWORK”: https://git.allpurposem.at/mat/GameplayProg

The versatility of having a cross-platform project allowed me to add tons of niceties for some of these. The one I'm most happy with is the “GameDevEngine2” framework from Programming 2, to which I added web support and ended up using it for my and 2FoamBoards's entry in the 2023 GMTK game jam.

Programming 3

I'd been having it easy. A couple nonstandard Microsoft Visual C++ (MSVC) bits of syntax here, a couple win32 API calls (functions that are specific to Windows) there... I wasn't expecting what arrived in my downloads folder today. I applied my usual CMake boilerplate, with SDL support, hit run to see the perhaps 50-100 errors... and instead was greeted with a simple but effective singular error.

apm@apg ~/S/Prog3 (main)> clang++ source/GameWinMain.cpp 
In file included from source/GameWinMain.cpp:9:
source/GameWinMain.h:12:10: fatal error: 'windows.h' file not found
#include <windows.h>
         ^~~~~~~~~~~
1 error generated.

Oh, no

There's no SDL. There's no OpenGL. No GLFW, Qt, or GTK. It's all bare Windows API calls. I think I was in some form of state of disbelief, as I spent the next 30 minutes slowly creating #defines and typedefs to patch in all the types. Maybe, just maybe, I could patch around the types and it would magically open a window and I could get started with my classwork. No such thing happened.

Options

So: what are my options? Is this salvageable, without having to boot the dreaded virtual machine? Let's see... I could:

• continue patching around the 3-4k lines of win32 API calls like I was ineffectively doing before
• rewrite the engine from scratch to support SDL
• build the native .sln file by somehow running MSVC on WINE (a Windows compatibility layer for Linux)
• cross-compile from Linux to Windows and run the .exe file with WINE

Obviously the first two options would be preferable, as they don't come with a hard dependency on the unfamiliar world of WINE. However, they sadly also take the most time. I have not yet discarded the second option (the author of the engine gave me the green light to rewrite it for native Linux, and even use it in exams (that's a first!!)), but as I have to follow the class from the start, I think I'll be going with WINE.

aur/msvc-wine-git

Of course, I'm not the first person to want to build a .sln project from Linux. This appears to be a solved problem, with the polished-looking msvc-wine toolchain available as a native package for my distro. So I went ahead and installed it:

apm@apg ~/S/Prog3 (main)> gimme msvc-wine-git
[sudo] password for apm: 
:: Resolving dependencies...
:: Calculating conflicts...
:: Calculating inner conflicts...

Aur (1) msvc-wine-git-17.7.r4-2

:: Proceed to review? [Y/n]: 

It diligently fetched MSVC, the Windows 11 SDK, and all the necessary components from Microsoft's servers, while I had time to read the documentation. I happened upon the CMake instructions, which is how I've managed all my school-related projects so far, and it didn't stick in my brain. I don't intend to criticize the writing, but something about it being all the way in the bottom in a FAQ, with no code blocks or example commands, or having a class going on around me while I was doing this prevented me from understanding how I'm supposed to use it. The only time I've ever used a separate toolchain was Emscripten; it provides a nice little emcmake wrapper for CMake which takes care of a lot of the details for you. I gave it a few tries, but seeing I was getting nowhere, and every second was lost class time, I decided to move on to my last option.

LLVM

I knew a little about LLVM before this, from having used clangd as my language server for C++ projects. As I understand it, it's a group of compilers designed in such a way that the “frontends” (which read the text code and output an intermediate language) and “backends” (read intermediate language and output the final binary) are swappable and interchangeable. This means you can use the same backend to compile both C++ and Rust code, while still getting equally well-optimized machine code out the other side. I enlisted the help of @JohnyTheCarrot@toot.community, who I knew has worked with clang before. He told me about the concept of an “LLVM triple”, which is a setting for LLVM compilers that tells it what sort of machine you want it to output code for. Crucially, you can specify a triplet for a completely different system than your own, and it should still work. I tried the following command:

clang++ -target x86_64-w64-mingw32 source/*.cpp -o game

This currently outputs 227 linker errors. I know there were many syntax-related compiler errors which I've since fixed, but it does get us past the dreaded #include windows.h! All of the linker errors take the following form:

/usr/bin/x86_64-w64-mingw32-ld: /tmp/GameEngine-ac27d8.o:GameEngine.cpp:(.text+0xc95f): undefined reference to `__imp_DeleteObject

Fun with the linker

Each of these is related to a call of a Windows-related function. It looks like we're missing the libraries! Adding the -mwindows flag tells Clang it's compiling & linking a GUI Windows app, instead of a command line one. This causes linking against a lot of win32 GUI-related functions, reducing the linker errors to a mere 9. There's two kinds:

• __imp_AlphaBlend and __imp_TransparentBlt According to the code, these are used for transparency. I have yet to use this engine, but from the names I'm guessing they allow for drawing semi-opaque images on top of each other and blend the colors together. According to Microsoft's documentation, these are located in Msimg32.dll.

• __imp_mciSendStringA These are functions from the defunct Multimedia Control Interface (that's the mci at the start of the name!), which this engine uses to play audio. Microsoft helpfully kept the legacy documentation online, informing me that these belong to Winmm.dll.

At first, I assumed I'd have to get these from a copy of Windows. However, I remembered WINE has a lot of open source reimplementations of these DLLs (Windows's version of .so shared libraries), and sure enough locate msimg32.dll (note the lowercase: I wasted some time with this because Linux is case sensitive, while Windows is not!) pointed me straight to a DLL I could yoink. I added it to the list of files to compile, and the msimg32-related linker errors were gone. Hooray!

...or so I thought. I excitedly copied in winmm.dll and tried to compile...

clang-16: error: unable to execute command: Segmentation fault (core dumped)
clang-16: error: linker command failed due to signal (use -v to see invocation)

Excuse me?? The linker is segfaulting?? To be honest, I have no idea whether this is an actual bug in LLVM's linker, but it sure did stump me for a while. I thought maybe my copy of winmm.dll was corrupt, or WINE did something weird with it. I went as far as downloading Microsoft's version of the DLL, but was met with the same sad message. What could I be possibly doing wrong?

Oh. I'm not supposed to be copying the DLLs into here, am I? The last time I used a linker without going through CMake, I was passing libraries to it was -l<libname>. But it can't be that easy for this... can it? It'd have to go to my default WINE prefix to fetch them, which sounds plain weird. Libraries come from system paths, not user-specific folders. Well, might be worth a try anyways...

apm@apg ~/S/P/build (main)> clang++ -mwindows -target x86_64-w64-mingw32 ../source/*.cpp -o game -lmsimg32 -lwinmm
In file included from ../source/GameWinMain.cpp:10:
../source/GameEngine.h:19:9: warning: '_WIN32_WINNT' macro redefined [-Wmacro-redefined]
#define _WIN32_WINNT 0x0A00                             // Windows 10
        ^
/usr/x86_64-w64-mingw32/include/_mingw.h:239:9: note: previous definition is here
#define _WIN32_WINNT 0xa00
        ^
1 warning generated.
Warning: corrupt .drectve at end of def file
Warning: corrupt .drectve at end of def file
Warning: corrupt .drectve at end of def file
apm@apg ~/S/P/build (main)> ls
game.exe*

wait. That built?? HUH???? There's no way it—

apm@apg ~/S/P/build (main)> ./game.exe
-snip-
0130:err:module:import_dll Library libgcc_s_seh-1.dll (which is needed by L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe") not found
0130:err:module:import_dll Library libstdc++-6.dll (which is needed by L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe") not found
0130:err:module:LdrInitializeThunk Importing dlls for L"Z:\\home\\apm\\School\\Prog3\\build\\game.exe" failed, status c0000135

Right. Not so fast, heh. Still, this is great news! I don't know how or why this works, but we're linking to the DLLs somehow somewhere. WINE can't find some mingw32 libraries which were pulled in by -mwindows, but we can easily point it to them with export WINEPATH="/usr/x86_64-w64-mingw32/bin"

And that's it! Here's the engine in all its glory, with audio support and all! It's beautiful...

Right, there's nothing built on it yet. It's just a blank canvas. But hey, it doesn't crash!

What's next?

Having this run through WINE does come with a few limitations:

• All WINE apps take a long while to launch, though you can vastly improve this by running wineserver --persistent beforehand.
• Usually, I attach gdb (the GNU debugger) to my code from my IDE, neovim. However, with this program running under WINE, I don't know how I would do that. Debugging remains an unsolved mystery (EDIT: see Addendum, I figured it out!).
• WINE is slowly merging Wayland support, but at the moment it runs under X11, meaning I'm sacrificing some performance and convenience.
• Finally, of course, this will never have Linux support. I don't like that.

Long-term, depending on the course workload and how complex the engine functions end up being, I think I will rewrite it in SDL. This will have the added bonus of enabling, like with my other engine ports, web support (see my Programming 2 end project here and a game jam game made in the same engine here). However, I think this will take longer than I think is reasonable to spend while procrastinating on other classes, so I'm leaving it here. I wrote down my process while it was still fresh in my mind, so I hope this was an interesting read! As always, any and all constructive feedback is welcome directed to me: @mat@mastodon.gamedev.place .

I am considering writing up my general porting process in a separate blog post, so perhaps expect that next!

Addendum

After doing some additional research, and asking around in the very helpful WineHQ IRC room, I found a way to get debugging working! The first step is adding the -g flag to the clang++ invocation, which tells clang we want it to generate debug information (namely source maps, so the debugger can show which line of code we're at). Then I simply have to run winedbg --gdb game.exe, and I am presented with a (nearly) full-featured gdb prompt!

I'm unsure how to hook this up to neovim (maybe I can look into the Debug Adapter Protocol for this?), but for now just having a gdb environment is awesome enough. Unto more adventures!

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at

The game launches, and you can move around the 64x64x64 world with WASD. Space is used to jump. Look around with the mouse. You can left click to break a block, and right click to place dirt.

You can scan it with the following command on Linux:

zbarcam -1 --raw -Sbinary > /tmp/m4k && chmod +x /tmp/m4k  && /tmp/m4k

• -1: exit after scanning the code
• --raw: don't process it as text
• --Sbinary: use the binary configuration

The project is available on GitHub at TheSunCat/Minecraft4k

How???

Short answer: Pain, suffering, evil dark magic, compression, a couple years, some more pain, and a bit of luck.

Long answer: Well, it's a long story. If you're ready to learn about various creative game programming techniques and cursed incantations (don't worry, I will explain each concept as it becomes relevant), strap on!

NOTE: This is my first ever blog post. I'm trying it out, as I really enjoy writing! I tried my best to keep it accessible and entertaining, but I am very much open to constructive feedback on how it and future posts can be made better. Every time I describe a major change or evolution, there will be a link to the file or commit in question.

The goal

So, how big is a QR code? Truth is, they come in many sizes. The one you're probably most familiar with is “version 1”, with 21x21 pixels. This can store 25 characters of text, but only 17 8-bit bytes. The difference is because text can be more efficiently encoded than bytes can, as there are less possible values for a QR-compatible text character than the 255 values a byte can have.

The biggest existing QR code, which you can see at the top of the document, is “version 40” and fits a whopping 2953 bytes. Let's see how many max-size QR codes it would take to fit the following Villager idle sound from Minecraft:

That's, uh, 8605 bytes. It fills up (nearly) three QR codes. We have to fit playable Minecraft into a third of the Villager “hmm” sound.

Exposition

On December 2009, Markus Persson, the creator of Minecraft, released a very cut-down version of Minecraft for the Java 4k Game Programming Contest, in which contestants develop games in Java which fit under 4 kilobytes (4096 bytes). An archived version of the game is available on Archive.org. The game renders a pixelated image vaguely resembling Minecraft. All this in... 2581 bytes. Woah. That's much less than 4k. AND it already fits in a QR code. Remember, our size limit is 2953 bytes.

So the deed is done, right? Not quite! This version depends on the Java Applets framework, which was phased out by browsers starting in 2013, and is now completely unusable. Also, I think we can do better. The game suffers from bugs, poor performance, and low resolution. Not only that, but running it required a full web browser, Java installation, and the Java Applets plugin enabled. Can a standalone version of Minecraft really fit in 2953 bytes?

The Java era

To improve upon software, we need to change the source code. Unfortunately, the code for Minecraft4k was never made available, and likely never will be. All original posts about it now lead to a 404 error, and the original Minecraft4k page where you could play it now redirects to Microsoft's Minecraft homepage. Thankfully, Java code is not too hard to retrieve from JAR files! Unlike most compiled languages like C and C++, which compile to optimized assembly language, Java programs compile to an intermediary called Java bytecode (which is then interpreted into normal assembly by the Java Virtual Machine when you run the JAR file!). This bytecode still bears a strong resemblance to the original source code, and preserves a lot of information, such as the names of variables and functions. This means that using tools made by very smart people, we can “de-compile” the bytecode stored inside Minecraft4k and get back usable source code!

Let's pop the JAR file into the wonderful jd-gui decompiler, and...

float f13 = 0.0F;
float f14 = 0.0F;
f14 += (this.M[119] - this.M[115]) * 0.02F;
f13 += (this.M[100] - this.M[97]) * 0.02F;
f4 *= 0.5F;
f5 *= 0.99F;
f6 *= 0.5F;
f4 += f9 * f14 + f10 * f13;
f6 += f10 * f14 - f9 * f13;
f5 += 0.003F;
int m;
label208: for (m = 0; m < 3; m++) {
    float f16 = f1 + f4 * ((m + 0) % 3 / 2);
    float f17 = f2 + f5 * ((m + 1) % 3 / 2);
    float f19 = f3 + f6 * ((m + 2) % 3 / 2);
    for (int i12 = 0; i12 < 12; i12++) {
        int i13 = (int)(f16 + (i12 >> 0 & 0x1) * 0.6F - 0.3F) - 64;
        int i14 = (int)(f17 + ((i12 >> 2) - 1) * 0.8F + 0.65F) - 64;
        int i15 = (int)(f19 + (i12 >> 1 & 0x1) * 0.6F - 0.3F) - 64;
        if (i13 < 0 || i14 < 0 || i15 < 0 || i13 >= 64 || i14 >= 64 || i15 >= 64 || arrayOfInt2[i13 + i14 * 64 + i15 * 4096] > 0) {
            if (m != 1)
                break label208; 
            if (this.M[32] > 0 && f5 > 0.0F) {
                this.M[32] = 0;
                f5 = -0.1F;
                break label208;
            } 
            f5 = 0.0F;
            break label208;
        } 
    } 
    f1 = f16;
    f2 = f17;
    f3 = f19;
}

Uh-oh. It's true that Java usually keeps variable and function names when compiling. But Persson likely turned this feature off, as it would take up place in the JAR. We have the code, but we don't know what any of it does. It looks like we'll have to figure out what every single statement does by staring at it and poking it repeatedly: let's do some reverse engineering!

Reverse engineering

The first step is to get it running. The code still uses the Java Applet framework, so I ported everything to use the newer (but still ancient) Java Swing framework. This allows the game to open a window and display (render) pixels inside it. Great! Let's start reversing the code (on May 30, 2020). I will only go over the major parts, as it took a long while.

The most obvious things are as follows: – the 214*128 BufferedImage is clearly the screen that's drawn on the window. There's a big loop that updates every byte inside it, every frame. The dimensions also match the resolution of the pixelated game view. – the 64*64*64 array of integers is the world. It gets generated at game start, and a single value is modified when you place/destroy a block. – the game uses fixed-update physics. This means that, instead of multiplying the player's movement by the length of each frame, it assumes a constant length for each physics step, and does enough steps to catch up with the time elapsed that frame. The benefit of this is that physics calculations are much simpler, since they don't have to adapt to different step sizes.

I documented the player's position, velocity, look direction, keyboard and mouse input. My pal @JuPaHe64 (Twitter) documented how the game checks whether your movement is valid and corrects it to not let you clip inside blocks. Thanks to this, we fixed a bug which gets the player stuck in the original game if they jump into a wall.

Over the next week, JuPaHe64 and I documented the rest of the game. There's two very unusual systems operating together to keep the size of the game down.

The texture atlas

A single 16x16px texture of the side of a block in the original game takes up ~350 bytes. Minecraft4k has 6 distinct blocks, with three unique sides. That's 6 * 3 * 350 = 6300 bytes. Even compressed by the JAR format (it's just a renamed ZIP file), this would take up a huge amount of our allotted space. So how does Persson do it?

In stead of storing a bitmap of the textures, Minecraft4k opts to generate them at runtime from algorithms. Woah.

I'd never seen this before, but it's a great way to save space. Here's the texture atlas generated with the default Java Random seed:

One unexpected boon of this is that it becomes really easy to up the resolution of the textures. As they're algorithmically generated, the same patterns will hold in higher detail, which led to this cursed image:

JuPaHe64 created a really nice texture pack for it, showing that this is actually a viable method to generate some kinds of textures for games:

I am especially fond of the tree bark and stone textures.

Ray”tracing”

No, really. Well, sort of. Minecraft is a very complex game to render, despite its relatively simple graphics. The real game has to do tons of calculations to avoid rendering block faces that are hidden, turn it all into triangles, and do math to distort them from 3D space into the shapes we see on our 2D screen. This is an oversimplification of the rendering technique called rasterizing. This complexity would be too much for our size limitations, so instead Minecraft4k employs a specific variation of raytracing: voxel raymarching.

NOTE: Voxel is just a word for a 3D pixel, which the Minecraft world is made out of.

This is what happens for each pixel that needs to be drawn:

1. Calculate the direction of the ray, based on where the player is looking and the pixel's coordinates
2. Store the initial position of the ray
3. Loop until we hit a block:
   1. Step (“march”) forward by one block
   2. Check if the ray has hit a solid block
4. Color the pixel with that block's texture

As you can probably guess, this is a pretty simple algorithm to implement, and therefore saves a lot of precious bytes. Additionally, we can use the result of raymarching the pixel at the center of the screen to tell what block the player is looking at. This saves writing a separate function to get the block, and saves a considerable amount of space.

Raytracing is known nowadays for enabling more complex effects, such as lighting and reflections. With minimal modifications to the code, JuPaHe64 was able to add pixel-perfect shadows, and I added ambient light illumination. Paired with a simple world generator, it can make for some interesting shots, despite the poor performance:

However, a problem arises: even without the fancy world generation and raytraced effects, the Java game now takes up 17757 bytes. That's over 6 QR codes. The code required to make this work on contemporary systems with Java is simply too big. It's time for a change of approach.

June 2020: Porting to C++

Let's kill two birds with one stone, and rewrite the game to: 1. Use C++, which is much more familiar to me, so I can make faster progress 2. Use the GPU to run the game in real time, as raytracing on the CPU can be very slow

After a few grueling days, the new port was finally functional, and ran beautifully thanks to GPU acceleration. It used an OpenGL compute shader, which is a type of program that can run on the GPU, once per pixel, all at once. This means that, unlike on the CPU, where each pixel has to finish rendering before the next one can be rendered, on the GPU all of it happens at once.

One of my favorite changes happened when my friend @HANSHOTFIRST joined in, and we wrote the commit title “bad the shader”. Here, “bad” is used as a verb, since we pretty much rewrote the entire thing and it, uh, didn't work. The idea was to reduce the space it took up, and improve performance, by processing all three XYZ axes together. The original game does ray steps per-axis, meaning that first it does the X axis, then the Y, and finally the Z step. This seemed unnecessary to us at the time, but we clearly missed something, as you can see here:

After a small hiatus, and porting the game to Linux on March 2021, I played a bit more with the graphical effects possible with raytracing:

I then introduced the first big size improvement: executable packing. gzexe is a tool which uses gzip compression (the same that was used to deliver this page to you!) to reduce the size of an executable. I also implemented usage of a Shader Minifier, whose job it is to automatically reduce the size of the shader code by removing comments, shortening variable names, and getting rid of unneeded newlines and spaces. The reason why this is so important to do with the shader is that, unlike C++ code which is compiled into the binary, OpenGL shaders must be stored as source code, and compiled by the GPU drivers at runtime. Therefore, any single character we can save in the shader code should translate directly to a byte saved toward our goal. So, how small did this get it? Well, an impressively small, QR-code fitting... 11314 bytes. Well, peck. That's four QR codes. We need to divide the size by four. How is that even possible?

C that? It's my sanity evaporating!

Yeah. I, uh, rewrote it in C. After a long break. In June of 2022, the game is born anew, this time more broken than ever. Once I got all the basic features in by August, I was left with a very functional game, which does everything I think defines Minecraft, in 4598 bytes. Woah. That's 1645 bytes over our limit, for a total size of just over 1.5 QR codes. Suddenly this looks feasible. By this time, the build process has picked up some dirty tricks. We're far from done, but there are already a few things that make me mildly uncomfortable. Let's go through the most egregious ones:

-nostartfiles

The C compiler we'll be using is called GCC. There's a few obvious flags we can pass to it to reduce the size of the output executable (the binary). We can remove the debug information with -s, and optimize the code for size rather than performance with -Os. However, there's one different argument. We all know the ubiquitous int main function, yes? The entry point to every C and C++ program? The first code that runs? I'm here to reveal to the world that we've been lied to: the real entry point is void _start, but Big Compiler doesn't want you to know. This is part of their great ploy to sell more argc and argv. In all seriousness, C programs actually start at, well, _start. This contains code to set up the stack, global variables, the values of argc and argv, and various parts of the C runtime. Because we don't need most of this, we can elect to just skip main and use _start in stead:

void _start() {
    // set up the stack
    asm volatile("sub $8, %rsp\n");

    // Minecraft goes here

    // exit
    asm volatile(".intel_syntax noprefix");
    asm volatile("push 231"); //exit_group
    asm volatile("pop rax");
    asm volatile("xor edi, edi");
    asm volatile("syscall");
    asm volatile(".att_syntax prefix");
    __builtin_unreachable();

vondehi

Like gzexe, vondehi is a tool that allows shrinking a binary by decompressing it and then running it. It's a bit of very well-optimized assembly that can be prepended to any xzcat-compatible compressed executable, and will make it runnable on Linux. At this point, all semblance of cross-platform support is completely gone.

strip

Another GNU tool, strip allows you to remove specific sections from a Linux executable (in the Executable Linker Format—ELF). It turns out that, even with -s passed to GCC, a lot of nonessential information is still kept in the form of ELF sections. We can use strip -R to get rid of them. I simply tried each one by one until the game would no longer run.

With all this, we're at 4598 bytes. Which is pretty great, but we have a whole journey ahead to get it below 2953. There's a few obvious shader code optimizations that can be made, such as shortening the names of uniforms (variables in the shader which can't be changed by the Shader Minifier), and putting everything in the shader's main function rather than using function calls. This makes up for a total of 42 bytes. Yikes. Where are we going to get the 1k+ savings we need? That's a great question, and it's going to take a year of hiatus to answer. I left Minecraft4k at 3786 bytes on September 2022 and focused on my first year at university.

The final run (sanity--)

It's September 2, 2023. University starts in less than two weeks. I have recently rediscovered MattKC's excellent snake game in a QR code. I'm reminded of Minecraft4k, and how close I was to the finish line. It's time for one last sprint.

Embracing the dark side

One big chunk of bytes lies in my calls to C standard library functions. sin, cos, fmodf, and friends. I had tried to get around this by implementing some of them myself, where it was smaller than just calling the libc function.

// TODO tune this, or use inline x86 ASM
#define TRIG_PRECISION 20
static float my_sin(float x)
{
    float sine;

    float t = x;
    float sine = x;
    for (int a=1; a < TRIG_PRECISION; ++a)
    {
        float mult = -x*x/((2*a+1)*(2*a));
        t *= mult;
        sine += t;
    }
    return sine;
}

I'll let you guess how the TODO comment was applied.

float my_sin(float x) {
    float sine;
    asm (
        "fsin;"
        "fstps %0;"
        : "=m" (sine)
        : "t" (x)
    );
    return sine;
}

Directly using the x86 instruction fsin, I was able to save 80 whole bytes from the binary.

API Abuse

OpenGL defines a standard language to talk to the GPU in, so you can get it to do your bidding. This is great, because it will work on any machine, no matter the platform or hardware... supposedly. I've already encountered weird crashes running Minecraft4k's C++ edition on Intel GPUs, because their OpenGL implementation didn't like my way of storing the world data. Every OpenGL driver has its quirks and bugs, which make programming in OpenGL so much more fun. It adds the surprise factor that is very welcome in an otherwise consistent field. Your raytracer might work on one computer, but you'll never know whether it works on all computers, because of the fun factor that is OpenGL driver bugs.

Conversely, we can take advantage of some of these bugs to get away with removing a lot of otherwise strictly necessary code. That'll be 26 bytes, coming right up!

dlsym

Rather than asking Linux to make the OpenGL functions available to us, we can manually load and fetch them using the pair of functions dlopen and dlsym. This requires storing the plain text name of every function we need in the binary, which does take up a lot of bytes, but it ends up being just slightly shorter by 21 bytes. Whew.

This is not okay

Remember that we're compressing the binary and attaching a small decompressor to it, so any improvement to the “compressibility” of the binary directly translates to saved bytes for us. So, I opened the binary in a hex editor, and started zeroing out parts of it. Surprisingly, I found that a lot of seemingly important parts, defined in the ELF specification, are simply not necessary. The game still runs after being so heavily mutilated, although I cannot say the same about Linux ELF utilities. Check out this very disappointed readelf output, where almost everything is a zero:

apm@apg ~/D/m4k ((37570fc8)) > readelf -a Minecraft4k_prepacked.elf
ELF Header:
  Magic:   7f 45 4c 46 00 00 00 00 00 00 00 00 00 00 00 00 
  Class:                             none
  Data:                              none
  Version:                           0
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x0
  Entry point address:               0x102ca
  Start of program headers:          0 (bytes into file)
  Start of section headers:          64 (bytes into file)
  Flags:                             0x0
  Size of this header:               0 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0
readelf: Warning: possibly corrupt ELF file header - it has a non-zero section header offset, but no section headers

There are no section groups in this file.

There are no program headers in this file.

There is no dynamic section in this file.

I also truncate the last 50 bytes of the uncompressed file, and the last 8 bytes of the compressed archive. This gives us a fun new error when running the program: /usr/bin/xzcat: (stdin): Unexpected end of input, though the game still plays fine! That's 28 more bytes for me!

Transcending sanity

After rewriting the math in the shader code more than once, drawing many pages of diagrams to figure out how the equations can be simplified, and making a few compromises (goodbye, crosshair and window resizing), Minecraft4k took up a measly 3006 bytes. Where are the remaining 53 expendable bytes?

Excluding the dlsym strings, the biggest data Minecraft4k stores is a couple floating point constants. A float takes up four uncompressed bytes, but Inigo Quilez points out that we often don't need the last two bytes, and can therefore make floats compressible down to just 2 bytes. That's a 2x reduction!

Thanks to @b0rk@jvns.ca sharing the very useful float.exposed website, which has a great interface to poke at the floating point binary format, I was able to check that clearing the last two bytes of the constants did not affect their values too much. Sure, I lost some minor precision, but will the player notice if gravity is 0.00299072265625 instead of 0.003? I don't think we can fit any analytics library to tell us, so it must be good enough. Combining this trick with some tuned compression flags, we have Minecraft4k in 2981 bytes. Just 28 more...

At this point, I was out of ideas. I asked everyone I knew for advice, and we tried a few great ideas. Porting large functions to assembly and hand-optimizing. Getting rid of the stdlib by implementing dlopen and dlsym myself. Linking against the minimal musl libc implementation instead of the GNU C library. None of these ended up working out. So, I asked a very important question:

Notice anything that looks wrong? Hopefully not.

I removed the shadow from under the grass tufts. I'm sorry. It was the only way. Thankfully nobody I asked noticed it, so it's fiiiiiine.

And with that, we're done! Minecraft4k now fits snugly into 2952 bytes, a single byte under the maximum.

Thanks for reading! Feel free to contact me if you have any suggestions or comments. Find me on Mastodon and Matrix.

You can follow the blog through: – ActivityPub by inputting @mat@blog.allpurposem.at – RSS/Atom: Copy this link into your reader: https://blog.allpurposem.at

My website: https://allpurposem.at