Pixel Perfect Rendering

In last week's playable demo, the edge of the world would jitter back and forth about the width of a pixel when you moved your character - which looked terrible and was very distracting.

This was caused by the rendering (the camera) not being "pixel perfect", and - I'm not going to lie - fixing it was a hell of a rabbit hole! You might think it's a trivial problem to solve, but each game has slightly different constraints, so let's take a quick tour through the many different aspects of it!

The Problem

First, let's take a good look at the problem by zooming in to 8x scale; you can see that when the player moves, the edge of the "pixel physics world" (which is devoid of any actual physics-pixels, so it's just a grey rectangle) wobbles around:

The white pixel (fixed to the bottom left of the pixel physics world) makes the wobbling super obvious

This only shows up when the camera follows the player; when I move the camera with the arrow keys in "freelook" mode, there's no wobbling:

The white pixel is offset from the rectangle, but doesn't wobble

To understand the problem, and why it only shows when the camera follows the player, we need to understand the rendering process for the "physics pixels":

1. Generate a big purple texture (image) the size of screen resolution divided by the current scale (1, 2, 4, 8, etc).
2. Draw the pixels from the pixel physics world onto that texture (including a white pixel at the corner of each world, for this visualization).
3. Render the texture onto the screen at screen position 0,0, at a size scale times bigger (which makes the pixels - and the dark grey background of the physics world - drawn that much bigger).
   • (Aside: drawing at screen position is what causes the purple background outside the physics world: only empty pixels in the physics world are colored in dark grey, so everything outside the bounds stays purple so I can see it easily.)
4. Draw the rest of the world, such as the player and platforms (currently using rectangles) at exactly whatever position they're at.

The camera is calibrated such that 1 unit = 1 screen pixel, so when the camera pans in freelook mode and I wrote code like "move the camera up by 5 units", it happened to move the camera by integer amounts, which meant that the pixel world would always fall perfectly on a pixel boundary.

But when the camera follows the player using the fancy algorithm I wrote about last week, the camera ends up at positions like 5.24982,80.23948; that gets rounded to a screen pixel, but each frame the result is a little different, which causes the pixel world to wobble back and forth.

This is compounded by the fact that the rest of the world is positioned at (let's pretend) arbitrarily precise coordinates, and drawing them using Macroquad's drawing routines, which will happily draw them at those coordinates rounded to screen pixel position. Since the ground platform is drawn right next to it at a much "courser" position, it makes the wobbling seem even worse. And if you actually draw some physics pixels inside the pixel physics world, it gets even worse!

Attempt 1: Snap camera to pixels

Firstly I tried rounding the camera position to pixel-aligned coordinates; this worked great in that it stopped the pixel physics world from wobbling back and worth relative to the rectangle, but didn't work so great in that it made everything else jitter:

You can feel the camera being snapped to the (metaphorical) grooves of a pixel-sized grid

Attempt 2: Snap world things to pixels too

The next attempt was to keep the camera snapped, but force the rest of the world to be aligned to texel coordinates - the "fake pixels" as per their size in the physics world (the pixels as drawn onto the physics world texture).

It might be a little hard to see in the capture, but now the _player_ is moving jerkily - yuck.

Unfortunately, it also doesn't actually fix the problem, as you can see our white canary pixel shaking away.

Attempt 3: Sub-texel smooth scrolling

I did a bunch of reading to see how other people solved this problem; in my mind I almost had this solved with attempt 1, and I just needed to have the camera not make everything feel shaken about - so I focused on that.

Eventually I stumbled on this sweet little demo and the accompanying writeup (and code). To summarize, it uses a clever technique that takes advantage of having a higher resolution screen than your low resolution source material.

1. Say you have a one dimensional screen, and it renders images that are exactly 100 pixels wide.
2. Now pretend you have created a 20 pixel wide (also 1D) image which you want to render onto that screen.
3. The naive approach is to scale it up 5x and draw it at position 0, and when your camera moves left twice once then your camera moves 5 pixels at a time so you would then render the image at screen position 10 (since 2x5=10).
4. But you could instead say "hey actually rather than moving the camera 1 image pixel over (5 screen pixels at 5x scale), let's move the camera 1 screen pixel over instead", in which case two moves left has you draw your image at position 2, and your scrolling feels a lot smoother.

(Aside: They also propose a further refinement where you could allow scrolling by e.g. 0.1 pixel, and then when drawing at position e.g. `0.2`` you actually draw at position 0, but post process each pixel with a linear filter: determine the color of each pixel from 20% of the color of the left pixel plus 80% of the color of the right pixel. We'll come back to this idea later!)

This way you could construct that image however you want (without having to worry about perfect pixel alignment, because anything you draw would inherently be aligned to those pixels), and then only pay attention to pixel alignment in that last step #4. (Ominous voice: or so it seemed.)

So that looked like exactly what I wanted, and the only problem was that it was implemented as a shader (little graphics card program) and I hadn't worked with shaders at all in Macroquad. So I hacked things up in a separate program and eventually got it working nicely, so that when the camera scrolls then things are smooth.

When I finally got it working, I was super happy with it!

The happiness was great and lasted for about 37 seconds, which was about as long as it took me to make that bottom chicken move horizontally according to a Sine curve (like you can see above).. because the problem with this approach is that objects moving slowly will jitter. Specifically anything that travels less than 1 texel (aka image-pixel) per frame (aka screen-update) will appear to stop for a bit then jump over a pixel. And with this technique, the larger the scale up, the more obvious it is, because that rounding happens at the texel level.

Now some games just deal with this cleverly. For example, one of the artists of Celeste wrote that they use this technique, and I think you don't notice the character in Celeste moving jerkily at low speeds because you can't move less than a texel per frame (at least I couldn't). They even seem to move quickly at the apex of their jump!

However, even if I give my players and enemies sharp and fast movement to hide the issue, I know I want somewhat accurate (rigid body - i.e. rocks, planks, etc) physics. Which means that when objects slow to a stop from friction (or reach the peak of their arc) this kind of jittering would be noticeable there.

Attempt 4: Pixel Art Filtering

Pixel art filtering is an umbrella term for tricks to make pixel art look good (or at least "okay") when it isn't rendered properly aligned to pixels. For example, if you scale an image up by 1.5X (or any non-integer amount), then some screen-pixels will need to render pixels from multiple texels. Same thing if you rotate a pixel art image by non-quarter-turn increments.

I had disregarded this technique because I didn't need to do non-integer scaling or arbitrary rotations, and these tricks don't come for free. Ideally, what I wanted was:

1. Smooth camera scrolling
2. Smooth object movement
3. All texels drawn pixel-aligned (each screen pixel is influenced by exactly 1 texel)

Pixel art filtering gets you #1 & #2 (plus arbitrary scaling & rotation) at the cost of #3. But after my previous attempt, I realized that I was going to have to give up #3 anyway. (And I later realized I was going to need rotation, because boxes and planks subjected to physics do tend to rotate! Doh.)

Also, it turns out that Noita and Broforce (two of the inspirations for this project) both do some kind of pixel art filtering, so that made me feel better about this approach.

So, into the deep end we went! I had previously stumbled onto a jackpot of pixel art filtering roundups, and at a high level, the way all this filtering stuff work is all pretty much the same:

1. For each pixel on the screen, for each source image...
2. The graphics card looks at the screen-pixel and determines which texel(s) (image pixels) should determine its color.

In fact, that's how pretty much everything is rendered by the graphics card, so let's go deeper. Very high level, there's 3 ways that such color determination can be made:

1. "Nearest neighbour", where the pixel determines the color based on whichever texel is closest. This is what you want if you are drawing your images (and hence texels) perfectly aligned to screen pixels, but we'd already ruled that out.
2. "Linear blending", where each pixel color is determined by the color of the (usually) two texels near it (technically bilinear, since there are two dimensions so we'd be sampling from 4 texels in the usual case), and blending according to the distance away from each texel. Effectively this is guessing what the color should be in between two texels, and it works fine for non-pixel art. But for pixel art, the crisp pixels become smeared.
3. You write your own totally custom approach, by writing a fragment shader, which is a program for the graphics card that controls exactly how a screen-pixel's color is determined.

Obviously #3 is where the money is for us. Shaders let you do all sorts of amazing things, but in this case all the filtering techniques boil down to something like:

1. Use the image size to work out how wide each texel is
2. Use some built in functions (fwidth or other partial derivative magic) to work out how wide each screen pixel is
3. When it seems like the pixel is fully contained by a texel, use standard nearest neighbour sampling. (This alone reduces the bluriness significantly)
4. But if the pixel is crossing one or more texels, then do linear blending, but rather than weight purely by distance between the texels, instead weight the blend by distance from the edge of the texels.

If that makes no sense at all, I can recommend 40 minutes of this Pixel Art Games and nSight Shader Analysis screencast from Handmade Hero. (Naturally, I only found this after I had already finished all this work!)

Anyway, after several hours poking and stumbling I picked an approach and got it working in my little test project:

For comparison, here's the same thing but using just nearest neighbour filtering:

Then I just had to integrate the shader into the game, spend half a day figuring out how to get the pixel-scale-up determined implicitly by the camera transform matrix, correct the mouse coordinates, and use all this to line up the rendering of the pixel physics texture nicely with the rest of the things being drawn.. but this update is long enough so let's leave it at that.

Playable web build‎

Okay one more thing: getting a playable web build working was actually a pain in the butt. The partial derivative magic used in my new shader to infer the size of the screen and the function to get the size of the image texture are both not available in WebGL1 (which is how I draw pixels to the screen in your web browser, via Macroquad); they only exist in WebGL2.

There is a way to enable the former with an OpenGL extension, but I couldn't find a way to get a texture size*, so instead I hacked Macroquad's JavaScript code to create a WebGL2 graphics context instead. This means that if you have a browser from before 2018 (or are using a Safari version from early 2021.. it's always Safari lagging behind, isn't it?), then this won't work for you - try a newer browser instead.

Anyway here we go:

Nothing new gameplay wise in here, so same controls as previously. Mainly you'll be wanting to move around with A and D keys, and jump with W - and then revel in the fact that things are pretty smooth. You might even want to zoom in and out with = and - to see it all up close.

In fact, if you pay attention to each corner of the screen (zoom in a few times), you can see a red pixel surrounded by 8 orange pixels; those are my debug markers for the corners of the texture that's getting drawn to the screen. So as you move around you'll see those judder around to compensate for the movement of the camera, but the pixel physics world (the big dark grey rectangle, plus any pixels you draw with the mouse), the platforms, and the player will all stay smooth and non-juddery.

All the other rectangle shapes are still being drawn via Macroquad as primitive shapes, rather than as regular pixel-art sprites (images) - but in theory the same smoothness should be maintained when I swap them over to sprites. We'll see though - it wouldn't surprise me if I need to revisit this again at some point. (I also suspect the shader I picked does more work than other alternatives, but we can deal with that if it turns out to be a performance problem.)

*Okay, technically I could upload the texture size as a shader uniform before each draw call of a different texture. But Macroquad uses OpenGL immediate mode and so I think setting shader uniforms incurs a slight performance cost each time? Or at least it looks like it breaks sprite-batching, so I held off doing that. But I didn't benchmark it, so who knows, maybe it would be fine. It's not really a concern for the end product anyway since that will be desktop only: I won't have to worry about web compatibility.