Last week I quipped that networked multiplayer should work on a dialup network connection.
But would it really? I dove into simulating bad network conditions to find out.
Tubes
Previously I've explained that clients send data to each other to keep each other in sync via electrical mediums of varying reliability.
That was a lie.
In truth, for almost two decades, leading scientists have agreed that the internet is a series of tubes.Click to see America's Preeminent Internet Scientist
Accordingly, the way networked multiplayer truly works is that the game client talks to an interface in your computer, which converts electrical signals to and from fluid.
And said fluid then travels over aforementioned tubes to reach the computer of other players - and that is how game clients keep each other in sync.
Now, if you are lucky enough to have a fibre connection at home, you have an enormously-thick and leak-free tube that delivers fluid to your co-players via the most direct path possible.
Bad Tubes
On the other hand, if you have a dial-up or ADSL1 connection, you will have: 1 2
- Less bandwidth: your tube is thinner, so less fluid can flow through at once.
- Packet loss: your tube has leaks, so some fluid will get lost en-route.
- Higher latency (worse pings): your tube runs through an unnecessarily convoluted path, so it takes longer for fluid to reach its destination.
Now, a dialup connection has 40-50kbps of bandwidth, and last week's network stats showed that we require under 20kbps per connected player - so that box is ticked.
But a dialup connection also has latency of around 150ms and packet loss of up to 5%.
To test that, I set up a "virtual networking laboratory" on my computer:
- I made 8 "virtual network interfaces", each of which can be used by a running game client. 3
- These virtual network interfaces can send and receive data to and from each other, and the internet.
- I can apply "profiles" which add latency and packet loss to any or all of those interfaces. 4
Now let's put these fake tubes to work.
Testing 4 Players
First up, I wanted to test whether 4 players on mediocre connections would work.
Why? Well, this rollback networking approach I'm using is typically applied to 2 player games, and I was worried that with 4 players worth of inputs, there would be too many "rollbacks",5 which would kill performance.
For some reason all my friends ghosted my offer of a 10 hour unpaid work-day sitting next to me pressing buttons on command, so I made each player press their jump key 6 times per second to simulate some input. 6
Amazingly, the game is perfectly smooth, even though last week's network statistics display shows pings averaging 150ms and ~4% of all data between any 2 players gets dropped on the floor. 7
Those bad network conditions mean that sometimes almost every tick is getting rolled back, 8 but the game still remains smooth. Yay!
Testing Dialup
Now let's test the case where two players each have a bad quality dialup connection:
If you watch the blue player, you will notice some rollback artifacts:
- Blue "pops" upwards instead of traveling upwards on their jump.
- Blue seems to "warp" back and forth when they're moving.
Those artifacts are caused by the game rolling back and instantaneously correcting the blue player's movement when the yellow player's game client receives the blue player's inputs. And it sure looks glitchy.
Now, a combined 300ms of ping with ~10% packet loss is a spectacularly bad connection by modern standards: that's significantly worse than a connection from London to New Zealand.
But the beautiful part is, each player can still play the game just fine! Neither player actually feels any lag: it's only noticeable as (purely visual) glitches if they watch the other player. 9
Testing The Extreme
I still like the idea of offering an 8 player mode as a hidden semi-supported feature, so how does that work with the same settings as the 4 player test?
It works totally fine! Only slightly more artifacts are visible than in the 4 player test - go internet tubes, go!
And for the record, if you're on the same continent as all the other players, you would have pings of 80ms or less, rather than 150ms as in this test - and there are basically no observable artifacts in that case.
Web build is back
Yes, the playable web demo is back!
Temper your expectations though: it's running in local-multiplayer-only (i.e. offline) mode, so gameplay-wise it's the same as the previous demo from about a month ago.10
Still, I'm sure you were missing running around as a little astronaut-sorta-person wizard, so now you can get your fix again:
In truth there are some other characteristics of bad connections and bad networks that don't fit well into the fluid metaphor (out of order delivery of packets, corrupted packets, buffer bloat, etc).
But, in the context of game networking, you usually design your network protocol to guard against those things (e.g. ignore packets older than ones you've already handled), so you end up being able to approximate the effects of those other bad things by limiting bandwidth, adding latency, or adding packet loss.
Also, these bad things can come from problems in the wider internet in-between your and your friends' connections (especially increased latency from bad routing) - but that is much less common than problems in the "last mile" of internet connections.
I thought this was going to be a nightmare of many virtual machines, but nope, it's not at all! It's just a casual 70-odd ip (and friends) commands:
You create a virtual ethernet pair, put half of the pair in a new "network namespace", keep the other interface of the pair in the root network namespace, and run your application inside the new namespace, so it is forced to use that virtual ethernet interface.
Repeat that a few times and then you can connect those interfaces in the root namespace together in whatever topology you please using iptables or bridge them together (optionally setting some iptables rules to allow bridge packets to be NAT-routed to the internet via your default actually-corresponding-to-real-life-hardware network interface).
It took about a half day of fiddling with the help of an llm, just because I wanted to package it into a somewhat reusable script; you can steal my WTFPL'd script if you want.
I used netem, which basically lets you set rules for packet loss, adding latency, packet reordering, etc on a particular network adapter in a compact (in my opinion, a little too compact) domain-specific language.
I predefine named sets of rules so I can apply them easily using my script from the previous footnote.
If you develop on Windows then you can use Clumsy instead - it's less flexible but definitely easier to use.
(But I use Linux because filesystem performance is much better than on Windows - enough to make a meaningful difference to my game's compile speed and hence my iteration speed & productivity.)
Take that friends! If you had realized you were so easily replaceable, perhaps you would have taken my offer of unpaid work more seriously.
Rollback = get some new input from a remote player, rewind the game simulation to apply that input at the right point in history, then fast forward to the present time with that new input in mind. The fast forwarding costs a lot of CPU time.
For the non-gamers: 150ms ping with 4% packet loss is a pretty bad connection. You'd probably put your fist through your monitor if you tried to play a competitive online game with a connection like that!
The bottom right graph is new and shows you rollbacks as they happen for the yellow player; each bar is a rollback, and the height of the bar is the amount of ticks (slices of game time) that had to be resimulated as a result of the rollback. Notice the rollbacks are almost non-step when the blue player spins around casting fire everywhere!
Although, if you look at the bottom right rollbacks graph, you can see that there are a lot of rollbacks happening, and that means both players will require a more powerful computer.
If either player has a slower computer, the game will slow down - or worst case, do some start-stop stuttering.
In the end, physics will win; there's only so much I can do for my poor remote-Australia players (they're more likely to be on Starlink these days, but I've heard that also has really bad latency).
Well, it should perform a little better, and I managed to get the built-in profiler I'm using working on the web build too. But that's not really that exciting.