The netplay saga, ep 2

< The local multiplayer saga, season 2: NetplayThe netplay saga, ep 3 >

The netplay saga, ep 2
Mar 27th 2023, by Arisotura

I've been building the basic network infrastructure for netplay lately. Oh, the fun that is dealing with synchronization.

If you remember the graph from the previous post:

Implementing this proves to be tricky, because since each individual instance there is its own process, there's a lot of moving parts. So first, we're going to name them.

Assuming player 1 is the player who initiated the game: player 1's instance 1 acts as the game host, while player 2's instance 2 and player 3's instance 3 act as game clients. The game host transmits useful information to the game clients, tells them when to start running, ...

Then player 1's instance 1 acts as a mirror host: players 2 and 3's instance 1, the mirror clients, connect to it, and receive their input from it, thus mirroring player 1's input on players 2 and 3's machines. Similarly, player 2's instance 2 and player 3's instance 3 are also mirror hosts.

As is typical with netplay implementations, inputs are delayed by a fixed amount, which is hardcoded to 4 frames in the current test branch, but will be configurable in the final product. The basic idea is to delay inputs a bit on all sides to counter network lag.

Each input frame sent by a mirror host is given a frame count, which lets mirror clients make sure to apply that frame at the exact same time as their host, thus ensuring all sides are given the exact same inputs. If a mirror client runs out of input frames (because the mirror host is running slower), it will need to block until it receives input frames -- missing an input frame would cause a desync.

During a local multiplayer game, this has shown to be enough to form a somewhat viable netplay implementation: this crude synchronization mechanism, combined with local multiplayer sync, do a good job at keeping all instances in sync. But when the players aren't engaging in a local multiplayer game yet, there is the possibility that mirror clients run too slow and end up lagging behind an awful lot.

My basic idea for dealing with this was to have mirror clients report their frame count, and the mirror host then waits for them to catch up if any of them is more than 16 frames behind. This mechanism doesn't have to be tight, but just enough to keep things reasonably synced up when local multiplayer sync isn't doing the job.

Except it didn't work. It always caused big fat lag spikes when starting the game. I feared I was running in an interlock situation. I thought again about how my sync mechanism worked, checked my code, and, well...

if (clientframes < (NDS::NumFrames - 16))
{
event.peer->data = (void*)1;
block = true;
}
else
...

Basically, checking if the mirror client's frame count (clientframes) is less than the current frame count (NDS::NumFrames) minus 16. Innocuous code, huh?

clientframes and NDS::NumFrames are unsigned integers. This means that if NDS::NumFrames is less than 16, (NDS::NumFrames - 16) overflows to a large positive number, causing erroneous blocking.

Oops.

Casting to signed integers fixed the issue. So as of now, the whole sync mechanism seems to be behaving as expected. This will need more serious testing, though. I have only tested with two players, because I only have so many viable computers.

This is still far from being a viable netplay implementation, though. Because for this to work reliably, we need to ensure that mirror clients maintain the same state as their mirror hosts.

As of now, they are fed the same inputs, but the initial state isn't guaranteed to be the same. I observed this in Mario Kart, for example: each side may pull different items from boxes, AI players will behave different, etc. In my test branch, I hardcoded the RTC to a fixed time, but we will need to adopt a solution for it. But I believe Mario Kart may be initializing its RNG from other sources, like firmware configuration data or save data, so we need to ensure that these are properly synchronized when starting a game.

And even then, there is another thing I'm worried about: uncertainty in the local multiplayer comm resulting in slightly different state on each side. We will have to see whether this can be a problem, and if so, how we might address it.

There is also a lot to be done on the user interface side of things. Failing gracefully when something bad happens, presenting an interface that is user-friendly, and so on.

Speaking of, I'm open to suggestions and input from end users regarding this feature. If you have any ideas, I opened a thread for them: right here.

13 comments have been posted.

< The local multiplayer saga, season 2: NetplayThe netplay saga, ep 3 >

nyanpasu64 says:
Mar 27th 2023

I wonder if subtracting unsigned integers is a code smell or bug worth turning into a compiler warning. I've gotten many bugs from it, but such a warning would have many false positives as well. Rust debug builds panic on subtraction overflow, but release builds don't check lmao.

Back in C++ land, -Wtype-limits catches at compile time, bugs like (x - y >= 0), but not cases where both sides of a comparison are variables. One alternative to casting to signed is to move the subtraction to the other side and turn it into an addition, which might be less readable but doubles the usable range before integer overflow. Sometimes I'll add a comment of the untransformed expression before the actual expression.

^.^ says:
Mar 27th 2023

Glad to know you are back to action ;)

The DS only has only 4 MBs of ram (ok the expansion pack/DSi/debug versions have more), it would only take a few seconds to upload it at the beginning for today's connections, wouldn't it be easier to send it to the other players's instances to ensure the initial state is 100% identical?
I remembered this article from the Dolphin blog and the amount of MADNESS found inside all the possible settings combinations caused a lot of headaches back then: https://it.dolphin-emu.org/blog/2018/08/01/dolphin-progress-report-july-2018/

Also this is going to be 100% trust based, a la don't-netplay-with-strangers-they-might-hack-your-pc-or-at-least-cheat I guess? I don't know if there's any reasonable mitigation that could be put in place...

Sorry for the long comment ''''^^

big bad baz says:
Mar 27th 2023

this is awesome!! why don't you stress test it with Dragon Quest IX's co-op feature? it is effectively open-world, 4 players can be in one session and players can be at different places in the world & do things independently

Minessota Klei says:
Mar 27th 2023

Hi melonDS team,

These technical explanations are cool, it lets people like me who aren't in the area understand a little of the innards of the emulation process!

Playing Bomberman 2, very good (^o^)//

lucaspltn says:
Mar 27th 2023

This looks like a _lot_ of complicated stuff...Amazing how much you were able to figure out so far!! 😮

^.^ says:
Mar 28th 2023

BTW, I imagine not having to render stuff on the screen for the other instances saves some processing time on the gpu, how big is the difference?

Arisotura says:
Mar 29th 2023

it wouldn't save a lot actually -- due to the display capture feature, we would still have to render everything to make sure all sides maintain the same state.

Generic aka RSDuck says:
Mar 29th 2023

melonDS processes all graphics everything on the cpu (except for 3D graphics when using the OpenGL renderer).

There is definitely some gain to be found from what would essentially be infinite frame skip. Though it depends a bit, e.g. the software renderer can already run almost completely independently in another thread. So if you have enough cpu cores, it wouldn't bring much improvement, though if not, it could free up a core for you.

One issue would be that if a game uses display capture (e.g. for dual screen 3D or some effects like motion blur) rendering graphics cannot be skipped without loosing determinism. It would probably still work, but it could lead to some subtle issues.

Arisotura, we can still skip most of GPU2D and skip 3D rendering conditionally based on whether display capture is enabled.

^.^ says:
Mar 29th 2023

Thanks for the reply! I'm glad my suggestion was useful =D I hope my other suggestion on RAM uploading to ensure the same state across instances was too.

UnluckySpade7 says:
Mar 31st 2023

Does this method use any kind of predictive netcode? you might want to look into GGPO if save states are fast enough.

SIGMA says:
Apr 1st 2023

@UnluckySpade7 I already suggested that in the thread but it doesn't seem feasible.

Batman2044 says:
Apr 6th 2023

Hello Melon DS team,

I have greatly enjoyed your emulator and it has worked near flawlessly! Just a personal request, can you please add in more hotkeys such as emphasize bottom/emphasize top screen functions as well as a hotkey setting to open/change roms? Thanks for all the hard work and have a great week!

Jawlshy says:
May 28th 2023

Can you please make sure when youre testing this new netplay, to test it on Metroid Prime Hunters as well? There is a large active player base for us and the lag has terrorized us all for for like 17 years! It would be greatly appreciated <3