melonDS RSS http://melonds.kuribo64.net The latest news on melonDS. Immediate plans -- by Arisotura http://melonds.kuribo64.net/comments.php?id=81 Sun, 17 Mar 2019 19:32:46 +0000



I also wanted to fix one of the issues with local multiplayer: when more than two players are involved, clients receive replies sent by other clients, which they shouldn't receive, and this likely contributes to it shitting itself big time.

But alas, this will be more complicated than anticipated. This is also the main reason why local multiplayer pretty much stagnated after melonDS gained its "wifi emulator" reputation back in 2017: emulating local MP is a large pile of issues that are all interconnected. Wifi emulation in melonDS is more or less a pile of hacks, and it works, but it's full of issues. Some stem from incomplete understanding of the DS wifi hardware (after 15 years. welp), but most of them are timing issues.

(which are also why I'm pretty much pessimistic about ever connecting melonDS to a real DS)

Local multiplayer 'a la Nintendo' works on a host-client scheme detailed here. Long story short, the host polls the clients at a regular, small interval (for example, every 4ms for Pictochat). Dealing with the sheer amount of packets transferred is in itself a challenge.

But that's not all. When the host sends its packet, each client is given a window within which it should send its response. Miss your window and it's considered a failure.

This works well with actual DSes because they're all running at the same speed, so the timings are reliable.

With melonDS, it's another story. We get lag inherent to the platform on which we're running: the network stack, thread scheduling, etc... Running multiple DSes in one melonDS instance might help alleviate these lag sources, but it wouldn't be a perfect solution either (it would likely be running the DSes on separate threads).

We also learned by experimentation that the framerate limiter is a problem, and connections worked better when disabling it. As silly as that sounds, it makes sense when looking closer. When disabling the framerate limiter (or when running below 60FPS), the melonDS instances run as fast as possible, and they may end up running at roughly the same speed, consistently, which makes for a better connection (less chance that MP replies miss their window). However, when enabling it, your melonDS instances may be running at any speed above 60FPS, but they will be spending several milliseconds per frame doing nothing in order to bring the framerate back to 60FPS.

Which, you guess, is bad bad bad for MP communications. The wifi system is driven by the emulator's scheduler, so it will end up running faster than it should, squishing MP reply windows and making it way more likely that replies get dropped.

And indeed, disabling the framerate limiter greatly reduces the amount of MP replies missing their window, even if the framerates are barely above 60FPS. The framerate limiter might be a bit zealous there.

However, the emulator instances might run at different framerates and possibly desync. And that's not too convenient if they end up running at absurdly high framerates.

blarg.

To get anywhere with this local multiplayer shito, we'll need a redesign. No amount of hacky solutions will get us anywhere with this pile of hacks.

First part is how the wifi system is driven.

melonDS runs ~560190 cycles per frame, which is 33611400Hz under ideal circumstances, close enough to the DS clock frequency.

Wifi is updated every microsecond. The handler is called every 33 cycles, which means that one emulated microsecond actually lasts ~0.9818 microseconds. I'll spare you the nerdy calculations about how much that represents in offset, because so far that hasn't prevented it from working.

But on the DS, the wifi system is driven by a 22MHz clock, independently from the system clock. So driving melonDS's wifi system independently from the scheduler would not only be accurate, but also isolate it from the core's variable execution speed. However, two main issues arise from this:

1. We need to keep the core somewhat in sync with the wifi system, or the game would eventually shit itself. How to synchronize and when to do so? That's the question.

2. The current wifi system needs to be updated per microsecond. If we put it on a separate thread, how to take care of this without pegging the host CPU and killing performance? We could design a scheduler similar to the core one. However the wifi system has a readable microsecond counter, and we need to take care of that somehow. Without per-microsecond updates, we can only approximate it and hope it will be good enough.


To further prove my point:




welp.


We'll think of all that later. Past 0.7.4, we're going to focus on the hardware renderer.]]>
http://melonds.kuribo64.net/comments.php?id=81
Getting somewhere, finally -- by Arisotura http://melonds.kuribo64.net/comments.php?id=80 Sat, 09 Mar 2019 01:10:25 +0000

Finally, ClIRC is cooperating!

This took a bit of hackery to get around the limitations we face when working with plain old BSD sockets. Namely, how we're handling TCP acks for now.

Regardless, it's finally working without exploding! Well, I don't know about things like altWFC. But ClIRC is working about as well as when using direct mode.

We'll be polishing this and testing it with things like altWFC. You can expect a release real soon.

(also, a side effect of using sockets is that closing melonDS terminates connections correctly, which is nice for testing)]]>
http://melonds.kuribo64.net/comments.php?id=80
Change in plans -- by Arisotura http://melonds.kuribo64.net/comments.php?id=79 Tue, 05 Mar 2019 14:15:23 +0000
So I figured I would just rewrite it to use regular sockets instead of libpcap. There will be a lot less issues getting things to work this way, and this has the significant advantage that it doesn't require libpcap, and likely can work without requiring admin privileges.

The downside is that we have less control this way, as all the low-level TCP/IP details are handled by the host OS, so some things might break, but I'm confident that this will work just fine for the most typical uses, which involve standard TCP connections.

Anyway, I have DHCP, ARP and DNS covered. I started work on TCP, it's now able to initiate a connection to a server. Now I need to get the actual data exchange working.

I'm also unsure this can work for any use case where the DS is used as a server, but for now we don't have to care about that. And there's always direct mode.]]>
http://melonds.kuribo64.net/comments.php?id=79
Updatez0rz -- by Arisotura http://melonds.kuribo64.net/comments.php?id=78 Wed, 27 Feb 2019 18:57:04 +0000

1. Coming-out, sorta.

I am, well... nonbinary, likely some flavor of girl.

I'm a bit sick of the constant assumption that internet users are male by default, or that 'there are no girls on the internet'. Other genders, trans or not, are here too.

If you don't know, you can either ask or use gender-neutral prounous, it's still better than perpetrating the 'masculine by default' norm.


2. Network shito

The issue I mentioned with the DS receiving packets addressed to the host, also happens the other way around, ie the host receives traffic addressed to the DS. This might be the cause for the lag I observed.

Still thinking of a workaround to this. May need to read about network routing and all.


That's all folks!]]>
http://melonds.kuribo64.net/comments.php?id=78
Indirect mode progress -- by Arisotura http://melonds.kuribo64.net/comments.php?id=77 Sun, 24 Feb 2019 12:36:11 +0000
The idea is that outgoing network traffic is altered to make it look like it's coming from the host machine, so that it will get past wifi access points without trouble. In return, incoming traffic is addressed to the host machine, so I first took the easy path of redirecting all traffic to the emulated DS regardless.

Bad idea tho.

This caused all connections on the host machine to die constantly. Reason is that on the DS, the sgIP stack is receiving TCP traffic it did not initiate (because those are destined to the host), and interfering with these connections, likely just trying to kill what it perceives as bogus connections.

To work around this, we have to keep track of TCP sockets. Examining the flags in the TCP header, we know when a connection begins, and can know to only redirect incoming traffic if the source IP and source/destination ports match an existing socket.

Basically, NAT.

This gave promising results with ClIRC, but for some reason there's quite some lag on messages sent from ClIRC, which I don't remember being that laggy.

The current implementation will be limited to TCP and UDP, but for uses like AltWFC, this should be enough. For more advanced shito, there's always direct mode, which I think can be made to work on a wifi connection with some hackery.]]>
http://melonds.kuribo64.net/comments.php?id=77
Full steam ahead! -- by Arisotura http://melonds.kuribo64.net/comments.php?id=76 Thu, 21 Feb 2019 12:32:33 +0000
One of the main reasons why coding is going at this weird pace, besides the recent surge in local political activity and moving to a betterer apartment, is, well, how I achieve work. Whenever big-ish changes are required (designing something new, redesigning something...), I need time to think about it, and sometimes just do something else entirely for a long while. Until I go 'fuck it' and just code the damn thing.

So, I started coding the damn thing.



The "bind socket to any address" setting for local wifi has been moved to this spanking new wifi settings dialog.

There's also the new UI for setting up online mode, so you can, you know, pick a network adapter, etc.

That's why it took a while btw. It displays adapter information such as MAC, IP, DNS servers, which aren't provided by stock libpcap. So I had to use the Win32 API to retrieve those (and I have yet to code the Linux equivalent). We also get more meaningful adapter names this way, instead of having them all be named 'Microsoft'.

Then, there is the 'direct mode' thingablarg. What's that for?

Direct mode is basically the old way of forwarding network traffic: directly passing packets from/to libpcap. While this is the most straightforward and less likely to shit itself, it has the downside that it requires an ethernet connection. Reason is that under this mode, melonDS is seen as an entirely separate device on the network, with its own MAC address and all. On an ethernet connection, you can just use promiscuous mode to retrieve the packets sent to melonDS. But wifi is a different affair. Devices need to be associated to the access point. As that isn't the case for melonDS, its packets are just thrown away.

To work around this issue, not-direct mode is being developed. The idea is to do DHCP/NAT inside the emulator. melonDS would be in a little subnet along with its access point melonAP and its miniature DHCP server, and a bridge to the actual network. This way, packets sent to melonDS would bear the MAC address of your computer instead of that of melonDS, and wifi access points wouldn't throw a fit.


Hoping I can get this to work. But first, I have to finish fixing up the water heater.]]>
http://melonds.kuribo64.net/comments.php?id=76
Some news -- by Arisotura http://melonds.kuribo64.net/comments.php?id=75 Sat, 26 Jan 2019 00:45:00 +0000
Things have slowed down a bit since 0.7.3. I wanted to address some of the newer timing issues, but there's bigger trouble arising from this, I'll get to it.

Also, attempting to get things done(tm) in real life, which is another reason why it's slowing down.

Regardless, we're still alive and kicking.


For melonDS 0.7.4, I want to fix some interface issues, and finally add some interface for setting up online-mode wifi. There might be other fixes, but that release will hardly be a revolution.


0.8 will be, as promised, the hardware renderer and upscaling. I don't want to delay this any further, so instead I'm going to delay what will come next.


For 0.9, or something like that. We're running into timing issues that can't be addressed by simply tweaking our timing values for cached memory. Not without breaking other games. I am reluctant to using per-game hacks.

So, the last line before the dreaded "per-game hacks or full ARM9 cache emulation" situation: perfecting CPU timings. The current CPU timings are imperfect, making things run faster than their hardware counterpart, especially on the ARM9 side. The ARM9 is complex so working out its complete timings will take a while.


Also, there's been some changes to this site, and we're not done :)]]>
http://melonds.kuribo64.net/comments.php?id=75
melonDS 0.7.3 -- by Arisotura http://melonds.kuribo64.net/comments.php?id=74 Sat, 05 Jan 2019 13:05:52 +0000

Something that should have been done long ago, and has finally been done: the emulator's main loop was rewritten to use absolute timestamps to keep track of cycle counts. Overall, it's more reliable (far less chances of desyncs), more accurate, and also a bit faster, and the code is cleaner. So we win on all fronts.

This should fix the recent flickering issues, like that of Colour Cross.

The downside is that this breaks savestate compatibility with older melonDS versions.


In the same vein, there's another fix to GX timings. We knew that vertex timings were different when a vertex completed a polygon, but we just found out that timings also differ based on whether the polygon passes culling and clipping. This will help with games that need their display lists to be running fast enough.


To hopefully fix the saving issues, we built a new database, resulting of a mix between the Advanscene XML and savelist.bin from the R4 Wood firmware. For better detection, the new romlist.bin indexes games by their serial code rather than by the ROM's CRC32, which also means that it will work with hacked ROMs. This means you will need to make sure to replace romlist.bin with the newer one if you're extracting this melonDS release over an older one.

There's also a particular fix for Pokémon Mystery Dungeon - Explorers of Sky, which uses 128K EEPROM. It should now save properly.

However, note that the correct savefile size for this game is 128KB. If you have a savefile which is 256KB, the game will read it fine but fail to save to it (because that size is detected as FLASH memory and not EEPROM). You can fix this by opening your savefile in a hex editor and deleting the upper half of it. This is especially important if you use a savefile coming from DeSmuME as it will create a 256KB file.


We also have a neat little pile of UI improvements.

melonDS now supports hotplugging joysticks. That being said, it still defaults to the first joystick available on your system. I'm not quite sure how to go about handling multiple joysticks.

Under Linux, the crashes coming from the input config dialog should be fixed.

The main window is smarter about remembering its size: it also remembers whether it was maximized.

There's a tentative fix for hiDPI under Windows, which should atleast make the screens scale correctly.

There's a new menu for setting the window size to an integer scaling factor (1x to 4x).

The setting for savefile relocation when using savestates is now disabled by default. While the feature may be useful, having it enabled by default was confusing for unaware users.


And finally, the misc fixes:

* fix STRD_POST (fixes music in Just Sing - Vol 2)
* 2D: fix blending bugs
* add support for 8bit reads to DISPCNT/BGxCNT (fixes The Wild West)
* make nocashprint also work in ARM mode


Enjoy!


melonDS 0.7.3, Windows 64-bit
melonDS 0.7.3, Linux 64-bit

melonDS Patreon if you're feeling generous]]>
http://melonds.kuribo64.net/comments.php?id=74
Some brighter news tho -- by Arisotura http://melonds.kuribo64.net/comments.php?id=73 Thu, 03 Jan 2019 00:55:26 +0000

Regarding the Colour Cross issue, I have an idea. It seems to stem from the GXSTAT busy flag, like, sometimes it's still set when it should be cleared because we aren't syncing the GX often enough.

I could go back to the old way (syncing it per-opcode), but I'm afraid this is going to be suboptimal. The better solution I have will require a bit of reworking how melonDS handles timing. As a bonus it would be more straightforward, easier to maintain and less error-prone.


I still have no idea about Spellbound though. I found that DMA timings can differ depending on how source/destination addresses are updated, but that doesn't apply in the Spellbound case. So, still no idea there.


I guess we can try releasing 0.7.3 after fixing Colour Cross, though.


I also have a few ideas for improving this site. A nice one to have would be an intro blurb before the blog posts, showing off melonDS and why it's awesome and you should totally download it, like the lolSNES site has.

Might also change the download page a bit, namely, to add downloadable extras like the latest romlist.bin or the dsbf_dump.nds copy that's been squatting the Kuribo64 uploader since forever.

Maybe even later, some form of compatibility list. Which will need some dedicated testers to fill it and keep it up to date.

If you have ideas how to improve the site, visually or technically, I'm open to all suggestions!


I'm contemplating running a sorta-buildbot on blargcity (the server), like we had for lolSNES. Looking at MinGW and how we can use it to produce 64-bit Windows and Linux builds.

Might even produce 32-bit builds if it's any worth doing. Noting that it wouldn't run on Windows XP as it is, libui uses several APIs that were introduced in Vista.

One fear I have over that is that having a buildbot might kill incentive to do proper releases. Kind of what happened with lolSNES.


Oh well.]]>
http://melonds.kuribo64.net/comments.php?id=73
Happy new year from melonDS -- by Arisotura http://melonds.kuribo64.net/comments.php?id=72 Tue, 01 Jan 2019 16:23:26 +0000
This year is not starting too well. I implemented an instruction cache in melonDS, and what are the results? Absolutely none whatsoever (other than being a bit slower). It achieved as much as farting in the wind. The problem games I previously mentioned are still giving me the finger.

My motivation is utterly sapped.

..

We hope this year will be better for you.]]>
http://melonds.kuribo64.net/comments.php?id=72