melonDS RSS http://melonds.kuribo64.net The latest news on melonDS. JIT beta builds and introducing myself -- by Generic aka RSDuck http://melonds.kuribo64.net/comments.php?id=107 Sat, 07 Dec 2019 00:20:22 +0000
As Arisotura has stated in her last post, there's now another head working on melonDS. I'm Generic on IRC and RSDuck on Github.

For some reason I started messing with DeSmuME a bit more than a year ago, trying to write a JIT recompiler targetting ARM64, to improve performance on the Switch. After many struggles something working came to be, but it missed it's original goal of being fast for several reasons (see the link if you're interested).

After recovering from the disappointment I put my hopes into melonDS, but since it lacked any JIT compiler (besides the abandoned attempt by Arisotura herself) I wanted to start with a x64 recompiler. Skip forward a bit and the JIT started take on form. After taking everything apart and putting things more cleanly back together around the beginning of last summer things only got better, with most instructions being recompiled and the opening of the pull request on Github, which later got merged.

At this time the ARM64 backend was started this time avoiding my previous mistakes. With autumn it reached a similar level of completion as the x64 backend while both backends received some optimisations. Until this point we were catching up to DeSmuME but at this point started to surpass it. Here is where I want to thank all developers of open source emlulators whose dynarec I was eable to look into and especially the developers of the JIT of Dolphin. We're not only using their code emitter but they also gave me some great advice on IRC!

With some further optimisations of the 2D GPU emulation using ARM NEON instructions, most 2D games already run at fullspeed on the Switch, while most 3D games run at fullspeed with overclocking. The ARM64 JIT and the GPU optimisations are currently located in my own fork and are distributed by me in binary form on Gbatemp in an admittedly chaotic way.

I am very grateful to Arisotura for creating the emulator in the first place, as well as trusting in me and my work. For the future the already mentioned GPU optimisations are my current focus. I'm currently rewriting the old ARM NEON optimisations to be more clean and adding some additional ones which hopefully increase the performance a bit more. The logical step following would be a NEON 3D GPU backend. I have some more plans lying around, e.g. there is still one major JIT optimisation we're missing out on (fastmem).

So but now enough storytelling, here are the builds:

Note:

These builds are my JIT branch (generic_jit) merged with master into a separate branch.

The JIT recompiler has to be enabled in Settings -> Emulation. The default maximum block size is 10, this is a very conservative number, as I don't know of any game which breaks with high maximum block size. If you experience any bugs which can't be reproduced with JIT disabled, please try lowering the maximum block size and disabling one or both of the options Literal Optimisations and Branch Optimisations and let me know. Though there already has been some testing through people building from source themselves and the semiofficial Switch builds, both uncovered some bugs.

melonDS 0.8.3 JIT beta for x64 Windows
melonDS 0.8.3 JIT beta for x64 Linux
]]>
http://melonds.kuribo64.net/comments.php?id=107
Status update II -- by Arisotura http://melonds.kuribo64.net/comments.php?id=106 Thu, 05 Dec 2019 22:34:34 +0000

First of all, things have been a bit shitshow-y, and involved a few bursts of depression, but, finally, it's there, I have my apartment now. I'm not quite finished settling here, but that's a big relief for now. Plus, the place is fairly nice! Except for the wall sockets being upside-down, but, eh.


Next, I'm being sucky at delivering a JIT beta. Thing is, unlike the DSi branch which was hacked together by me, the JIT branch is Generic's work, so I will need to take a while and get familiar with it.

In the meantime, I figured I would let Generic handle things himself. I gave him access to this blog and all, so he will be able to post a beta build, which he said he would do tomorrow.

I figure the best way to work with this long-term is to work as a duo. I'm less skilled on things like fast CPU emulation, and more skilled on other parts of emulation, so other people like Generic can complement me nicely there and together we can deliver an excellent product.

And, holy fucking shit, reconciling the JIT and DSi branches is something I'm totally looking forward to. At this point, the DSi branch already has several conflicts with the master branch (it's based off 0.8.1, so yeah), and, no idea about the JIT branch.


Immediately tho, I'm going to improve the OpenGL renderer. This means reorganizing some of the code so it's easier to work with, fixing some longstanding issues like the case of dual-screen 3D, and adding some filtering.

You can follow the progress in the appropriately named blackmagic_II branch.


Hell, even writing this post is taking me forever.


Also, today's Dec 5, and it's the general strike day. Seeing how successful it was, if they manage to keep going at this rate (and they totally intend to keep going), president Macron is going to have a hard time. But, this also means that certain things may be delayed, or whatever, so we'll see.]]>
http://melonds.kuribo64.net/comments.php?id=106
Status update -- by Arisotura http://melonds.kuribo64.net/comments.php?id=105 Sun, 17 Nov 2019 10:28:33 +0000

Couple things though.


On melonDS, things are going to be mostly on hold until I manage to get a proper apartment and am all settled in and all. But, two things. First, Generic said that the JIT branch is ready for a beta release, so I'll be doing that. Second, as a next direction to take the project, I'm thinking about adding support for AR cheat codes.

I'll also be trying to fix the remaining issues with the OpenGL renderer, but some are going to be tricky. Also, things related to this renderer: I need to come up with a not-too-hacky solution for dual-screen 3D and upscaling, and to add support for filtering in OpenGL mode (including xBRZ or something of that ilk).


But, blegh. The apartment. I visited one a while ago, now I'm going to have to put together the paperwork and hope it isn't too late. There are other apartments I can try, but, hey, I liked that one. So, here's hoping.


Oh also, this is Nov 17, birthday of the yellow vest movement. French protest movement that started one year ago, was originally against a gas tax raise but turned radical and against precarity in general, which seems to be a common theme these days (thinking about Hong Kong, Chile, ...). Despite the fact that the movement as it is will not bring revolution, it seems to want to never end... (what kind of protest movement lasts one whole year?)

But we can see that it's having an effect on protests in general, they turn more radical now. Like recently, a student from Lyon set himself on fire out of despair after being thrown into precarity. Had that happened two years ago for example, nobody would have cared, it'd just have been "another depressive sicko attempts suicide, film at eleven". But instead, this sparked several student-led protests and blockades, including one that broke into the Ministère de l'Enseignement Supérieur.

Dec 5 is going to be the general strike, too. I first thought that would be lame, as most of these attempts were a while back, but, now, within the different context, this looks like it's going to be quite something. May also cause some trouble on my end, but, we'll see.

Anyway, happy birthday to the yellow vests. I've been following the movement closely, and even taking part in it, as it gave me hope when I was at the worst of depression, so this means a lot to me.]]>
http://melonds.kuribo64.net/comments.php?id=105
When things turn out unexpectedly simple -- by Arisotura http://melonds.kuribo64.net/comments.php?id=104 Sun, 20 Oct 2019 10:02:42 +0000
Shows that taking a break does wonders.

Just yesterday, I idly looked at that bug again. Reproduced the freeze, then pressed F11 (debug key printing some status to the console) a few times to see if I could see a pattern. The ARM9 seemed to exhibit typical 'idle thread' behavior; the ARM7 however, seemed to get stuck in a loop. A quick glance at my IDB of the firmware code confirmed this:

AES wait routine

If you aren't fluent in ARM assembly, this function waits for the AES engine to finish whatever crypto operation it is doing, by checking the busy flag in AES_CNT.

But, in our case, the ARM7 got stuck inside this because, somehow, said busy flag remained set forever.

Some logging of AES operations shown that, while loading the game, the firmware would start a decrypt operation with a block count of zero. melonDSi didn't check for that case, and instead waited for data input that would never come (heh), causing the busy flag to remain set forever. You guess how this goes.

After quickly fixing this bug, I tried loading a cart game again. It still froze on a white screen, but the logs showed it was getting further. In particular:

UNIMPLEMENTED ARM9 NDMA3 START MODE 04, 04100010->021153A0 LEN=1 BLK=1 CNT=A4044000

That meant the game was actually booting, and attempting to load its data, using NDMA ('new DMA' implemented in the DSi) for whatever reason. NDMA start mode 04 is for cart slot DMA, which is triggered every time a data word is transferred from the cart.

This was quickly taken care of, too. A bunch of the NDMA start modes mirror those of 'old' DS DMA, so I just hooked those into the functions responsible for taking care of DS DMA. The only one that needs more extensive checking is GXFIFO mode, since it has a maximum block length of 112 words, I'm not sure how that interacts with the block subdivision system of NDMA.

Anyway, with this, I was finally able to boot DSi-enabled games.

Nothing terribly exciting to show tho. After all, most of the DSi-enhanced games out there don't have much extra features other than making use of the DSi cameras, so we have to do something to emulate them.

Loading DSi ROMs is also a bit finicky as they have to be one specific way for it to work (right now, with the DSi-specific data encrypted). And, some of the ROM dumps out there lack the DSi-specific data, so it would be good to add some verification and warn the user if their ROM is a bad dump.


Speaking of which, I am thinking of the ways I can make this evolve, now that it's atleast working to some extent. Namely:

* lowering the entry barrier, that is, upgrading dsidumper to dump all the required material in one go

* getting rid of some of the required files. boot2_7/9.bin can be sourced from the NAND directly, provided I can find a way to derive the required decryption keys. initmem7/9.bin could be made optional, the required data can also be sourced from certain particular BIOS dumps (the ones that NO$GBA requires) (and dsidumper could also be upgraded to produce such augmented BIOS dumps). Not sure about aeskeys.bin.

* rewriting the trainwreck that is the SD/MMC controller code. As an example, Flipnote Studio tends to softlock while saving due to bugs in that code. It was written mainly to get things working, and it shows, but now that we know that things are working, we can afford to rewrite it cleanly. Some other DSi parts (AES engine, ...) could use some cleanup, but they aren't as bad.

* adding features such as cameras and wifi connectivity

* ability to direct-boot DSi games, and work out something for DSiware

* the holy grail, eventually, merging all that back into regular melonDS]]>
http://melonds.kuribo64.net/comments.php?id=104
melonDSi beta release -- by Arisotura http://melonds.kuribo64.net/comments.php?id=103 Tue, 15 Oct 2019 22:21:19 +0000
Anyway, here we go: we present you melonDSi, in unfinished beta glory. The JIT beta will follow, but I first want to ensure Generic is comfortable with it.

The end goal will be to merge this with regular melonDS once it's in a better state, but, for now, we're providing it for the curious folks.


How to use melonDSi

BE SURE TO READ THIS BEFORE USING MELONDSI. You are using a beta release, provided with zero polishing and with a debug console. I expect that you know what you are doing to atleast some extent.

melonDSi will require several files to be dumped from a DSi. It's a bit messy right now, but we hope to come up with more user-friendly solutions in the future.

So, if you have a DSi, you might want to install unlaunch on it.

A quick rundown of the files melonDSi requires:

* bios9.bin / bios7.bin: DS-mode BIOS images. Those are the same as the ones from a DS. They shouldn't be required to get DSi software running, but melonDSi will refuse to start if they aren't present.

* bios9i.bin / bios7i.bin: DSi-mode BIOS images. Those can be dumped from your DSi using the good ol' dsbf_dump.nds.

* firmware.bin: SPI FLASH image. See above. dsbf_dump.bin yields a 'FWxxxxxx.bin' file which you need to rename. It should be 128K.

* aeskeys.bin: AES keys at boot time, dumpable using dsidumper.

* initmem9.bin / initmem7.bin: Boot-time contents of ARM9 ITCM and ARM7 WRAM, containing several other useful keys. See above.

* nand.bin: DSi NAND image, ideally dumped with fwTool. This should be dumped from the same console as dsikeys.bin and initmem9/7.bin.

* boot2_9.bin / boot2_7.bin: Second-stage bootloader, to be extracted from the NAND with twltool, using the following command:

twltool boot2 --in nand.bin

Gives you files arm9.bin and arm7.bin, to be renamed to boot2_9.bin and boot2_7.bin respectively.


BEFORE STARTING

Make sure that your DSi-mode ARM7 BIOS is good. Open bios7i.bin in a hex editor. Ensure that the first 32 bytes are the following:

06 00 00 EA 06 00 00 EA 1F 00 00 EA 04 00 00 EA
03 00 00 EA FE FF FF EA 13 00 00 EA 00 00 00 EA

Make sure that your NAND dump has the 'nocash footer' at the end. Open it in a hex editor and check that it has the string 'DSi eMMC CID/CPU' at offset 0xF000000. This is the beginning of the 64-byte 'nocash footer', which holds the eMMC CID and console ID.

Ideally make sure that direct boot is disabled.


Using melonDSi

As with regular melonDS, System->Run will boot the firmware, which should land you into the DSi menu if you did everything right.

You will need to recalibrate the touchscreen through the system settings app to have it work correctly.

melonDSi can boot some DSi-mode apps, but they currently don't work that well.


Booting games

melonDSi is able to boot DS-mode games from the DSi menu. They currently run with the DS-mode BIOSes and backwards-compatible touchscreen mode, but at the 133MHz clock speed and with 16MB of RAM.

DSi-enhanced games do not boot yet.

Direct boot is likely to break things as it hasn't been updated yet.


Other issues

Do not attempt using savestates, they can and will go wrong.

melonDSi is based on melonDS 0.8.1, so it will have the bugs present in that version.



Enjoy! Or atleast, try to.


melonDSi, Windows 64-bit
melonDSi, Linux 64-bit

If you want to help us take melonDSi further: here's our Patreon]]>
http://melonds.kuribo64.net/comments.php?id=103
2D accuracy: it's a rabbit hole too -- by Arisotura http://melonds.kuribo64.net/comments.php?id=102 Sat, 14 Sep 2019 19:01:18 +0000
Anyway, mosaic is a typical feature of old consoles with 2D engines, including the DS. It basically applies a pixelation effect to graphics, as shown here:



The basic idea is that the screen is split in a grid, whose dimensions are variable (configured by register 0x0400x04C on the DS). For each 'cell' in the grid, all the pixels are colored the same as the first (top-left) pixel. In reality, it's a bit more complex, as on the DS the effect can be applied per-layer and per-sprite, and you can even specify different grid sizes for BG layers and sprites, but fundamentally it's more or less the same thing, it pixelates shit.

Sounds simple enough, right?

It's a bit tricky to implement when you're trying to write a performant renderer, though. Which is more or less why blargSNES never supported it.

As far as melonDS is concerned, mosaic was implemented in version 0.5, but (among other silly bugs) it never worked quite right as far as sprites were concerned, especially when those use rotation/scaling. But, at the time, I didn't do much past the original implementation, mostly because I don't know of a lot of games that use the mosaic effect. The lack of test cases meant it stayed supbar.

Until, well, now.

First thing to do is to write some test cases for sprite mosaic. BG mosaic seems quite simple, even though I would still have to probe it extensively for edge cases, but sprite mosaic is a bit more oddball, as seen here:






Those snapshots were taken from Grey, my capture-card DS. This isn't some weird buggy emulator, sprite mosaic really is that weird.

The sprite being mosaic'd is a simple ball sprite. The cyan sprite behind doesn't have mosaic applied to it, but both share the same size and rotscale parameters. The sprites on the left have double-size mode enabled, which merely doubles the bounding box of rotscaled sprites.

I have yet to work out all the oddities, but, from what I have observed, it seems that sprite mosaic doesn't work as we thought. If vertical mosaic is simply done by adjusting the source Y coordinate when rendering BG layers or sprites, horizontal mosaic is a different topic. As far as sprites are concerned:

* horizontal mosaic is restrained to the sprite's bounding box (vertical mosaic is too, but only in one direction)
* seeing how it is affected by the presence of neighbor sprites, horizontal mosaic seems to be applied after all the sprites are rendered
* it seems to keep track of which pixels belong to each sprite, even when said pixels are transparent


As of now, none of the existing DS emulators get sprite mosaic right. Probably, none of the GBA emulators get it right either, since the GBA uses a similar renderer, and it doesn't look like anybody has ever figured out the mystery of sprite mosaic.


I wanted to be able to check my mosaic implementation against hardware for correctness in many situations, to make sure I'd gotten the logic right. But constantly dumping frames from Grey and hand-comparing them against melonDS would be tedious. I wanted to have a better tool.

So, I set to work. I opened ds_capture.exe in IDA, opened up the Linux DS-capture code example and WinUSB documentation, and set to work. And, in one evening, MelonCap was born.



This is pretty simple. Left screen is the output from melonDS, middle is what is captured from Grey, right is a color-coded visualization of the differences.

The precision isn't exactly optimal though: melonDS outputs RGB666 color, like the DS does, but the capture card degrades it to RGB565 due to technical constraints. Well, this is atleast better than RGB555.

This is pretty much made for graphical testing in specific use cases, as there's no synchronization mechanism, one has to ensure that both melonDS and the DS are rendering the same things.

Anyway, the example in the screenshot above shows something interesting. There's been all that effort put towards pixel-perfect 3D, but our 2D engine isn't pixel-perfect (neither is the 3D one, but... yeah). We figured pixel perfection with the 2D engine would be a given, and, yet... here we are.

The bottom screen has BLDCNT configured so that all layers/sprites will be dimmed, with BLDY=1. This is hardly noticeable, and is likely an oversight. However, the colors generated by melonDS in this situation don't quite match hardware output.

The function for the brightness-down effect is described by GBAtek as:

out = in - ((in * EVY) / 16);

However, a quick hardware test tends to show that the actual function is:

out = in - (((in * EVY) + 7) / 16);

There's probably more of this shit everywhere, so the 2D engine will have to be heavily tested for accuracy. Fun shit ahead.]]>
http://melonds.kuribo64.net/comments.php?id=102
melonDS 0.8.3 -- by Arisotura http://melonds.kuribo64.net/comments.php?id=101 Wed, 04 Sep 2019 15:20:40 +0000

So what does this release bring? Well, we have been trying to address the issues present in previous 0.8.x releases (or sometimes even older releases, heh).


For example, I fixed the bug that was introduced with the new support for Ctrl+K type hotkeys. Basically, using Shift/Ctrl/etc as regular keys mapped to buttons was no longer possible. So, support for key mappings with modifiers was restricted to hotkeys. Meaning that using right Shift as R (as done by the default key mapping) should no longer cause input problems.


I have been trying to fix the issues we had with the framerate limiter and audio output, too, with moderate success. As I haven't been able to come up with a one-size-fits-all fix, there are now three different sync modes you can use, individually or together:

* Limit framerate: the oldschool framerate limiter. Although this is a revamped version that tries to average over several frames, reducing the likelihood of limiting too aggressively on certain games that internally run at 30FPS and are otherwise able to run fullspeed.

* Audio sync: synchronizes emulation to the audio output system. Seems to result in a bit more fluctuation in the framerate, but should prevent any audio stuttering.

* VSync (in the video settings dialog): synchronizes video output to your monitor's refresh rate. This only works with OpenGL, and currently only works under Windows (OpenGL support under Linux still needs more love). Also, DS games/programs may alter their framerate by messing with VCount, which VSync would be ill-equipped to deal with (unlike the two other sync methods).

I think most of the audio issues came from not properly syncing, which resulted in semi-regular overflows or underflows in the SPU FIFO, causing stuttering. The current audio output system cannot be precise enough to prevent those, as it works with small audio frames.

Speaking of which, I have also been revising it to use a more standard output frequency, in case some bad audio driver doesn't appreciate the previous frequency of 47340Hz. Now, it will attempt to run at 48000Hz, but it also allows SDL to specify another frequency if needed.


There are also a few other fixes that were long due (like OpenGL initialization failing under OpenGL <4.2), and some accuracy improvements, as usual.


As promised, beta builds of the JIT and DSi branches are coming soon, so stay tuned! Those will be based off older melonDS versions (0.8.2 and 0.8.1 respectively), though.


Enjoy!


melonDS 0.8.3, Windows 64-bit
melonDS 0.8.3, Linux 64-bit

If you're feeling generous: here's our Patreon]]>
http://melonds.kuribo64.net/comments.php?id=101
We're alive -- by Arisotura http://melonds.kuribo64.net/comments.php?id=100 Fri, 23 Aug 2019 13:21:03 +0000

Also, on the DSi front:

I'm also unsure what to do. there are BIOS dumps out there that have been filled as much as possible, by dumping the lower half and adding back the known parts of the upper half that can be sourced from elsewhere, as seen there: http://problemkaputt.de/gbatek.htm#biosdumping . NO$GBA expects these particular dumps. I might want to adopt that method too, but I'm afraid such dumps are less complete than dsikeys.bin/initmem7.bin/initmem9.bin.

Opinions on this are welcome. I'll need to check how much of the AES keys is sourced from the BIOS, vs how much is needed to run shit like the DSi firmware.

For reference: dsikeys.bin contains all the AES keys, as extracted from the AES engine after an unlaunch boot. initmem7.bin and initmem9.bin are chunks of ARM7 WRAM and ARM9 ITCM that contain keys (RSA, Blowfish) and other shit needed for initialization, those are mostly (but not entirely) sourced from the BIOS.


On other fronts, all sorts of exciting things are being developed for melonDS! Those who follow repo activity may have noted that DLDI support is being worked on. Stay tuned!


Edit- comments about how "I don't give a shit about DSi emulation!! Work on the features I want!!" or "nobody cares about DSi" or whatever, are getting on my nerves. There are very good reasons to work on DSi emulation, so I swear that if this shit continues I'll be deleting comments and handing out bans.]]>
http://melonds.kuribo64.net/comments.php?id=100
Progress on the DSi front -- by Arisotura http://melonds.kuribo64.net/comments.php?id=99 Sun, 04 Aug 2019 00:09:35 +0000
So instead, I did what I typically do in these situations: procrastinate and work on something else.

In this case, the DSi front.

If you remember, this was at a point where it needed wifi initialization to be able to launch titles. I was stuck at a particular point of the init procedure, where it waited for something and I had trouble figuring out what. But, regardless, I set to work.

In the meantime, googling some of the strings present in the firmware's wifi code brought me to this Atheros driver codebase which is very close to what the firmware uses. This helped me a lot in figuring out how things work and what the firmware was expecting.

And, well...

melonDSi config melonDSi flipnote

It's possible to boot shit from the firmware now!

Couple issues with this though.

* the system settings app boots, but it's impossible to get further due to the lack of touchscreen support
* Flipnote boots but freezes when starting a new flipnote
* pure DS-mode games seem to be bootable, DSi-enhanced games are not

Also, some DS-mode games suffer from audio issues for whatever reason. Well, it's worth noting that atm melonDSi does not support dynamic ARM9 clock adjustment, so DS games are basically running in forced DSi mode.

Well, we're getting there, I guess.]]>
http://melonds.kuribo64.net/comments.php?id=99
The JIT -- by Arisotura http://melonds.kuribo64.net/comments.php?id=98 Sun, 21 Jul 2019 15:11:00 +0000
I meant to fix a few of the issues we had lurking around, so we could release 0.8.3. I managed to fix some issues that caused random crashes on exit, then began trying to fix the framerate limiter. Thing is, it is directly related to the audio output code and the issues that has. So, yeah, I still haven't found a solution to that tricky problem.

I also need to struggle to keep some motivation at times. Looking at the patterns in how I work, it is likely that I have ADHD. I have yet to get diagnosed and all, but this wouldn't surprise me.

Anyway!

As you may have noticed, Github user RSDuck sent a pull request for the JIT recompiler. This is a bit of a tricky situation though. I don't want to have to deal with a situation where people make third-party JIT builds, but at the same time, the JIT isn't quite ready for general use, as we're still ironing out various issues with it.

However, the JIT is said to give a performance boost ranging from 30% to 100%, which is fairly nice, especially as there's room for improvement. RSDuck also said that porting the JIT to other architectures, like ARM, should be easy, so we might finally have Switch builds that run at acceptable speeds.

Or, also, Android builds. But, dunno. On one hand, this is a market I want to address, but on the other hand, history from Dolphin has shown that the Android userbase isn't exactly the nicest to deal with, so... yeah. And don't get me started on Android itself and the fun that coding for it is.


But first, let's focus on what we have in front of ourselves.


I'm going to try as hard as I can to fix the audio issues for 0.8.3 (and also add a vsync setting).

We will also be releasing beta builds of the JIT and the DSi branch, for those of you who want to mess around with things.]]>
http://melonds.kuribo64.net/comments.php?id=98