< Tragic newsBuffing up DSi mode >
Audio interpolation
Jul 22nd 2021, by Arisotura
Apologies for the slow Summer! We don't have air conditioners in the melonDS HQ. The current climate is causing the team to slowly melt.

Anyway, audio interpolation is one of the emulation improvements that have been requested for melonDS. My general policy for emulation improvements is that they should allow for keeping the accurate code paths, and they shouldn't add too much complexity to the code. Audio interpolation is well within these bounds. Actually, I had implemented it in DeSmuME back then, and due to the way DeSmuME's mixer works, it was quickly done.

So I figured I would give it a try in melonDS.

The basic idea behind audio interpolation is to smooth out the audio samples as they're being upsampled. DS games may have downsampled audio to save on space and bandwidth, and the DS mixer doesn't perform any interpolation, which can lead to rough sounding samples. The reason the DS does no interpolation is most likely due to how its mixer hardware works, but obviously as an emulator we can ignore these constraints and do a better job.

It's also noting that, as far as melonDS is concerned, there are two parts we need to take care of: the DS mixer and the audio output.

In the DS, the mixer is driven by the system clock, like nearly everything else. If you ever coded for the DS, you might have wondered why the frequency registers for the audio channels are weird:

40004x8h - NDS7 - SOUNDxTMR - Sound Channel X Timer Register (W)

 Bit0-15  Timer Value, Sample frequency, timerval=-(33513982Hz/2)/freq

The PSG Duty Cycles are composed of eight "samples", and so, the frequency for Rectangular Wave is 1/8th of the selected sample frequency.
For PSG Noise, the noise frequency is equal to the sample frequency.

The SOUNDxTMR registers directly control the channel timers, which are driven at half the system clock. These work like the general purpose timers: they are incremented at half the system clock, and every time they overflow, they are reloaded to the SOUNDxTMR value and the channel advances to the next sample.

This is a fairly simple and efficient design, but you can probably guess why it doesn't lend itself to interpolation. Basically, to get the sub-sample position you need for interpolation at any given time, you would need to subtract the current timer value from the reload value, then divide that by 0x10000 minus the reload value, which isn't convenient to implement in hardware.

The mixer in melonDS works in a similar way, although it is only sample-accurate, for several reasons: sample accuracy is good enough for DS games, we don't know how the mixer operates on a per-cycle basis, and of course, performance reasons. To reach its sample rate of approximately 32.7 KHz, the DS needs to output one audio sample every 1024 system-clock cycles, and that is how often we run the mixer in melonDS. We have to be a bit smart about updating our channel timers, but it works well enough.

However, this design means the output sample rate of the melonDS core depends on how fast it's running. Basically, melonDS runs 560190 cycles per frame and outputs one audio sample every 1024 cycles, like the real thing. Assuming a framerate of 60 FPS (which is a bit faster than the real thing), this means an audio output rate of 32823.6328125 Hz.

Well, yeah. Generally, you can't go and ask your audio library for a weird non-integer sample rate.

So what do we do, here? Well, early melonDS versions would just pick the closest integer sample rate, send out the audio output as-is, and pray. You guess, it didn't work that well. Not only was it impossible to attain perfect sync, but on some platforms we just could not get a sample rate of 32824 Hz.

Hence, a proper audio output stage was added. It lets us pick a more standard output rate of 48 KHz, lets the audio driver give us another sample rate if that one isn't available, then it resamples melonDS's audio output to match that output rate. The resampler also supports a small margin, which can make up for small variations in framerate.

This resampler would be another point of concern: currently, it upsamples audio with no interpolation, so there's room for improvement here too.

Anyway, I made a quick proof-of-concept in a separate branch. For now, it applies linear interpolation to all channels, and seems to work decently well. A few notes on this:

1. PSG channels are quite muffled. They should not be interpolated, but I'm partly tempted to keep that as a fun option.

2. Linear interpolation is the easiest but certainly not the best. I could implement better algorithms: cosine, cubic, gauss...

3. Of course, the feature would be made optional, and disabled by default.

I might also add an option for interpolation in the resampler, or keep the two tied together for simplicity? Not sure. Noting that interpolation makes things sound smoother but can also muffle sound to an extent. Your input is welcome!
35 comments have been posted.
< Tragic newsBuffing up DSi mode >
Generic aka RSDuck says:
Jul 27th 2021
melonDS currently doesn't do anything fancy like that, if emulation runs too slow, there will be short gaps in the audio, which if large enough is audible as crackles.

Also the time a frame rate on a DS (which is slightly less than 60 Hz) and most monitors (which won't be exactly 60 Hz either, though a lot closer) don't match up 100%.

An ideal solution to this would be to implement something like this: https://raw.githubusercontent.com/libretro/docs/master/archive/ratecontrol.pdf

Though for this too work we would need working vsync, which doesn't work with current messy way we do our OpenGL business (shared contexts across two threads, …).
Rin Tohsaka says:
Jul 28th 2021
I find it interesting that you consider the down-pitching fancy since, from an LPCM waveform perspective, it's actually the simplest.

For example, if the original waveform is 48kHz at full speed, then at half speed it'd simply play back the exact same waveform without any modification, albeit at 24kHz. 75% speed would be 36kHz, 80% would be 38.4kHz, etc etc.

As a non-emulation example, this is effectively what the 'Sample Rate' setting controls in the program "Header Investigator" with regards to WAV files; no alteration is actually made to the waveform itself and it merely changes playback rate: http://www.railjonrogut.com/HeaderInvestigator.htm
Generic aka RSDuck says:
Jul 28th 2021
> I find it interesting that you consider the down-pitching fancy since, from an LPCM waveform perspective, it's actually the simplest.

well from a programming perspective the easiest way to handle something is to not handle it at all. To have any kind of stretching would require taking the frame time into account when upsampling the audio data, while the way it's currently done can just go with constants everywhere.

DS emulation is not that expensive, it's safe to assume most users rarely experience slowdowns.
Rin Tohsaka says:
Jul 28th 2021
> To have any kind of stretching would require taking the frame time into account

So are you saying that, unlike something like Dolphin, the audio subsystem of melonDS is not in direct synchronized lock-step to the rest of the emulator, including its rendering pipeline and the like?

Because from my perspective, what I describe is no different from if the program is spitting out 40fps instead - it doesn't drop frames and still renders the same amount of frames as if it were at 60fps, but those frames are simply delivered at a slower rate. So instead of the frames being sent every 16.6ms, they're sent every 25ms.

The same would be for audio, where the same source samples are simply being output at a slower rate. It's just that, unlike video, audio will naturally change in pitch when you change the speed at which those samples are delivered, which is what I meant by "down-pitching".

Now yes I agree that dropping in performance for a DS emulator may be uncommon, but I was using that as a more reasonable example since my true interest is being able to run the emulator at a locked frame rate that is less than the DS's native 59.??fps.
Rin Tohsaka says:
Jul 28th 2021
Err, that is, a locked frame rate that is less than the DS's native 59.??fps without any loss in audio fidelity, and AFAIK the only way to do this is to do what I described.

(yes I realize that the pitch change can be noticeable if being locked to a much lower frame rate, but that's arguably a much better alternative than the audio hiccups from audio samples being skipped)
Generic aka RSDuck says:
Jul 30th 2021
> So are you saying that, unlike something like Dolphin, the audio subsystem of melonDS is not in direct synchronized lock-step to the rest of the emulator, including its rendering pipeline and the like?

the entire audio hw which produces a sample every 1024 cycles is properly emulated in sync with the rest of the emulation. The samples generated from one video frame are collected and at the end of the frame appened to a bigger FIFO. Inside the SDL audio callback which is called whenever it runs out of samples to play the FIFO is read, the read samples are upsampled from ~32khz to 48khz and then passed on.
Dash2k5 says:
Aug 8th 2021
When will Melon DS Get a Rich Presence for Discord? :(
Arisotura says:
Aug 8th 2021
what's the point of that
Rayyan says:
Aug 8th 2021
it's a useless, purely cosmetic feature that is not important.
plus it will lead to us pulling in more dependencies.
poudink says:
Aug 8th 2021
Discord gamesdk is bloat
Lenny says:
Aug 9th 2021
MeleonDS is by far the best DS emulator I have found! Thanks all the dev's for that!
However, do to slow hardware I have troubles running DS emulation at full speed without frame skip for a lot of games.

So, in short, I am requesting frame skip for MeleonDS. I understand it is a lot of work to implement any feature.
I do think it will help users on Android and on the Raspberry Pi run MeleonDS rather than (the horror!) Desmume.
Really, it is my only issue with this emulator.
Rayyan says:
Aug 9th 2021
Make sure you have the JIT recompiler on. melonDS does have a fast-forward hotkey but I'm not sure about the implementation.
Lenny says:
Aug 11th 2021
I tested the fast-forward hotkey and it does not have frame skip.
Yes the JIT is on. Testing on Raspberry PI4 8GB, 64bit OS. Official latest (0.9.2) ARM64 build.

Mario Kart DS - full speed in non 3D scenes, sometimes drops as low as 30 FPS with 3D scenes.
Super Mario 64 DS -Game intro with Mario and stars 45FPS, in game 45-60 FPS

All tested using software renderer. For some reason the OpenGL renderer will not enable, even with overriding the GL version.
Lenny says:
Aug 11th 2021
Fixed the OpenGL renderer -had to manually edit the config file. It is actually a little slower but otherwise works fine.
Rayyan says:
Aug 11th 2021
Lenny: that is emulated OpenGL, meaning it runs on CPU. You probably have hw acceleration off.
Pages:123
Post a comment
Name:
DO NOT TOUCH