State of the library in comparison to OpenAL Soft?

@mackron

Hi @mackron, congrats for such an amazing lib, I've been looking for a library like this for long time now.

I'm thinking about replacing current raylib audio module backend (OpenAL Soft) by mini_al and I would like to know how mini_al compares to OpenAL Soft in features... Actually, I'm mostly just using audio playback for complete loaded sounds and music streaming.

My final goal would be to remove any external library dependency for raylib, I mean, include with raylib basecode all required dependencies.

Thanks!

The first thing to keep in mind is that mini_al is not a 3D audio API, so if that's something you need then it won't be suitable (or you'll need to do your own mixing). Besides that, there's still a bit of work to do before it's on par with OpenAL-Soft:

I still need to get a CoreAudio back end done for macOS and iOS. I don't own a Mac, though, and they're kind of expensive where I live so I don't know when I'll get around to that
- There's an OpenAL back end in place, but it's never been tested on Mac
PulseAudio and JACK back ends are not currently supported, but they're in the pipeline
mini_al has a feature where it does automatic format conversion between the format requested by the application and the format supported by the internal back end device. This conversion pipeline needs a bit of work in terms of both optimizations and features:
- The sample rate conversion algorithm is just a dumb, low quality linear algorithm
- Channel conversion (stereo to 5.1, for example) is very basic at the moment - it simply drops excess channels or inserts silence into extra channels. I want to add support for something a bit smarter where it properly blends/averages channels instead.

I took a quick look at your code. You'd need to do a bit of work to implement your own mixer. mini_al does not do any sound/source/buffer management like OpenAL, so all of the logic for mixing the audio data of currently playing sounds and music will need to be done manually. Is volume and pitch the only effects you apply to sounds and music? From what I can tell, it looks like you've actually got fairly simple audio requirements so your own mixer wouldn't be too difficult. If you like, I can probably get something quick and dirty up and running for you as a proof of concept? If I can get that working, and then have somebody test/fix the OpenAL back end on Mac, it may actually work...

I also noticed you're using my dr_flac library. Nice :) 👍

@mackron

Hi @mackron! Thank you very much for the quick answer! :D

I still need to get a CoreAudio back end done for macOS and iOS...

That's not a big issue, I can keep my OpenAL Soft backend on those platforms... I neither have a Mac to test it, all raylib testing on macOS has been done by users.

PulseAudio and JACK back ends are not currently supported, but they're in the pipeline

No problem.

mini_al has a feature where it does automatic format conversion...

That's great, actually, I also tried to implement some functions to format wave data... but very bad quality...

You'd need to do a bit of work to implement your own mixer. mini_al does not do any sound/source/buffer management like OpenAL, so all of the logic for mixing the audio data of currently playing sounds and music will need to be done manually.

Yeah, that's what I saw... probably that's the difficult part... dealing with sounds/source/buffers was quite easy...

Is volume and pitch the only effects you apply to sounds and music?

Yes, actually I exposed that for raylib users because it came with OpenAL Soft...

If you like, I can probably get something quick and dirty up and running for you as a proof of concept?

Wow! That would be great! :D

I also noticed you're using my dr_flac library.

Yeah, it's great! Been following your libs for some time now.

raylib is a simple and easy-to-use library primary intended for education, prototyping and small games so, no need for advanced audio features, there are bigger libraries out there intended for bigger projects. :)

I've done some work on this and it seems to work fine so far. You can follow my progress here: https://github.com/mackron/raylib/tree/dr/mini_al (the dr/mini_al branch).

I've got Sound, AudioStream and Music working, but I've not tested it thoroughly. It's very quick and dirty and probably has a few bugs, but it shows that mini_al can work. I can polish it up a little bit if you decide mini_al integration might be worth pursuing.

A few things:

I haven't got the pitch effect working yet, mainly because I just don't know how to do it off the top of my head...
The output volume seems to be slightly louder with mini_al... Not fully sure why, but suspecting it's got something to do with different algorithms being used between mini_al and OpenAL for format conversion.
The implementation of mini_al needs to be put into a separate translation unit to audio.c because it causes conflicts with Win32 APIs like CloseWindow().
I've left the OpenAL implementation in place. Toggle it with the USE_MINI_AL define at the top of audio.c.
mini_al does mixing asynchronously on a separate thread so you'll need to keep thread-safety in mind if you use mini_al. I've got some basic stuff in there that should work, but it's hardly been tested. Just keep that in mind...
- The synchronization technique I use makes it not "real-time", but I don't think that matters for this project (and I'd be surprised if OpenAL-Soft is real-time anyway, but I could be wrong).
From what I can tell it looks like mini_al uses less memory.
I've not tested this, but I think mini_al will be much more CPU efficient than OpenAL-Soft because it's software mixing is so much simpler.
With a relatively small amount of work you should pretty easily be able to support any format/channels/rate combinations for both playback output and wav/flac/ogg files. Anything that's supported by mini_al should work fine.

I also have some suggestions for improving your API design in a few places, but I think I'll raise a separate issue about that one...

Have fun playing around with this. Let me know if you have any issues!

@mackron

Hi @mackron! Amazing work! Thank you very much for your time and your effort on this feature!

About your comments:

I can polish it up a little bit if you decide mini_al integration might be worth pursuing.

Undoubtly it worths it! Replacing such a big dependency as is OpenAL Soft by a more lightweight, portable and faster alternative is a goal!

I haven't got the pitch effect working yet, mainly because I just don't know how to do it off the top of my head...

No worries, it's not crucial... I think it's not that easy, it involves some Fourier Transformation... Here I found a good reference and some code.

The output volume seems to be slightly louder with mini_al... Not fully sure why, but suspecting it's got something to do with different algorithms being used between mini_al and OpenAL for format conversion.

No big issue, it can be adjusted to a lower base value.

The implementation of mini_al needs to be put into a separate translation unit to audio.c because it causes conflicts with Win32 APIs like CloseWindow().

Yeah, I know... the infamous windows.h, I've dealt with that in the past just defining the myself the required parts of windows.h but it seems in this case it requires lots of typedefs... well, no problem.

I've left the OpenAL implementation in place. Toggle it with the USE_MINI_AL define at the top of audio.c.

That's great, allowing the user to choose between both as a transition step is the best idea!

mini_al does mixing asynchronously on a separate thread so you'll need to keep thread-safety in mind if you use mini_al. I've got some basic stuff in there that should work, but it's hardly been tested. Just keep that in mind...

Ok.

The synchronization technique I use makes it not "real-time", but I don't think that matters for this project (and I'd be surprised if OpenAL-Soft is real-time anyway, but I could be wrong).

Ok.

From what I can tell it looks like mini_al uses less memory.

That's great!

I've not tested this, but I think mini_al will be much more CPU efficient than OpenAL-Soft because it's software mixing is so much simpler.

That's really great! :D

With a relatively small amount of work you should pretty easily be able to support any format/channels/rate combinations for both playback output and wav/flac/ogg files. Anything that's supported by mini_al should work fine.

Amazing!

I also have some suggestions for improving your API design in a few places, but I think I'll raise a separate issue about that one...

Don't hessitate to open an issue or just send the improvements with a pull request.

Have fun playing around with this. Let me know if you have any issues!

I tried compiling it with MinGW32 (GCC 5.3.0) and I got some errors:

In file included from external/mini_al.c:4:0:
external/mini_al.h:2720:3: error: conflicting types for 'WAVEFORMATEXTENSIBLE'
 } WAVEFORMATEXTENSIBLE;
   ^
In file included from external/mini_al.h:2683:0,
                 from external/mini_al.c:4:
c:\raylib\mingw\include\mmreg.h:48:3: note: previous declaration of 'WAVEFORMATEXTENSIBLE' was here
 } WAVEFORMATEXTENSIBLE,*PWAVEFORMATEXTENSIBLE;
   ^
In file included from external/mini_al.c:4:0:
external/mini_al.h:2826:25: fatal error: audioclient.h: No such file or directory

You can send me a pull request to my develop branch if you want, I will need some time to review all the changes and understand how everything works but after a quick look it seems quite clear and comprehensive.

Thanks for that MinGW compiler error - I'll get that fixed up. I'll do a bit of polish and more testing before I doing a pull request. Not sure when that'll be, but I won't forget :)

The complexity with pitch shifting is keeping the original sound the same length. I can easily change the pitch by simply running every frame through a sample rate converter, but that lengthens or shortens the sound, depending on the direction of the shift. I'll take a look at those references you provided.

My original intent for the USE_MINI_AL macro was mainly just for testing and safety in case mini_al doesn't work out too well. In the future when the mini_al version matures I would just remove the OpenAL version entirely just to keep it simpler.

I'll do a bit of polish and more testing before I doing a pull request. Not sure when that'll be, but I won't forget :)

Great! No hurries, take your time! :)

I've updated the dr/mini_al branch with a fix for the first compiler error, but regarding this error:

external/mini_al.h:2826:25: fatal error: audioclient.h: No such file or directory

That's one of the headers for the WASAPI backend. You need to update your Win32 API. Look at the "w32api" download on this page to get you started. In the short term, add "#define MAL_NO_WASAPI" before the implementation of mini_al (in mini_al.c). Let me know how that goes!

As a side note, I'm going to start using the "__has_include" statement with GCC (and any other compiler that supports it) to cleanly exclude backends when the compiler does not have the necessary development packages installed.

@mackron

Hi @mackron, thanks for the quick solution!

I'm updating my MinGW headers and libs, it seems some packages were missing.

I just pushed some changes to branch dr/mini_al which adds support for pitch shifting. The mini_al backend should now be at feature parity. Were you ever able to get it working on your end? Pull request coming soon.

Ok, managed to compile it and it works flawlessly. 😄

Just tried with mingw32 updated to latest available version (GCC 6.3.0); neither MinGW32 or TDM-GCC-32 toolchains include the required libraries for WASAPI and DIRECT SOUND backends, so, I had to comment those lines to force WinMM backend; also tried to define it in audio.c:

#define MAL_NO_WASAPI
#define MAL_NO_DSOUND
#include "external/mini_al.h"

but it didn't work, MAL_WIN32 forces support on mini_al.h, it seems some check is not working...

When compiling mini_al.c module with GCC 6.3.0 and -Wall got some warnings, nothing significant but just listing them here:

gcc -O2 -c external/mini_al.c -Wall -I.
Process started >>>
In file included from external/mini_al.c:4:0:
external/mini_al.h: In function 'mal_device_init':
external/mini_al.h:8738:10: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
     if (((mal_uint64)pDevice % sizeof(pDevice)) != 0) {
          ^
external/mini_al.h: In function 'mal_src_read_frames_ex':
external/mini_al.h:9357:23: warning: variable 'algorithm' set but not used [-Wunused-but-set-variable]
     mal_src_algorithm algorithm = pSRC->config.algorithm;
                       ^~~~~~~~~
external/mini_al.h: In function 'mal_pcm_f32_to_u8':
external/mini_al.h:10459:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
         s = ((*((int*)&x)) & 0x80000000) >> 31;
         ^
external/mini_al.h: In function 'mal_pcm_f32_to_s16':
external/mini_al.h:10475:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
         s = ((*((int*)&x)) & 0x80000000) >> 31;
         ^
external/mini_al.h: In function 'mal_pcm_f32_to_s24':
external/mini_al.h:10490:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
         s = ((*((int*)&x)) & 0x80000000) >> 31;
         ^
external/mini_al.h: In function 'mal_pcm_f32_to_s32':
external/mini_al.h:10505:9: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
         s = ((*((int*)&x)) & 0x80000000) >> 31;
         ^
In file included from external/mini_al.c:4:0:
At top level:
external/mini_al.h:2538:13: warning: 'MAL_GUID_KSDATAFORMAT_SUBTYPE_IEEE_FLOAT' defined but not used [-Wunused-variable]
 static GUID MAL_GUID_KSDATAFORMAT_SUBTYPE_IEEE_FLOAT = {0x00000003, 0x0000, 0x0010, {0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}};
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/mini_al.h:2537:13: warning: 'MAL_GUID_KSDATAFORMAT_SUBTYPE_PCM' defined but not used [-Wunused-variable]
 static GUID MAL_GUID_KSDATAFORMAT_SUBTYPE_PCM        = {0x00000001, 0x0000, 0x0010, {0x80, 0x00, 0x00, 0xaa, 0x00, 0x38, 0x9b, 0x71}};
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<<< Process finished. (Exit code 0)

Again, thank you very much for all the hard work! It's trully amazing!

I'll get those issues sorted out. Thanks.

Strange that they don't come with dsound.h and audioclient.h... I've never known a distribution of MinGW to not include headers for DirectSound...

I'm just looking at these compiler issues you're having, and indeed it looks like the old 32-bit versions of MinGW do not include dsound.h nor audioclient.h (why?!). The 64-bit version of TDM-GCC definitely includes them, though, because that's what I use for GCC on Windows. Also, you shouldn't comment out those MAL_SUPPORTS_DSOUND and MAL_SUPPORTS_WASAPI lines.

I also just wanted to clarify something. When you say #define-ing MAL_NO_DSOUND and MAL_NO_WASAPI didn't work - what exactly didn't work? What errors were you getting? I just tried on that same version of MinGW you're building against and it works fine (except for those other warnings you mentioned).

Sorry, my fault, double checked it... I was defining on audio.c:

#define MAL_NO_WASAPI
#define MAL_NO_DSOUND
#include "external/mini_al.h"

but not on mini_al.c module compilation. You're right, it works fine.

Yeah, I also checked MinGW-w64 and it comes with all required libraries and headers, actually, the different MinGW32 versions also include the libdsound.a library but not the header... weird.

Probably I should use MinGW-w64 32bit version of the package, more updated and complete than the old MinGW32 project...

I can confirm that years ago, my mingw setup did not include any DirectX header/library at all.
It changed at some point later, for good.

I've updated the dr/mini_al branch with fixes for those warnings.

I can confirm that years ago, my mingw setup did not include any DirectX header/library at all.

I had been using old MinGW (not MinGW-w64) for ages and never realized it didn't have dsound.h!

OK, I've pushed some potential fixes for RPI and SDL, but there's a few things I need you to do for me.

Raspberry Pi
Unfortunately my RPI has horrible interference with the analogue output to the point that I can barely hear anything which makes it very hard for me to tell if my fixes are working. I have added a little piece of code at line 404 in audio.c with some RPI specific device configuration settings:

Could you first try the settings I've already got in place?
Could you try toggling alsa.noMMap mode between true and false and see if that makes any difference?
Could you also try commenting out the bufferSizeInFrames = 2048 and see if that makes a difference? (Not explicitly setting this property makes it fall back to defaults.)
Could you try setting alsa.noMMap = MAL_FALSE and changing DEVICE_SAMPLE_RATE to 48000 (I had a bug recently with 44100 on ALSA backends and I'm wondering if it's related)

When I use a USB output device everything works perfectly fine for me on RPI. It's only the built-in analogue output that has issues for me (though, I can't test HDMI output because I don't have a device to test with).

Emscripten
I think the long delay is a result of an excessively large internal buffer which thus resulted in excessive latency. I've put a potential fix in place, but if that doesn't work, could you comment out line 8401 #define MAL_USE_SDL_1 in mini_al.h and pass -s USE_SDL=2 when compiling and see if that makes a difference?

Regarding the issue with what sounds like a buffer initialization glitch, assuming we're talking about the same thing, I think that issue is not so much a bug with the audio system, but rather the way in which the music stream is updated. It looks to me in my testing that the first call to EndFrame() takes a very long time which causes a delay for the next call to UpdateMusicStream() - long enough that the audio system is starved of new data. I think if you were to move UpdateMusicStream() to after the rendering section it would work fine, though I've not tested.

Could you also show your output again like you did in your previous message? I've update it with some more detailed information.

@mackron

Hi @mackron, thanks for the quick update!

Not sure if I could look at this during the weekend, I've got my RPIs at work... maybe I could try the Emscripten version... in any case, I'll let you know my results ASAP.

I'm testing everything right now, just a detail:

When I use a USB output device everything works perfectly fine for me on RPI. It's only the built-in analogue output that has issues for me (though, I can't test HDMI output because I don't have a device to test with).

I use HDMI output by default, I can also try analogue output...

More info in a while!

Ok, everything tested, here my results:

Raspberry Pi 3

First I tried all proposed solutions with no luck, also tried analogue jack output vs HDMI output, jack output has some interferences/noise, HDMI output was clear.

After some tests, found the solution to get it working:

deviceConfig.bufferSizeInFrames = 32768;

That was the minimum value to get it working in par with Windows desktop.

Output info:

INFO: Audio device initialized successfully: Default Playback Device
INFO: Audio backend: ALSA
INFO: Audio format: 32-bit IEEE Floating Point
INFO: Audio channels: 2
INFO: Audio sample rate: 44100
INFO: [AUD ID 0] Audio stream loaded successfully (44100 Hz, 16 bit, Mono)

HTML5 (emscripten)

Tried modified code and the delay is way shorter but you can hear some glitches/noise on music and sounds.

After that tried with line commented and -s USE=SDL=2 and it works perfectly, no delay and no glitches/noise.

Output info:

INFO: Audio device initialized successfully: Default Playback Device
INFO: Audio backend: SDL
INFO: Audio format: 32-bit IEEE Floating Point
INFO: Audio channels: 2
INFO: Audio sample rate: 44100
INFO: [AUD ID 0] Audio stream loaded successfully (44100 Hz, 16 bit, Mono)

The only problem I find is that linking against full SDL2 generates a ~9MB file instead of the ~900KB file generated when using OpenAL Soft backend...

Thanks for your feedback on this!

A buffer size of 32768 is huge - what is the latency like with that? In any case, I'm suspecting it's just because the RPI has crappy built-in audio which isn't suited for low-latency applications. I'm just not sure how I could detect the RPI at run-time and automatically tune it. I really don't want applications to require a compile time macro like "PLATFORM_RPI" - I want mini_al to Just Work. If you have any ideas, let me know!

A 9MB output for HTML5 is no good - I'll look into that. Does emcc have a "-s" option like GCC? Does that do anything? What is the output size with "-O2"?

I will look into the SDL issues and try getting OpenAL working with Emscripten, but I'll be out of action on this project for the next few days so I don't know when I'll get a chance to get to it.

@mackron

@mackron https://raspberrypi.stackexchange.com/questions/754/how-can-i-detect-that-im-compiling-for-raspberry-pi ?

ifdef __arm__ && stat(/opt/vc/include/bcm_host.h) >= 0 :)

@mackron

A buffer size of 32768 is huge - what is the latency like with that?

Yes, it's quite big... but actually latency is not that bad, when playing a sound there is a delay between key pressed and playing of around 0.5 seconds.

If you have any ideas, let me know!

Actually not many ideas but I remember a similar issue with OpenAL related to AUDIO_BUFFER_SIZE, not being big enough, just checked code usage of that parameter:

    // The size of a streaming buffer must be at least double the size of a period.
    unsigned int periodSize = device.bufferSizeInFrames / device.periods;
    unsigned int subBufferSize = AUDIO_BUFFER_SIZE;
    if (subBufferSize < periodSize) {
        subBufferSize = periodSize;
    }

I saw device.periods defaults to 2, so checking buffer for refill twice every buffer size... maybe issue is related with that, maybe check timming is not accurate... I got a similar issue on windows with minimum time resolution... not sure, just an idea...

A 9MB output for HTML5 is no good - I'll look into that. Does emcc have a "-s" option like GCC? Does that do anything? What is the output size with "-O2"?

There is no -s for strip on emcc, using -O2 reduces generated javascript filesize to ~4.5MB.

...I'll be out of action on this project for the next few days so I don't know when I'll get a chance to get to it.

No worries @mackron, I'm also quite busy lately on my current job...

EDIT: @r-lyeh, I think @mackron means not requiring any platform dependent check to get it working correctly, just get a generic solution.

I can't reproduce the crackling issue on my machine for SDL. Could you perhaps try playing around with that deviceConfig.bufferSizeInFrames trick and see if that helps at all? Just wondering if perhaps the buffer size is too small, thus resulting in glitches.

I've pushed a potential fix for RPI, but I was hoping you could test something out for me. I have some logic in mini_al where it scales the default buffer size depending on known audio devices (currently only the RPI audio devices are used). I've currently got this set the 32, but I'm not sure if that's over the top. Would you be able to play around with those numbers to see what values work for you? The lower the better... Said code looks like the following, at about line 5484.

struct
{
    const char* name;
    float scale;
} g_malDefaultBufferSizeScalesALSA[] = {
    {"bcm2835 IEC958/HDMI", 32},
    {"bcm2835 ALSA",        32}
};

"bcm2835 ALSA" is the important one.

@mackron

Hi @mackron, just tested new code. Tried with values: 32, 16, 8.

Using HDMI audio output. With 32 and 16 it sounds very similar to me, probably better with 16... despite some small glitch (maybe some interference), both sound good. With 8 value it seems to be more glitchy, like audio saturation... In all cases, when playing sounds, seems to be a small delay at the beggining (comparing to Windows playing) but is less than half second...

Sorry for not being able to provide more detailed information.

On loading, output info is similar in all the modes (just changes the Audio buffer size data):

INFO: Audio device initialized successfully: Default Playback Device
INFO: Audio backend: mini_al / ALSA
INFO: Audio format: 32-bit IEEE Floating Point -> 32-bit IEEE Floating Point
INFO: Audio channels: 2 -> 2
INFO: Audio sample rate: 44100 -> 44100
INFO: Audio buffer size: 17600

I know it's a bit difficult to track those small issues working remotely... but now it sounds pretty good to me, maybe we can integrate those changes and keep working from there?

Pull request submitted. Probably best to move this discussion to over there.

@mackron

Hi @mackron, do you think we can close this issue for now?

Yep, closing.

Navigation Menu

Repository navigation

State of the library in comparison to OpenAL Soft? #3

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

State of the library in comparison to OpenAL Soft? #3

Description

Activity

mackron commented on Nov 11, 2017

raysan5 commented on Nov 11, 2017

mackron commented on Nov 12, 2017

raysan5 commented on Nov 13, 2017

mackron commented on Nov 13, 2017

raysan5 commented on Nov 13, 2017

mackron commented on Nov 13, 2017

raysan5 commented on Nov 14, 2017

mackron commented on Nov 15, 2017

raysan5 commented on Nov 15, 2017

mackron commented on Nov 15, 2017

mackron commented on Nov 16, 2017

raysan5 commented on Nov 16, 2017

r-lyeh-archived commented on Nov 16, 2017

mackron commented on Nov 16, 2017

18 remaining items

mackron commented on Nov 24, 2017

raysan5 commented on Nov 24, 2017

raysan5 commented on Nov 27, 2017

raysan5 commented on Nov 27, 2017

mackron commented on Nov 28, 2017

r-lyeh-archived commented on Nov 28, 2017

raysan5 commented on Nov 28, 2017

mackron commented on Dec 2, 2017

mackron commented on Dec 3, 2017

raysan5 commented on Dec 4, 2017

mackron commented on Dec 5, 2017

raysan5 commented on Dec 5, 2017

mackron commented on Dec 5, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions