Example for using miniaudio in Unity 3D on Android

[ludzeller]

Hi,

I am thinking about using miniaudio to capture audio from the microphone of Meta Quest headsets from a native library for a Unity project.

Is there a similar example somewhere, maybe for Android in general?

How are the permissions to be handled?

Is low latency possible in that context?

[mackron]

I have no experience at all with using miniaudio with Unity so I won't be able to help you there.

I've not tested miniaudio with any kind of headset so not sure how that would work. What audio backends do they typically use? If they coincidentally use any of the backends that miniaudio already supports I'd be happy to make some tweaks to make it work cleanly if it doesn't already. I would need to help and advice from the community for testing though because I don't have the hardware to test with.

The general process for capturing should be the same everywhere. You can check the simple_capture example to get an idea on what's going on. In terms of latency, it's all up to the backend, but miniaudio supports the ability to give the backend a hint as to what the latency should be (periodSizeInMilliseconds/Frames in the device config).

Out of curiosity, is there any reason in particular you don't want to use Unity directly for this? I would think that would already support capturing and would also have explicit support for headsets.

Sorry I can't be of more help.

(Moving this to the discussion section.)

[mackron]

Further to this, I'm assuming most likely the speakers and microphones of the headsets just show up as regular audio devices on the host operating system. In this case it should "Just Work" like anything else.

And about permissions, you'd have to experiment with that. I don't have a lot of experience with capture mode on mobile platforms.

[ludzeller]

Thanks for your reply!

It's unclear what audio backends Unity is using exactly as they are cascading them away. They offer the possibility to complete use different audio engines such as FMOD and Wwise, but I wouldn't want that for my project OpenSoundLab.

So basically I would have to try the different backends you offer on see what runs on Android / Quest headsets, right?

The reason I am looking for an alternative is that Unity's Microphone object has too high latency for real-time audio processing, especially when talking. It's a problem to many.

[mackron]

I think the way it should work is that the headset's microphone should just show up as a normal microphone on the operating system. Then it should just work like any other capture device. The problem, however, is that I don't have first-hand experience with capture mode on Android. I also don't know how you would use it alongside Unity. Does Unity have some kind of mechanism to load compiled C code and make it available in-engine somehow?

[ludzeller]

Yes, you can load dynamic native libraries and call them from C#.

The idea would be to share a buffer between my Unity audio code and your library, in order to copy your audio input buffers to my code.

Unfortunately that would add at least one buffer of latency...

[mackron]

OK, if you can load native code and run it from Unity it could be possible. You'll just need to make sure the native binary includes the Android build. That means you'll have to compile it from an Android toolchain (the way miniaudio works is that when you compile it, it only includes the backends relevant to the platform you're compiling from). Once you have that, you're on your own regarding the creation of bindings back to C# - I don't offer any kind of C# or .Net bindings. You might benefit from implementing your own simplified API that you export to make your life easier.

I'm not sure how that'll work for when you distribute your game on different platforms. You might need to have different native miniaudio binaries for each platform.