Spatial audio in VR

Hey folks,

For those who are creating VR projects in OF, what are you using to spatialize audio? There are a handful of ambisonic libraries out there in addOn land but none seem to perform head related transfer functions.

Oculus has their own library that does ambisonics and creates a room box for reverberation. The c++ library is available to developers upon request.

Thanks for any pro tips!

2 Likes

hello,

I’ve been using the Facebook 360 tools to create a native Android application that does 360 video playback + spatial audio. The toolset offers a full production pipeline and custom format (*.tbe) for mixing spatial to which can also include a head locked track. They have an SDK available which will allow you to include this in openFrameworks or any other C++ application. You can’t fiddle with the HRTF’s though, something you can with Oculus (or at least I thought so). Furthermore, this technology is actively developed and has a growing community.

https://facebook360.fb.com/spatial-workstation/

FMOD is actually pretty good. There are some old examples of 3D sound here.

thanks for your insights @fresla and @Vjacobs!

i’ve been looking at the Oculus library (avrAudio) today and on first impression it seems fairly straightforward to use.

@fresla, the fmod example is interesting, but doesn’t take head orientation into account. i might still end up using some of that logic for playing audio files and passing the wave data to avrAudio.

i should compare with the FB360 library to see what the primary differences are.

@mantissa

You just need to change the listener settings, you may need to expose more from FMOD than this class does. I used it (a long time ago) and it was giving good results for listener to sound source orientation. It is pretty comprehensive, but this implementation may not do everything you need out of the box.

In the setup you have:

listenerUp.x = 0.f;
listenerUp.y = 1.f;
listenerUp.z = 0;
listenerForward.x = 0.f;
listenerForward.y = 0.f;
listenerForward.z = 1.0;
listenerPos.x = 3.f;
listenerPos.y = 3.f;
listenerPos.z = 1.f;

Hmm, I just had another look, I had though that that the forward parameter in this function was listener rotation:

FMOD_RESULT F_API FMOD_System_Set3DListenerAttributes(FMOD_SYSTEM *system, int listener, const FMOD_VECTOR *pos, const FMOD_VECTOR *vel, const FMOD_VECTOR *forward, const FMOD_VECTOR *up);

But after testing you are correct, it does not seem like that.

EDIT: I had another go with headphones and it is the up vector seemingly, but this does not seem to make sense with the description. Even being unsure about that you can rotate all the sources around the listener which has the same effect as rotating the listener. It may be neater than passing audio to another program.

@fresla, isn’t the forward vector for viewer orientation? it’s strange to see both forward and up vectors. i suppose you’d just translate the hmd orientation into a euler angles and go from there.

@mantissa I also thought forward was viewer orientation, but in a fast test up seemed to behave as so. Up also does not make sense at first but in a gaming scenario it kind of does - FPV game player looks up and hears something from above.

In the past i’ve used HOA library(igh order ambisonics library) : http://www.mshparisnord.fr/hoalibrary/en/

OFX library:
https://github.com/CICM/ofxHoa

I really love the work the have done on the most optimal way of using Ambisonics.