Best way to get audio from two WebCams?

[s_e_p]

I have two USB web cams - a Logitech (Logicool) C920 and a C922 - and want to use both of their mics simultaneously.

I’m not able to get both of them working at the same time in ASIO4All using either pure data, touchDesigner or openFrameworks. I am able to get ASIO4All to work with certain combinations of other devices, eg. BOSS GP-10 & Presonus AudioBox iTwo, but not with the Logitech cameras. (Also, if anyone knows of an ASIO4All community, please let me know)

The only way I’ve been able to use both cam mics at all is in touchDesigner: by having two audioDeviceIns set to DirectSound driver and the respective camera device. But I don’t want to have to use TD for audio input unless absolutely needed, so I’ve been looking up how to do this in oF…

I’ve found this topic, though it’s very large, old, and the bulk of it seems to be about how to use two different sound cards simultaneously…

I’ve also found this topic where it’s implied that you can use two input devices on the same soundcard without any addons. @nylki was seemingly able to do it, but it’s also kind of old I’m not able to replicate it. (I also have only ever had success setting up audio using ofSoundStreamSettings objects, which @nylki doesn’t do)

Here’s my current code. The audioIn code I think I got from an oF audio example for getting the rms from a stereo mic and I’d like to implement it for two mics instead of one:

void ofApp::setup(){
	ofSoundStreamListDevices();
	ofSoundStreamSettings s1 =	audioInPreSetup(m_audioIn1, m_sound1L, m_sound1R, "マイク (HD Pro Webcam C920)");
	//ofSoundStreamSettings s2 = audioInPreSetup(m_audioIn2, m_sound2L, m_sound2R, "マイク (C922 Pro Stream Webcam)");
	m_audioIn1.setup(s1);
    //m_audioIn2.setup(s2);
}

ofSoundStreamSettings ofApp::audioInPreSetup(ofSoundStream& stream, std::vector<float>& l, std::vector<float>& r, std::string dev)
{
	int bufferSize = 512;
	
	ofSoundStreamSettings settings;

	l.assign(bufferSize, 0.0f);
	r.assign(bufferSize, 0.0f);
	auto devices = stream.getDeviceList(ofSoundDevice::Api::MS_DS);
	int devInd = 0;
	for (int i = 0; i < devices.size(); ++i)
	{
		std::cout << "device " << i << " : " << devices[i].name << endl;
		if (devices[i].name == dev)
		{
			devInd = i;
			std::cout << "setting audio device " << i << ": " << devices[i].name << endl;
			break;
		}
	}

	auto inDevice = devices[devInd];
	settings.setInDevice(inDevice);

	settings.setInListener(this);
	settings.numBuffers = 4;
	settings.sampleRate = 44100;
	settings.numOutputChannels = 0;
	settings.numInputChannels = 2;
	settings.bufferSize = bufferSize;
	return settings;
}

void ofApp::audioIn(ofSoundBuffer& input)
{
	float curVol1 = 0.0f;
	//float curVol2 = 0.0f;

	int numCounted = 0;
	
	for (size_t i = 0; i < input.getNumFrames(); i++) {
		m_sound1L[i] = input[i * 2] * 0.5; //would it be i*4 if using two stereo mics, ie. would all four channels come into the same buffer?
		m_sound1R[i] = input[i * 2 + 1] * 0.5;
		//m_sound2L[i] = input[i * 4 + 2] * 0.5;
		//m_sound2R[i] = input[i * 4 + 3] * 0.5;

		curVol1 += m_sound1L[i] * m_sound1L[i];
		curVol1 += m_sound1R[i] * m_sound1R[i];
		//curVol2 += m_sound2L[i] * m_sound2L[i];
		//curVol2 += m_sound2R[i] * m_sound2R[i];
		numCounted += 2; // 4 if two mics?
	}

	curVol1 /= (float)numCounted;
	curVol1 = sqrt(curVol1);
	//curVol2 /= (float)numCounted;
	//curVol2 = sqrt(curVol2);
}

EDIT: Using both cameras doesn’t consistently work in TD either. The C922 seems to be the problem- it doesn’t even work as as an MME input in PD.

[burton]

the problem is intriguing and to confirm things on the internal OF side, I made a class that allows 2 different output device drivers running on macOS, with a lambda-interface to pass the audio processing from the top-level.

this might not be exactly what you need but the implementation (in the class AudioClient) should give you enough to adapt (the same principles apply to input buffer). there a lots of assumptions; making something totally generic and adaptive to all setups/situation is not a quick thing.

src.zip (3,1 Ko); it would be great if you can confirm it works with 2 output devices on windows.

(in your specific case there might be an issue with the camera driver which might not expect 2 devices. note that i don’t have 2 identical output devices to rule out a problem within OF when using the same driver, but I don’t see why it might cause problem, unless the OS gets involved in some way).

the core usage looks like this:

void ofApp::setup(){
	clients_[0].setup(256, 48000, "DEL: DELL S2721QS", [](ofSoundBuffer& buffer){
		for (size_t i = 0; i < buffer.getNumFrames(); i++) {
			for (size_t n = 0; n < buffer.getNumChannels(); n++) {
				auto phase = int(ofGetElapsedTimef()) % buffer.getNumChannels();
				buffer[i*buffer.getNumChannels()+n] = ofRandom(-1,1)*(phase==n);
			}
		}
	});
	clients_[1].setup(256, 44100, "Apple Inc.: Mac mini Speakers", [](ofSoundBuffer& buffer){
		for (size_t i = 0; i < buffer.getNumFrames(); i++) {
			for (size_t n = 0; n < buffer.getNumChannels(); n++) {
				auto phase = int(ofGetElapsedTimef()*1.3) % buffer.getNumChannels();
				buffer[i*buffer.getNumChannels()+n] = ofRandom(-1,1)*(phase==n);
			}
		}
	});
}

[edit to add] the behaviour above is to cycle noise one output after another once per second, a little faster (*1.3) on the second device

[s_e_p]

Firstly, the reason C922 wasn’t working with anything was that I needed some kind of Windows update for it since it’s a new device. Doing that allowed me to use it in ASIO4All, TD, etc.

@burton I got your code working! It outputs the differently alternating noises from both my laptop and external speakers (using MS_DS api). Thanks to your help, I was able to do the same thing for input (and also became more familiar with lambda expressions). Two little issues I’d like your input on…

1. I get the vector-out-of-range error when I try to use an std::vector of AudioClient instead of a c-type array (as per the commented-out code below)

2. When using the c-type array, the two mics work fine at runtime, but I get a vector-out-of-range error whenever I close the project. Running ofSoundStreamClose() in ofApp::exit() doesn’t help this. Maybe I need to add an exit() to my custom classes that removes listeners and call this exit() from ofApp::exit()… Not sure how to do that exactly. Maybe with ofRemoveListener()?

Here’s the code:

ofxCustomSetup.h:

#include "ofMain.h"

class AudioClient : public ofBaseSoundInput, public ofBaseSoundOutput
{
private:
	ofSoundStream m_soundStream;
	ofSoundStreamSettings m_soundSettings;
	const float m_decaySmooth = 0.99f;
	const float m_attackSmooth = 0.75f;
	std::vector<float> m_soundL, m_soundR;
	int m_deviceIndex;
public:

	void audioIn(ofSoundBuffer& buffer) override
	{
		m_audioCallback(buffer);
	}

	float getRMS(ofSoundBuffer& buffer)
	{
		float curVol = 0.0;
		int numCounted = 0;

		for (size_t i = 0; i < buffer.getNumFrames(); i++) {
			m_soundL[i] = buffer[i * 2] * 0.5;
			m_soundR[i] = buffer[i * 2 + 1] * 0.5;

			curVol += m_soundL[i] * m_soundL[i];
			curVol += m_soundR[i] * m_soundR[i];
			numCounted += 2;
		}

		curVol /= (float)numCounted;
		curVol = sqrt(curVol);
		if (curVol > 0.05)
			cout << "vol " << m_deviceIndex << " : " << curVol << endl;
		return curVol;
	}

	std::function<void(ofSoundBuffer& buffer)> m_audioCallback;

	bool setup(int ind, size_t bufferSize, std::size_t sr, std::string deviceName,
		std::function<void(ofSoundBuffer& buffer)> callback)//, std::vector<float>& l, std::vector<float>& r)
	{
		m_deviceIndex = ind;
		m_audioCallback = callback;

		m_soundL.assign(bufferSize, 0.0f);
		m_soundR.assign(bufferSize, 0.0f);

		ofLogNotice("desiring device named") << deviceName;
		auto devices = m_soundStream.getDeviceList(ofSoundDevice::Api::MS_DS);
		for (int i = 0; i < devices.size(); ++i) {

			std::cout << "device " << i << ": " << devices[i].name << endl;
			if (devices[i].name == deviceName) {
				std::cout << "setting audio device " << i << ": " << devices[i].name << endl;
				m_soundSettings.setInDevice(devices[i]);
				m_soundSettings.setInListener(this);
				m_soundSettings.numInputChannels = 2;
				m_soundSettings.sampleRate = sr;
				m_soundSettings.numBuffers = 2;
				m_soundSettings.bufferSize = bufferSize;
				m_soundStream.setup(m_soundSettings);
				return true;
			}
		}
		return false;
	}
};

class ofxCustomSetup
{
private:
	int m_inIndex = 0;
public:
	//std::vector<AudioClient> m_audioClients;
	AudioClient m_audioClients[2];
	void audioInSetup(std::string deviceName);
	std::vector<float> m_audioInsRMS;	
};

ofxCustomSetup.cpp:

#include "ofxCustomSetup.h"

void ofxCustomSetup::audioInSetup(std::string deviceName)
{
	int ind = m_inIndex;
	//AudioClient a;
	//m_audioClients.push_back(a);
	m_audioInsRMS.push_back(0.0f);

	bool set = m_audioClients[m_inIndex].setup(ind, 256, 32000, deviceName, [this, ind](ofSoundBuffer& buffer)
	{
		m_audioInsRMS[ind] = m_audioClients[ind].getRMS(buffer);
	});
	if (set)
	{
		m_inIndex++;
	}
}

ofApp.h:

#include "ofMain.h"
#include "ofxCustomSetup.h"

class ofApp : public ofBaseApp{

	public:
		void setup();
		void exit() override;

		void keyPressed(int key);
		std::string m_camMic1name = "マイク (6- HD Pro Webcam C920)";
		std::string m_camMic2name = "マイク (C922 Pro Stream Webcam)";
		ofxCustomSetup m_custom;
		
};

ofApp.cpp:

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
	m_custom.audioInSetup(m_camMic1name);
	m_custom.audioInSetup(m_camMic2name);
}
void ofApp::exit() 
{
	ofSoundStreamClose();
}

[burton]

1. I can’t run the code now but for the vector usage problem I would expect a form of memory problem related to the scope of

	AudioClient a;
	m_audioClients.push_back(a);

Try with a vector of shared_ptr and see if it survives better.

2. For the exit error, you are calling ofSoundStreamClose(); which interacts with a “special” “freely provided” static “global” ofSoundStream object. I agree that there is a bit of funkiness around there, but at the same time that’s what’s making it possible to have ofSoundPlayer simply run transparently on all backends/platforms without dealing with ofSoundStream (which is actually always lurking). Perhaps those methods should be named along the lines of ofCloseSystemSoundStream() to reduce ambiguity. And/or maybe a way to prevent a given ofApp from instantiating that soundstream.

At exit you should call instead your own instances of ofSoundStream’s close methods.

But please post the complete error you get calling ofSoundStreamClose() as it should not crash the app.

[edit to add]: I put the std::function stuff there as a simple example of separating the definition of the DSP code from the class implementation, but in your case it’s an unneeded layer as all your instances will run the same RMS analyzer)

[burton]

allo keep in mind the sound stuff runs in it’s own thread – the callbacks are triggered by the Soundcard driver and will interrupt whatever is going on.

when you’ll be making use of your m_audioInsRMS[] data in update() or draw(), you may get sync problems. you may want to lock the access to the array with a mutex, or if you can live with just the “latest available value” (a 48000/256 audio callback runs at 187.5Hz) you can simply use std::atomic<float> and the locking will be done as well as the hardware can do it. (Not all types can be made atomic efficiently)

[s_e_p]

@burton Thanks for your help. Those problems are taken care of now.

1. Yup, a vector of shared_ptr does the trick. (I tried unique_ptr, but you can’t use push_back with that because unique_ptr can’t be copied and push_back copies)

2. ofSoundStreamClose() wasn’t causing the crash- the crash happened with or without it. As you said, calling close() on the individual soundstreams fixed the issue. I just call this method from ofApp::exit():

void ofxCustomSetup::exit()
{
	for (int i = 0; i < m_audioClients.size(); ++i)
	{
		m_audioClients[i]->m_soundStream.close();
	}
}

And yeah, you’re right about not needing the std::function since I can just put the call to getRMS() in the audioIn override. But still glad I learned how to use it.

I have another problem now- Not sure if it’s related to this setup but will make a new topic.