How to do simple audio amplification?

advanced
#1

I’m trying to get input from two audio sources and then output them with amplification.

@burton just helped me to get multiple audio inputs working. The two audioInputs use this method to set up their input:

AudioClient class (ofxCustomSetup.h)

bool setup(int ind, size_t bufferSize, std::size_t sr, std::string deviceName)
	{
		m_deviceIndex = ind;

		m_soundL.assign(bufferSize, 0.0f);
		m_soundR.assign(bufferSize, 0.0f);

		ofLogNotice("desiring device named") << deviceName;
		auto devices = m_soundStream.getDeviceList(ofSoundDevice::Api::MS_DS); //no asio option for these webcams
		for (int i = 0; i < devices.size(); ++i) {

			std::cout << "device " << i << ": " << devices[i].name << endl;
			if (devices[i].name == deviceName) {
				std::cout << "setting audio device " << i << ": " << devices[i].name << endl;
				m_soundSettings.setInDevice(devices[i]);
				m_soundSettings.setInListener(this);
				m_soundSettings.numInputChannels = 2;
				m_soundSettings.sampleRate = sr;
				m_soundSettings.numBuffers = 2;
				m_soundSettings.bufferSize = bufferSize;
				m_soundStream.setup(m_soundSettings);
				return true;
			}
		}
		return false;
	}
};

ofxCustomSetup.cpp

void ofxCustomSetup::audioInSetup(std::string deviceName)
{
	int ind = m_inIndex;
	std::shared_ptr<AudioClient> a = std::make_shared<AudioClient>();
	m_audioClients.push_back(a);
	m_audioInsRMS.push_back(0.0f);

	bool set = m_audioClients[ind]->setup(ind, 256, 32000, deviceName); //the camera microphones have a sample rate of 32000
	if (set)
	{
		m_inIndex++;
	}
}

The audioIn override uses this method (from examples/ audioInputExample) which saves the input buffer to two arrays:

float getRMS(ofSoundBuffer& buffer)
	{
		for (size_t i = 0; i < buffer.getNumFrames(); i++) {
			m_soundL[i] = buffer[i * 2] * 0.5; //this is what was in the original oF example code, implying that every other sample is from a different channel...?
			m_soundR[i] = buffer[i * 2 + 1] * 0.5;
            //m_soundL[i] = buffer.getSample(i, 0); //tried changing the code to this but it didn't change the result
			//m_soundR[i] = buffer.getSample(i, 1);

            //...code for getting RMS of the amplitude
		}
	}

In ofApp, I make a single audioOut override with a third soundstream:

void ofApp::audioOutSetup()
{

	int bufferSize = 256;

	ofSoundStreamSettings settings;

	auto devices = m_outStream.getDeviceList(ofSoundDevice::Api::MS_DS); //result is same with ASIO
	int outDevInd = 0;

	for (int i = 0; i < devices.size(); ++i)
	{
		if (devices[i].name == m_speakersName)
		{
			outDevInd = i;
			std::cout << "setting audio out device " << i << ": " << devices[i].name << endl;
			break;
		}
	}
	ofSoundDevice& outDevice = devices[outDevInd];
	settings.setOutDevice(outDevice);
	settings.setOutListener(this);
	settings.numBuffers = 2; //maybe two?
	settings.sampleRate = 32000;
	settings.numOutputChannels = 2;
	settings.bufferSize = bufferSize;
	m_outStream.setup(settings);
}

In the audioOut(), I try to copy from the audioIns’ buffers to the audioOut buffer:

void ofApp::audioOut(ofSoundBuffer& buffer)
{
	for (size_t i = 0; i < buffer.getNumFrames(); ++i)
	{
		auto input1 = m_custom.getStereoBuffers(0);
		auto input2 = m_custom.getStereoBuffers(1);
		float l = (input1[0][i] + input1[1][i]) * 0.5f;
		float r = (input2[0][i] + input2[1][i]) * 0.5f;
		buffer.getSample(i, 0) = l * m_outVol;
		buffer.getSample(i, 1) = r * m_outVol;
	}
}

The result sounds like the amplified input always autotuned to the same pitch- like a flanger effect, even if I set the volume low and point the speakers away from the mics. Can anyone tell me what I’m doing wrong? Should the output be the same soundstream as one of my inputs? N.b. when I change the output sampleRate to 44100 the effect goes from a clearly pitched flange to a noisy sound with a delay-echo. I vaguely remember @roymacdonald saying something about oF’s input and output buffers being connected- is that’s what’s causing this problem?

Best way to get audio from two WebCams?
#2

Hi @s_e_p
I would advise not using two different sound streams. Although the system might be able to manage it it feels like the source of a lot of potential problems.
Instead just use a single one, you can still set the in and out devices to be different ones but both will have the same samplerate, thus you will not have sample rate problems.
The in and out callbacks are connected, so to say, in the sense that when you create an ofSoundStream it runs its own threads in which it calls first the in callback (audioIn(ofSoundBuffer& buffer) ) and then audioOut(ofSoundBuffer&buffer). So having these in separate soundStreams means that you will have these running on different threads which gives you a lot of problems if not handled properly, which can be not easy. So, the easiest way is to handle all in the same ofSoundStream.

Also, you should be taking advantage of the ofSoundBuffer class as it already does several of the things you are doing manually. Like instead of storing its values in a vector and using the for loops you can just have an class instance of ofSoundBuffer to which you copy.

class AudioClient{
ofSoundBuffer workBuffer; // class instance
bool setup (...) {}// like the one you already have but call  `settings.setOutDevice(...);` and `settings.setOutListener(this);`

virtual void audioIn(ofSoundBuffer& buffer) override{
workBuffer = buffer; //this copies all the buffers data, including its sampleRate, size, etc.
}
float amplifyValue = 1.2;// whatever value.
virtual void audioOut(ofSoundBuffer& buffer) override{
buffer = workBuffer; // copy what came from input
buffer *= amplifyValue; // amplify.
}

};

But, are you using more than one input device? if so, it is ok that you use different streams but you will need to sync these properly using a mutex. Let me know if this is the case so I can send you some code to do such. and please explain me what is that you want to actually achieve.

As for amplifying the signal, just multiply the buffers values by a number. if that value is 1 nothing happens, less than one will diminish, more will amplify.

#3

Thanks for your reply. Yes, as discussed here, I’m trying to use the mics of two different web cams. I might still be able to do this with ASIO4All but wanted another option, whence the two input streams. I could set one of the two input streams to also handle the output, but yes, please share your syncing code.

(edit: not having luck with ASIO4All. It doesn’t even launch when I set it in oF…)

#4

To build on @roymacdonald’s insight: it’s one thing to use 2 “floating” input device for analysis, and quite another to mix them: individual digital audio devices have their own notion of time and sync, and some apparatus is required to make things sync. professional audio devices and sound cards have a way to be synched together, generally via a cable. in the case of USB devices, the OS has a bit of control, but at the cost of latency. the term to search for is word clock.

in my example I purposefully made a device at 44100 and one at 48000. it works without a problem for independent processes because each devices gets it’s own thread to control, but trying to mix them is of course a headache. even if your 2 devices had the exact same notion of time, there is no chance that they started their buffering at the same time, so your audio callbacks with never be synchronous.

I’m curious to see roy’s code for buffering/synchronizing streams, but I would advise finding an OS-level method of aggregating devices (there are ways to do so on macOS and linux; don’t know about windows), after which all the channels come in the same SoundStream.

audioIn and audioOut callbacks of a given SoundStream are in sync because both are invoked by the same device driver.

#5

I want the analysis and simple amplification, so until I hear from Roy, I guess two options are

  1. keep trying to get ASIO4All working (it seems to be the standard way of aggregating devices on Windows) EDIT: This is looking like a dead-end because the cam mics can be set max to 32000kHz and my Presonus AudioBox can be set to minimum 44100kHz (my laptop’s default Realtek soundcard is a fixed 48000!). It looks like you can’t mix and match devices with different samplerates.

  2. have two soundstreams and have each stream handle the amplification (in and out) for one of the camera mics independently of the other and output through one speaker respectively. (this is a limited approach, but good enough for what I’m doing this time) EDIT: I’m worried about the samplerate discrepancy also being an issue with this method, though at least in TouchDesigner it is possible with DirectSound to have both cameras as inputs and presonus AudioBox as an output- so there must be some way to do it in oF. For amplification, how can I convert a 32000kHz buffer to a 44100kHz buffer…?

EDIT: even in TouchDesigner, the audio quality of the amplification is really bad. I’m thinking it’s a mistake to even use these camera mics for amplification so will probably end up just doing everything through ASIO- Presonus Audiobox with two lavelier mics (with xlr converters). Still, it’s an interesting problem and I’m interested in having these potential fixes in my repertoire, so I’m still grateful for your help.

#6

Hi,

So I began writting the following a few days ago.

So you can convert between sample rates. ofSoundBuffer allows you to convert between sample rates but the algorithm does not work very well. So I made an addon for such, ofxSampleRate. I just realized it does not even have an example! But it is extremely simple, you make an instance of ofxSampleRate, and then call either changeSpeed or changeSampleRate functions.

As @burton says, each device has its own idea of time so these will never be in sync. Although if the input had the same sample rate as the output and same buffersize, a simple mutex would allow to sync, as each buffer will arrive at the same time difference relative to the each other, but this is not the case! So we need to do some additional buffering. There are several different techniques, but I think that a lock-free circular buffer is the most elegant one. I am using the following one right now where I am syncing succesfully several hundreds of audio streams, each one running in its own thread.
Check this class I have in another addon of mine.

I was about to paste the code I was using so I decided to check if it worked fine, but it did not.
Good thing, as I realized that the circular buffer was not working fine, it was introducing some ugly stuff, which was bothering in the analysis it was being used.

So, I will put here the code working properly asap.

1 Like
#7

@s_e_p my take on this (which is not an absolute, but acquainted with to the underlying issues of hardware devices buffering triggering interrupt callbacks into the OS up in multiples threads synchronized into the host app, and out again into an output, with no drops nor artefacts) is that it constitues a “hard problem”.

solving it for a specific set of operational parameters might be feasible, but it will be brittle, and even when apparently working, there will be issues lurking and waiting to manifest themselves just at the right time (i.e. opening or premiere night).

doing it with “floating” devices (not audio-synced, like your current 2 paths) or synchronizing instanced software devices (e.g. streaming clients) is an order of magnitude simpler. but as you notice just getting sample rate conversion correctly (in real time) is a challenge.

still looking forward to see @roymacdonald’s code on the subject!

1 Like
#8

Hey, so I ended up going into a rabbit hole, and came back victorious !

Good thing as I am actually working with this stuff and did not notice some problems there were.
Download the code from
src.zip (10.4 KB)
To run this code you will need this addon
So the important part is in the BufferedAudioInput class.
In particular this 2 functions

void BufferedAudioInput::audioIn( ofSoundBuffer& buffer ){
    if(_isSetup.load() && bOutputting.load()){
        if(ringBuffer){
            if(!sampleRateConverter){
                sampleRateConverter = make_unique<ofxSamplerate>() ;
            }
            
            ofSoundBuffer converted;
            
            sampleRateConverter->changeSampleRate(buffer, converted, _outputSampleRate.load());
            
            ringBuffer->writeFromBuffer(converted);
            
        }
    }
}

void BufferedAudioInput::audioOut( ofSoundBuffer& buffer ){
    if(!_isSetup.load() ) return;
    
    if( ! bOutputting.load()){
        _outputSampleRate = buffer.getSampleRate();
        bOutputting = true;
    }
    
    if(ringBuffer){
        
        ringBuffer->readIntoBuffer(buffer);
    }
}

in AudioIn the data is resampled to the output sample rate and added into the ringBuffer.
Then in audioOut the buffer is taken from the ringBuffer. The ringBuffer is thread safe, that is why it can be accessed from both functions, as each is being called by a differnt thread.
Converting from/to any sample rates should work. :).

If you want to use more than one input device, you can make another instance of BufferedAudioInput. and mix in the output.
For such I recommend you to use my ofxSoundObjects addon
and change the BufferedAudioInput inheritance from

class BufferedAudioInput:public ofBaseSoundInput, public ofBaseSoundOutput{

to the following

class BufferedAudioInput: public ofxSoundInput {
and remember to #include “ofxSoundObjects.h”

Then you just treat the BufferedAudioInputs as any other ofxSoundObject. Check ofxSoundObjects’ examples . In particular the Mixer and matrixmixer ones could be useful to you.

Let me know how it goes.

2 Likes