Playing audio input in real time, problems syncing

Hi friends,
still working on an app that grabs audio and video from an external HDMI input. I found a cheap USB3 grabber that can be used by OF like a webcam.

Video is working, the quality is not so good (the grabber is very cheap), but that’s not the problem. What is worrying me are troubles with audio. I need to play it in realtime, synced with video, and had success declaring two ofSoundStream objects, one for input (from the HDMI grabber) and the other one for output (on PC headphones/speaker).

Now I am familiar with circular buffers and the like, and with the requirement of having the lowest possible latency, but the problem is that on my setup (windows 10) the input streams appears to be faster than the output one, from some measurements around 1% faster.

This of course leads to buffer overflow, and to clicks and distortion in output sound. Is there a way to avoid this? I can understand that two independent clock frequencies will never be exactly the same, but 1% is a huge difference :frowning:

I really cannot think of a practical solution, in principle the difference in sampling rate could be measured and the data resampled (and may be time stretched?) accordingly, but this would need some sophisticated algorithm, overkill for a simple loopback.

Any idea? Thanks
Davide

Hi. you can use the ofApp audio callbacks. look at the audioIn and audioOut examples so you can set up the audio streams.
make an instance of ofSoundBuffer and copy into it what you get from audioIn and then pass it to audioOut´s buffer.

should be something like this.

// put these lines in the ofApp.h file
ofSoundBuffer buffer;
void audioIn(ofSoundBuffer & input);
void audioOut(ofSoundBuffer & output);

// the following ones go in the ofApp.cpp file
void audioIn(ofSoundBuffer & input){
    input.copyTo(buffer);
}
void audioOut(ofSoundBuffer & output){
    buffer.copyTo(output);
}

if you need to resample the buffer in order to adjust speed, you can use the ofBuffer’s resampleTo() methdod.
http://openframeworks.cc/documentation/sound/ofSoundBuffer/#!show_resampleTo
hope this helps

Thank you, I am aware of audioIn and audioOut, that’s what I am currently using. The problem is that if you use two different devices for input and output the two callbacks are called with different frequencies. This is a sample of my code:

from ofApp.h:

ofSoundStream inputSoundStream, outputSoundStream;
void audioIn(ofSoundBuffer& buffer);
void audioOut(ofSoundBuffer& buffer);
deque <ofSoundBuffer> soundBufferQueue;
ofMutex audioMutex;

from ofApp.cpp:

void ofApp::audioOut(ofSoundBuffer &outBuffer)
{
	audioMutex.lock();

	if (soundBufferQueue.size() > 0)
	{
		outBuffer = soundBufferQueue.at(0);
		soundBufferQueue.pop_front();
	}
	else
	{
		ofLogNotice() << "Audio input queue is empty!";
	}

	ofLogNotice() << "Read from queue, size is now " << soundBufferQueue.size();

	audioMutex.unlock();
}

const int QUEUE_MAX_SIZE = 4;

void ofApp::audioIn(ofSoundBuffer &inBuffer)
{
	audioMutex.lock();

	if(soundBufferQueue.size() <= QUEUE_MAX_SIZE)
	{
	    soundBufferQueue.push_back(inBuffer);
	    ofLogNotice() << "Writing to queue, size is now " << soundBufferQueue.size();
	}
	else
	{
		ofLogNotice() << "Audio input queue is too large (>" << QUEUE_MAX_SIZE << ")!";
	}

	audioMutex.unlock();
}

it is not optimal since it continuously allocates and deallocates buffers, but it shows what’s happening. Now it drops a whole buffer when queue is too large (or empty), and this produces a small but audible “click”. If there are no ways to sync the two devices (I’m afraid there is none) the only way will be to keep some statistics of the two frequencies and resample the audio data accordingly. This will need to be done on a larger scale, resampling single buffers will not be enough… more of them (at least two I suppose) will have to be joined and then cut at the appropriate length, and the result resampled to make it fit. I’ll see what I can do and then share the results here.