VoIP client. Problems with sound

Hey all,
I am trying my hand at a simple VOIP client. But I have the problem that when I send the recorded voice as vector via OSC and I want to play it via the soundstream just a beep comes.

//push to talk
void ofApp::keyPressed(int key){
	if (key == OF_KEY_RETURN) {
		push_to_talk = true;
		inputBuffer.resize(bufferSize * 1);
	}
}

void ofApp::keyReleased(int key)
{
	if (key == OF_KEY_RETURN) {
		std::cout << "Release" << endl;

		std::cout << " " << endl;
		for (int i = 0; i < inputBuffer.size(); i++) {
			std::cout << "Send" << endl;
			std::cout << inputBuffer[i] << endl;
			std::cout << " " << endl;
			ofxOscMessage m;
			m.setAddress("/typing");
			m.addFloatArg(inputBuffer[i]);
			clientSender.sendMessage(m, false);
		}
		push_to_talk = false;
		inputBuffer.clear();
		readPoint = 0;
		playSound = true;

		ofxOscMessage m;
		m.setAddress("/clear");
		clientSender.sendMessage(m, false);
		
	}

}

void ofApp::audioIn(float* buffer, int bufferSize, int nChannels)
{
	if (push_to_talk){
	for (int i = 0; i < bufferSize; i++) {
		inputBuffer[i * nChannels + 0] = buffer[i];
		ofxOscMessage m;
		m.setAddress("/typing");
		m.addFloatArg(inputBuffer[i]);
		clientSender.sendMessage(m, false);
	}
	
	}
	


}

void ofApp::audioOut(float* buffer, int bufferSize, int nChannels)
{
	

	std::cout << "AudioBuffer" + std::to_string(audioBuffer.size()) << endl;


	if (audioBuffer.size() > 0) {
		for (int i = 0; i < audioBuffer.size(); i++) {
			float inputSample;
			float currentSample;
			
			/*inputSample = audioBuffer[i];
			cout << "Sample: " + std::to_string(inputSample) << endl;
			currentSample = inputSample;*/

				currentSample = audioBuffer[i];


				buffer[i * nChannels + 0] = currentSample;
				buffer[i * nChannels + 1] = currentSample;
		}

	}

Is there a better way?
Thanks for the help

Hi,
This is a very inefficient way for sending audio over a network.
You can use ofxNDI if it is over a local network and it works really well.
I made this https://github.com/roymacdonald/ofxSoundObjects-NDI which should work for you.

In case you still want to implement with ofxOSC, you should try sending the data as a blob. In the examples it is shown how to send an image via OSC. Sending the sound buffer should be the same thing. But it is slow and I wouldn’t take it as a reliable method.

alternatively you might want to try using ofxNetwork and send data via UDP or TCP. Check the examples in examples/comminication/. Instead of sending a string you can send raw data, check the documentation
https://openframeworks.cc/documentation/ofxNetwork/

Hey @roymacdonald , thanks for your great help. Unfortunately I still have to ask a few stupid questions. With the UDP method I can only use chars, yet I have a vector with floats.
I had now implemented it with the OSC method, but still have problems with the output. It arrives, but it is played quite slow or even too fast.
Do you know what could be the reason for that?

Reciever

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
	//we run at 60 fps!
	ofSetVerticalSync(true);
	ofSetFrameRate(60);
	ofEnableAntiAliasing();

	//create the socket and bind to port 11999
	ofxUDPSettings settings;
	settings.receiveOn(11999);
	settings.blocking = false;

	udpConnection.Setup(settings);

	ofSetBackgroundAuto(false);
	ofSetBackgroundColor(255);

	sampleRate = 48000;
	bufferSize = 512;
	ofSoundStreamSetup(2, 1, sampleRate, bufferSize, 4);
	inputBuffer.resize(bufferSize * 1);

}

//--------------------------------------------------------------
void ofApp::update(){

	unique_lock<mutex> lock(audioMutex);
	char udpMessage[10000];
	udpConnection.Receive(udpMessage,10000);
	string message=udpMessage;

	if(message!=""){
		inputBuffer.clear();
		float x,y;
		vector<string> strPoints = ofSplitString(message,"[/p]");
		for (unsigned int i = 0; i < strPoints.size(); i++) {
			cout << strPoints[i] << endl;
			auto convert = std::stof(strPoints[i]);
			inputBuffer.push_back(convert/10000000);
		}
		
	}
	for (int i = 0; i < inputBuffer.size(); i++) {
		cout << inputBuffer[i] << endl;
	}

}

//--------------------------------------------------------------
void ofApp::draw(){

	ofSetColor(20);
	for(unsigned int i=1;i<stroke.size();i++){
		ofDrawLine(stroke[i-1].x,stroke[i-1].y,stroke[i].x,stroke[i].y);
	}

	ofDrawBitmapStringHighlight("UDP Receiver Example ", 10, 20);
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){

}

//--------------------------------------------------------------
void ofApp::keyReleased(int key){

}

void ofApp::audioOut(float* buffer, int bufferSize, int nChannels)
{
	unique_lock<mutex> lock(audioMutex);

	for (int i = 0; i < bufferSize; i++) {
		float inputSample;
		float currentSample;
		currentSample = inputBuffer[i];
		

		buffer[i * nChannels + 0] = currentSample;
		buffer[i * nChannels + 1] = currentSample;
	}
	
}

Sender

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
	ofSetWindowTitle("oscSenderExample");
	ofSetFrameRate(60); // run at 60 fps
	ofSetVerticalSync(true);

	// open an outgoing connection to HOST:PORT
	sender.setup(HOST, PORT);

	// preload image to send into a buffer
	imgAsBuffer = ofBufferFromFile("of-logo.png", true);


	sampleRate = 48000;
	bufferSize = 512;
	ofSoundStreamSetup(2, 1, sampleRate, bufferSize, 4);
	inputBuffer.resize(bufferSize * 1);
	push_to_talk = false;
	
	m.setAddress("/speaking");
}

//--------------------------------------------------------------
void ofApp::update(){

}

//--------------------------------------------------------------
void ofApp::draw() {
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){

	if (key == OF_KEY_RETURN) {
		push_to_talk = true;
		//inputBuffer.resize(bufferSize * 1);
	}
}

//--------------------------------------------------------------
void ofApp::keyReleased(int key){

	if (key == OF_KEY_RETURN) {
		push_to_talk = false;
		for (unsigned int i = 0; i < inputBuffer.size(); i++) {
			m.addFloatArg(inputBuffer[i]);
		}
		cout << m.getNumArgs() << endl;
		m.setAddress("/speaking");
		sender.sendMessage(m);
		m.clear();
		
	}
}



//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button){

}


void ofApp::audioIn(float* buffer, int bufferSize, int nChannels)
{
	//if (push_to_talk) {
		for (int i = 0; i < bufferSize; i++) {
			inputBuffer[i * nChannels + 0] = buffer[i];
			m.addFloatArg(inputBuffer[i]);
			
		}
		m.setAddress("/speaking");
		sender.sendMessage(m);
		m.clear();

	//}
}

Thanks for the help. Have a nice weekend

Hi, first dont lock both on audioOut and update. Just do it on update and you will be fine.

Then, the way you are filling the audio buffer will depend directly on the network speed. So it would be better to buffer that data before sending to audioOut. Say you have a buffer 4 times the size of the audio buffer, lets call it net buffer. when you get network data you put it at the end of the net buffer. Then in audioOut you get the data at the begining of the netBuffer and move everything forwards. Usually you would use something like a circular buffer, in order to avoid moving the data through the whole vector each time you pull out the beggining of the data.
Does this make sense?

Last but not least. Using the udp method you can cast from an array of floats to an array of unsigned chars. The data is not changed, at all, just you make the app think that it is reading an array of chars rather.

vector<floats> float_data;
unsigned char * data_as_uchars = reinterpret_cast<unsigned char *>(float_data.data());

then on the otherside you use the same but in reverse order.

Sorry, but I don’t really understand what you mean by that.
It would not really change anything. In update() I would do that:

for (int i = 0; i < bufferSize; i++) {
				netBuffer[i+512] = m.getArgAsFloat(i);
			}

and in audioout() this:

		for (int i = 0; i < bufferSize; i++) {
			float inputSample;
			float currentSample;
			currentSample = netBuffer[i];
			cout << netBuffer[i] << endl;

			buffer[i * nChannels + 0] = currentSample;
			buffer[i * nChannels + 1] = currentSample;
		}
		message = false;
		for (int i = 0; i < bufferSize; i++) {
			netBuffer[i] = netBuffer[bufferSize + i];
		}

Still only noise would be output and not the voice itself. Or am I just too stupid for that?

it is ok not to get something working and it does not mean you are stupid, at all. :slight_smile:

first, dont call this

cout << netBuffer[i] << endl;

it will slow down the for loop masively and it might be the reason why it does not work.

the thing is that what you get from the net is not in sync with what the sound device consumes, so you need to keep a few copies (buffers) of the received data before overwritting it.

then, using m.getArgAsFloat(i); is not the best idea because of the way OSC packs the data, it will increase each message size a lot and might be causing the data not to arrive in time.

ok. so, a super simple circular buffer would be something like

// declared in ofApp.h
vector<float> circularBuffer;
size_t netWriteIndex = 0;
size_t audioReadIndex = 0;
size_t numReceivedBuffers = 0;
size_t numReadBuffers = 0;

// intialize in setup
circularBuffer.resize(bufferSize *10, 0);

///lets think you use osc, in this case but it should be the same regardless the network method for sending data
//in update()
auto n = m.getNumArgs();
for (int i = 0; i < n; i++) {
				circularBuffer[(i+netWriteIndex)%n] = m.getArgAsFloat(i);
			}
netWriteIndex += n;
netWriteIndex %= n;
numReceivedBuffers ++;


// in audioOut(...)
if(numReceivedBuffers - numReadBuffers > 5){ this will allow it to have a minimum of 5 buffers before it starts reading. so the read and write indices dont overlap.
auto n = bufferSize;
for (size_t i = 0; i < n; i++) {
	buffer[i * nChannels + 0] = circularBuffer[(i+audioReadIndex)%n];
	buffer[i * nChannels + 1] = circularBuffer[(i+audioReadIndex)%n];
}
audioReadIndex += n;
audioReadIndex %= n;
numReadBuffers++;
}

btw, I guess that you are trying to implement this because you want to learn. Otherwise, check the addon i made which implements sending audio over a network using NDI. I posted it on a previous reply

Thanks :grin:, but sometimes it feels like that :joy:

It worked. Thank you very much. Now I am trying to understand it.

Yes exactly. I’m just trying to get into sound transmission and networks in general :slight_smile:
Thanks for your great help. :+1:

I will have a look at it as well.

1 Like