A method of adding a sound spectrum value to a videograbber pixel

void ofApp::setup(){

sound.load("beat.wav");
sound.play();

fft = new float[256];
for (int i = 0; i < 256; i++)
{
	fft[i] = 0;
}
bands = 256;

vidGrabber.setup(320,240);

int width = vidGrabber.getWidth();
int height = vidGrabber.getHeight();

for (int y = 0; y < height; y++){
	for (int x = 0; x < width; x++){
		mainMesh.addVertex(glm::vec3(x,y,0));	
		mainMesh.addColor(ofFloatColor(0,0,0));  
	}
}

for (int y = 0; y<height-1; y++){
	for (int x=0; x<width-1; x++){
		mainMesh.addIndex(x+y*width);				// 0
		mainMesh.addIndex((x+1)+y*width);			// 1
		mainMesh.addIndex(x+(y+1)*width);			// 10
		
		mainMesh.addIndex((x+1)+y*width);			// 1
		mainMesh.addIndex((x+1)+(y+1)*width);		// 11
		mainMesh.addIndex(x+(y+1)*width);			// 10
	}
}

extrusionAmount = 200;

}

void ofApp::update(){
//grab a new frame
vidGrabber.update();
ofSoundUpdate();
soundSpectrum = ofSoundGetSpectrum(bands);
for (int i = 0; i < bands; i++) 
{
	fft[i] *= 0.9;
	if (fft[i] < soundSpectrum[i])
	{
		fft[i] = soundSpectrum[i];
	}
}

for (int j = 0; j < bands; j++)
{
	extrusionAmount *= (fft[j] * 100000.0);
}

if (vidGrabber.isFrameNew()){

	for (int i=0; i<vidGrabber.getWidth()*vidGrabber.getHeight(); i++){

		ofFloatColor sampleColor(vidGrabber.getPixels()[i * 3] / 255.f,				// r
			                     vidGrabber.getPixels()[i * 3 + 1] / 255.f,			// g
			                     vidGrabber.getPixels()[i * 3 + 2] / 255.f);		// b
		
		glm::vec3 tmpVec = mainMesh.getVertex(i);

		tmpVec.z = sampleColor.getBrightness() * extrusionAmount;
		
		mainMesh.setVertex(i, tmpVec);
		mainMesh.setColor(i, sampleColor);
	}
}

}

void ofApp::draw(){

ofDisableDepthTest();
vidGrabber.draw(20,20);

}

Hello

I’m creating a webcam video that responds to sound by adding sound signals to the official example of openframeworks, “meshFromCameraExample”.

This is a method of multiplying the extruded value by the bright part of the pixel and storing it on the z-axis.

So I multiplied the extrusion value by the sound signal.

However, when I output the extrusion value, I output 0.

I think the code order is wrong.

Can I get a tip or method? I really want to know.

(I simplified the code to make it easier to see.)

Hey @jewel , is the following line correct?

	extrusionAmount *= (fft[j] * 100000.0);

I’m thinking that if any of the fft[] values are zero, or very very small, then extrusionAmount will be zero too. Maybe a sum would work well?

	extrusionAmount += (fft[j] * 100000.0);

Oh That’s not a negative value.

And if I express it in +=, the sound signal keeps adding up and the value gets too high.

Is there a way?

Can extrusionAmount be reset before a new value is calculated from fft[] each cycle? I’m thinking that using a sum will make extrusionAmount respond to the volume of the sound (more or less).

Wow, I don’t know why, but I succeeded.

for (int j = 0; j < bands; j++)
{
	extrusionAmount = 100;
	extrusionAmount *= (fft[j] * 7000.0);
}

I reset the extrusionAmount in the for statement, and the result I want comes out.

Thanks I used multiplication instead of adding.

Can I know why I succeeded?

The above loop will set extursionAmount equal to the last value in fft[] * 7000 * 100. How about some variation of the following?

extrusionAmount = 100;
for (int j = 0; j < bands; j++)
{
	extrusionAmount *= (fft[j] * 7000.0);
}

After modifying it like that, the extrusion amount does not work normally.

I don’t know why it’s like this either.

extrusionAmount = 100;
for (int j = 0; j < bands; j++)
{
	
	extrusionAmount += (fft[j] * 100.0);
}

The answer was to initialize outside the for statement and then +=.

The frame is very low, but…

It really helped me a lot. Thank you!

1 Like

It might run faster with a local ofPixels object, and avoid all the calls to VideoGrabber::getPixels():

        ofPixels pixels = vidGrabber.getPixels();
        unsigned char* pPixels = pixels.getData();

        for(size_t i{0}; i < pixels.getWidth() * pixels.getHeight(); i++)
        {
            ofFloatColor sampleColor(pPixels[i * 3] / 255.f, pPixels[i * 3 + 1]/ 255.f, pPixels[i * 3 + 2] / 255.f);
            glm::vec3 tempVec = mainMesh.getVertex(i);
            tempVec.z = sampleColor.getBrightness() * extrusionAmount;
            mainMesh.setVertex(i, tempVec);
            mainMesh.setColor(i, sampleColor);
        }