Loading opengl texture into openframeworks window

So I’m trying to combine opengl, opencv, and openframeworks all in the same window. Right now I’ve got a good openframeworks window open (a mod of the soundPlayerFFTExample that comes with openframeworks), and in another program I’ve loaded an image using opencv.
Now the hard part:

I open an openframeworks window. I load an image using opencv. I convert it pixel by pixel into an GLubyte array, so it can be used as a texture by openGL. Then I create an openGL shape to draw the texture onto. It builds correctly (Visual Studios 2005, XP professional), but crashes before filling in the window it loads.

Here’s the code:
#include “testApp.h”
#include <stdio.h>
#include <tchar.h>
#include <cv.h>
#include <cxcore.h>
#include “highgui.h”
#include <time.h>
void testApp::LoadTextureInit() {
image = cvLoadImage( “73.jpg”, 1);
imageWidth=667;
imageHeight=1000;

for(int m=0; m<667;m++)
{
for(int n=0; n<1000;n++)
{
CvScalar q=cvGet2D(image,n,m);
imagedata[m][n][0]=(GLubyte)q.val[0];
imagedata[m][n][1]=(GLubyte)q.val[1];
imagedata[m][n][2]=(GLubyte)q.val[2];
}
}
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_TEXTURE_2D);
glPixelStorei (GL_UNPACK_ALIGNMENT, 1);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
glGenTextures(1, &texture1);
glBindTexture(GL_TEXTURE_2D, texture1);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexImage2D (GL_TEXTURE_2D, 0, GL_RGB, imageWidth, imageHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, imagedata);
}

void testApp::loadTexture_Ipl() {
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_TEXTURE_2D);
glPixelStorei (GL_UNPACK_ALIGNMENT, 1);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
glBindTexture(GL_TEXTURE_2D, texture1);
glBegin(GL_QUADS);
glTexCoord2f(0.0f, 0.0f); glVertex3f(-2.0f, 1.0f, 0.0f);
glTexCoord2f(0.0f, 1.0f); glVertex3f(-2.0f, 1.0f, -1.0f);
glTexCoord2f(1.0f, 1.0f); glVertex3f( 0.0f, 1.0f, 0.0f);
glTexCoord2f(1.0f, 0.0f); glVertex3f( 0.0f, -1.0f, 0.0f);
glTexCoord2f(0.0f, 0.0f); glVertex3f(1.0f, -1.0f, 0.0f);
glTexCoord2f(0.0f, 1.0f); glVertex3f(1.0f, 1.0f, 0.0f);
glTexCoord2f(1.0f, 1.0f); glVertex3f( 2.41421f, 1.0f, -1.41421f);
glTexCoord2f(1.0f, 0.0f); glVertex3f( 2.41421f, -1.0f, -1.41421f);
glEnd();
glFlush();
glDisable(GL_TEXTURE_2D);
cvReleaseImage(&image);
}
void testApp::setup(){

// load in sounds:
beat.loadSound(“sounds/jdee_beat.mp3”);
ow.loadSound(“sounds/ow.mp3”);
dog.loadSound(“sounds/dog.mp3”);
rooster.loadSound(“sounds/rooster.mp3”);
//beet.loadSound(“sounds///beet.wma”);

// we will bounce a circle using these variables:
px = 300;
py = 300;
vx = 0;
vy = 0;

// the fft needs to be smoothed out, so we create an array of floats
// for that purpose:
fftSmoothed = new float[8192];
for (int i = 0; i < 8192; i++){
fftSmoothed[i] = 0;
}
// openCvShit();
LoadTextureInit();
nBandsToGet = 128;
}

//--------------------------------------------------------------
void testApp::update(){

ofBackground(80,80,20);

// (1) we increase px and py by adding vx and vy
px += vx;
py += vy;

// (2) check for collision, and trigger sounds:
// horizontal collisions:
if (px < 0){
px = 0;
vx *= -1;
dog.play();
} else if (px > ofGetWidth()){
px = ofGetWidth();
vx *= -1;
ow.play();
}
// vertical collisions:
if (py < 0 ){
py = 0;
vy *= -1;
rooster.play();
//beet.play();
} else if (py > ofGetHeight()){
py = ofGetHeight();
vy *= -1;
beat.play();
}
// (3) slow down velocity:
vx *= 0.996f;
vy *= 0.996f;

// (4) we use velocity for volume of the samples:
float vel = sqrt(vx*vx + vy*vy);
ow.setVolume(MIN(vel/5.0f, 1));
beat.setVolume(MIN(vel/5.0f, 1));
dog.setVolume(MIN(vel/5.0f, 1));
rooster.setVolume(MIN(vel/5.0f, 1));
//beet.setVolume(MIN(vel, 1));
//beet.setMultiPlay(1);
// (5) grab the fft, and put in into a “smoothed” array,
// by taking maximums, as peaks and then smoothing downward
float * val = ofSoundGetSpectrum(nBandsToGet); // request 128 values for fft
for (int i = 0;i < nBandsToGet; i++){

// let the smoothed calue sink to zero:
fftSmoothed[i] *= 0.96f;

// take the max, either the smoothed or the incoming:
if (fftSmoothed[i] < val[i]) fftSmoothed[i] = val[i];

}
loadTexture_Ipl();
}

//--------------------------------------------------------------
void testApp::draw(){

ofEnableAlphaBlending();
ofSetColor(255,255,255,100);
ofRect(100,ofGetHeight()-300,5*128,200);
ofDisableAlphaBlending();

// draw the fft resutls:
ofSetColor(255,255,255,255);

float width = (float)(5*128) / nBandsToGet;
for (int i = 0;i < nBandsToGet; i++){
// (we use negative height here, because we want to flip them
// because the top corner is 0,0)
ofRect(100+i*width,ofGetHeight()-100,width,-(fftSmoothed[i] * 200));
}

// finally draw the playing circle:
for( int i=0;i<nBandsToGet; i++) {
//beet.setVolume((py/512)+512);
//beet.setSpeed(px/384);
ofEnableAlphaBlending();
ofSetColor(((int)fftSmoothed[i]*500*(i%256))%256,((int)(fftSmoothed[i]*i*500)%128),((int)(fftSmoothed[i]*i*10)%256),64);
ofCircle(px+(i%200), py+(i%200),(float)fftSmoothed[i]*500.0f);
ofDisableAlphaBlending();
ofSetColor(0xffffff);
ofCircle(px+(i%100), py+(i%100)*pow(-1,(double)i),(float)fftSmoothed[i]*80.0f); }
}

//--------------------------------------------------------------
void testApp::keyPressed (int key){
}

//--------------------------------------------------------------
void testApp::keyReleased(int key){

}

//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y ){

}

//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button){
// add into vx and vy a small amount of the change in mouse:
vx += (x - prevx) / 20.0f;
vy += (y - prevy) / 20.0f;
// store the previous mouse position:
prevx = x;
prevy = y;
}

//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button){
prevx = x;
prevy = y;
}

//--------------------------------------------------------------
void testApp::mouseReleased(){

}

Contents of testApp.h:
#ifndef _TEST_APP
#define _TEST_APP

#include “ofMain.h”
#include “ofAddons.h”
#include “cv.h”
#include “highgui.h”

class testApp : public ofSimpleApp{

public:

void setup();
void update();
void draw();

//void openCvShit();
void keyPressed (int key);
void keyReleased(int key);
void mouseMoved(int x, int y );
void mouseDragged(int x, int y, int button);
void mousePressed(int x, int y, int button);
void mouseReleased();
void loadTexture_Ipl();
void LoadTextureInit();

ofSoundPlayer beat;
ofSoundPlayer ow;
ofSoundPlayer dog;
ofSoundPlayer rooster;
ofSoundPlayer beet;

GLubyte imagedata[667][1000][3];
IplImage* image;

int imageWidth;
int imageHeight;

GLuint texture1;
float * fftSmoothed;
// we will draw a rectangle, bouncing off the wall:
float px, py, vx, vy;
GLfloat angle;

int nBandsToGet;
float prevx, prevy;
};

#endif

Contents of main.cpp:
#include “ofMain.h”
#include “testApp.h”

//========================================================================
int main( ){

ofSetupOpenGL(1024,768, OF_FULLSCREEN); // <-------- setup the GL context

// this kicks off the running of my app
// can be OF_WINDOW or OF_FULLSCREEN
// pass in width and height too:

ofRunApp(new testApp());

}

THANKS in advance. I’m very new to OpenGL, and figure if I can get this running I can start playing with it, and learn some stuff. Thanks again!

Hi,

There are two things I would do differently.

First, to load an image to be used as an openGL texture, I wouldn’t use openCV but rather use an ofImage object, from which it is easy to get a pixel array:

  
  
ofImage myTexture;  
myTexture.loadImage("flyer2.png");  
unsigned char* pixels = myTexture.getPixels();  
  
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 512, 512, 0,GL_RGB, GL_UNSIGNED_BYTE, pixels); /// load tetxure  
  

There is probably an even better way of doing this, but this should work.

Second. If you really want to use OpenCV, i would recomend using the addon provided by Openframeworks and not use an external version (you can use the openCV addon example as a basis). You still have access to the whole openCV library minus highGui.

hope this helps

regards

david

**So I changed it to look like this (ofImage myTexture is declared in header):


void testApp::LoadTextureInit() {
//image = cvLoadImage( “73.jpg”, 1);
imageWidth=667;
imageHeight=1000;

myTexture.loadImage(“73.jpg”);
unsigned char* pixels = myTexture.getPixels();**
/*for(int m=0; m<667;m++)
{
for(int n=0; n<1000;n++)
{
CvScalar q=cvGet2D(image,n,m);
imagedata[m][n][0]=(GLubyte)q.val[0];
imagedata[m][n][1]=(GLubyte)q.val[1];
imagedata[m][n][2]=(GLubyte)q.val[2];
}
}*/
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_TEXTURE_2D);
glPixelStorei (GL_UNPACK_ALIGNMENT, 1);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
glGenTextures(1, &texture1);
glBindTexture(GL_TEXTURE_2D, texture1);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
// glTexImage2D (GL_TEXTURE_2D, 0, GL_RGB, imageWidth, imageHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, imagedata);
**
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 512, 512, 0,GL_RGB, GL_UNSIGNED_BYTE, pixels); /// load tetxure **

**
Should I still be loading the texture information into the GLubyte texture1? Also, I have been using openCV instead of the openframeworks openCV addon because I can’t find documentation for OF’s opencv. Do you know of any?
Thanks,
Steve**

You do realize that both ofImage and ofxCvImage create and manage openGL Textures. You can draw them directly with myImage.draw(x, y) or myImage.draw(x, y, w, h). So you don’t need to do any openGL stuff. Am I misunderstanding what you are trying to do?

As i understand it, the openCV addon is an interface between OF and OpenCV. If you want you can bypass the interface and use pure OpenCV or you could look at the example project and get an idea of the available commands (which i would recomend because it makes integrating openCV into OF a lot easier)

Anyway, my openCV comment was more of an intuitive remark. There should be no problem in using the “external” library. In fact, this is exactly what I did when I was reading the O’reilly book on openCV, and it worked fine in OF 0.5. It is just that I had a bad experience doing this in version 0.573. I had a program that would compile but not execute because of some kind of conflict between Poco events and Opencv (maybe because I was using the highgui component of Opencv).

I guess what I am trying to say is that it can be quite overwhelming to debug code that is crashing, especially if you have lots of things going on and you haven’t tried compiling a smaller subset before.

For instance, if i wanted to use an “external” version of the openCV library with OF, I would start with an empty project, include the openCv headers, declare an IplImage*, maybe load an image into it. and test that. If this works, build from there, and test-compile often.

Finally, if you’re new to the framework. I would recomend you play with the examples and stay within the boundaries of the framework until you get a good idea of what is made available. Only then would I recomend looking into “extending” the framework (using pure openCV and openGL etc).

I hope this helps.

regards

a bit off-topic:

[quote author=“david.demainlalune”]and test that. If this works, build from there, and test-compile often.[/quote] with all the event stuff, compile-test might not be enough. the newly integrated libpoco also heavily relies on unit-testing. specialy for integrating diffent frameworks within yours, updating, verify their functionality.

has anyone used tdd/bdd with cpp in here before? is there potenial for providing a test-suite with oF?

http://pocoproject.org/wiki/index.php/Unit-Testing
http://en.wikipedia.org/wiki/Test-driven-development

iam not saying its the way oF users should code, but for the library itself its important to build somethings solid users than can rely on.

Thanks for all the replies. With the program I uploaded, I got the OF stuff working, so jumped the gun and tried to implement OpenGL with an openCV texture on it at the same time.

Memo: You do realize that both ofImage and ofxCvImage create and manage openGL Textures.

I did not realize this, but discovered it around 2 AM last night when I used a glTranslatef and the rectangle and square i drew with ofCircle and ofRect which had previously stayed in 2d space started flying around the screen on some 3d axis. Most silly looking bug ever.

david.demainlalune: In fact, this is exactly what I did when I was reading the O’reilly book on openCV, and it worked fine in OF 0.5.

I’m reading the o’reilly book right now too, and that’s why I was using pure opencv. Eventually I want to implement machine learning with the opencv and from what little I’ve found about OF’s opencv addon, it doesn’t implement this.
By the way, what did you think of the O’reilly book? Do the things it describes go above and beyond OF’s opencv in the areas of image detection, blob detection and machine learning?

One more thing, I want to do particle stuff. Can OF handle that, or would you have to throw some pure openGL in the mix?