Warped FBO into openCV

Hi, maybe I have the complete wrong approach, but I am using quadwarping to match an image from a still camera to a video camera. They camera will be side by side but never in the exact same place as they need to be lined up in every venue.

I am overlaying the still image on top of the video image and warping the still image.

As the images are different sizes I scale the still image down to be close to the video image before I do the warp to line them up.

The warp is done with and FBO- I need to achieve a few things, the first is to apply the warp of the scaled image to the warp of the real larger image, the second is to use the warped image in openCV to do blob detection. ( I made a horrible hack where I grab the screen and feed it to openCV and this kind of works but I want to do the blob detection on the larger image as it has more resolution. The detection does not have to work on live video just the still. In the end I would like to use the found blob centroid coordinates on the incoming live video stream (I matched them up forst with the warping).

I have been having a lot of fun playing around with this, it would be great to get a pointer in the right direction- I have a feeling that drawing the warp with a glMatrix is not what I need to do. Here is some of the code.

void testApp::update()  
    if (bNewFrame){  
		bNewFrame = false;  
		badOnes	= new int[finder.blobs.size()];  
		frameTeller+= mouseY/ 100;  
		floatTeller+= mouseY/ 100;  
		if (frameTeller >= finder.blobs.size()*10){  
            frameTeller = 0;  
		if (floatTeller >= finder.blobs.size()){  
			floatTeller = 0;  
void testApp::draw()  
    ofSetColor( ofColor :: white );  
    //======================== draw image into fbo.  
        img.draw( 0, 0 ,1280,720);  
    //======================== get our quad warp matrix.  
    ofMatrix4x4 mat = warper.getMatrix();  
    //======================== use the matrix to transform our fbo.  
    glMultMatrixf( mat.getPtr() );  
        fbo.draw( 0,0);  
    copyImage.grabScreen(0, 0, 1920, 1080);  
    if (!findIt) {  
        glBlendFunc(1, 1);  
        grabber.draw(x, y, grabber.getWidth(), grabber.getHeight());  
    if (findIt) {  
    //ofTranslate(200, 200);  
	for(int i = 0; i < finder.blobs.size(); i++) {  
		ofRectangle cur = finder.blobs[i].boundingRect;  
		ofRect(cur.x, cur.y, cur.width, cur.height);  
void testApp::keyPressed(int key){  
	if (key == 'e'){  
	if (key == 'f' || key == 'F'){  
void testApp::keyReleased(int key){  
	if (key == 'e'){