Accessing texture of ofxHapPlayer using pointers

Hello,
I’m playing back hap video files with ofxHapPlayer, and I’m trying to layer two at a time on top of each other and blend them with a custom shader. I wrote a shader (based on some examples in the Intro to Shaders chapter of ofBook) that works the way I want it to by accessing textures of the standard ofVideoPlayer and passing them into the shader via shader.setUniformTexture(), but I’m having trouble applying it to ofxHapPlayer.
From what I understand, the key difference is that HapPlayer.getTexture() returns a pointer to an ofTexture, rather than a reference, as demonstrated in this post:

I’m new to programming in C++ so I’m still a bit shakey on how to use pointers and dereferencing operators. I’ve emulated the code in the post above to access the texture of ofxHapPlayer, but it doesn’t seem to be passing on to the shader correctly. When I run my code with ofVideoPlayers, I get something like the following, which is what I’m expecting:


But when I run it with ofxHapPlayers and pass in the shader by dereferencing (?) the pointer, the screen stays black. My code below shows how I’m trying to do it with ofxHapPlayer. What am I doing wrong?

Note: the video files play normally with ofxHapPlayer, and are in Hap format, not Hap Alpha or Hap Q.

main.cpp, based on a shader example:

#include "ofMain.h"
#include "ofApp.h"

//========================================================================
int main( ){
    
#ifdef OF_TARGET_OPENGLES
    ofGLESWindowSettings settings;
    settings.glesVersion=2;
#else
    ofGLWindowSettings settings;
    settings.setGLVersion(3,2);
#endif
    ofCreateWindow(settings);
    
    // this kicks off the running of my app
    // can be OF_WINDOW or OF_FULLSCREEN
    // pass in width and height too:
    ofRunApp(new ofApp());
    
}

ofApp.h:

#pragma once

#include "ofMain.h"
#include "ofxHapPlayer.h"

class ofApp : public ofBaseApp{

	public:
		void setup();
		void update();
		void draw();

		void keyPressed(int key);
		void keyReleased(int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased(int x, int y, int button);
		void mouseEntered(int x, int y);
		void mouseExited(int x, int y);
		void windowResized(int w, int h);
		void dragEvent(ofDragInfo dragInfo);
		void gotMessage(ofMessage msg);
    
    ofFbo destFbo;
    ofFbo maskFbo;
    
    ofxHapPlayer hap;
    ofxHapPlayer hap2;
    
    
    ofShader shader;
};

ofApp.cpp:

#include "ofApp.h"


//--------------------------------------------------------------
void ofApp::setup(){

   //load shaders:
    shader.load("colorShaders/shader");

    //set up FBOs
    destFbo.allocate(500,500);
    maskFbo.allocate(500,500);
    
    //set up hap players
    hap.load("hapnotq.mov");
    hap.play();

    hap2.load("hapnotq2.mov");
    hap2.play();
   
}

//--------------------------------------------------------------
void ofApp::update(){
//HapPlayer works without an update routine?
}

//--------------------------------------------------------------
void ofApp::draw(){
    maskFbo.begin();
    ofClear(0,0,0,255);
    maskFbo.end();
    
    
    ofTexture *hapTexture = hap.getTexture();
    ofTexture *hap2Texture = hap2.getTexture();
    destFbo.begin();
    ofClear(0,0,0,255);
    shader.begin();
    shader.setUniformTexture("tex0", *hapTexture, 1);
    shader.setUniformTexture("tex1", *hap2Texture, 2);
    maskFbo.draw(0,0);//not entirely sure why I have to do this, but it's in an example and it doesn't work without it
    shader.end();
    destFbo.end();
    destFbo.draw(0,0);

}

I didn’t read the entire post but I think ofxHapPlayer use a shader to make the color conversion with a help from GPU, so maybe this is your issue.
If it is the case you will be able to do what you are trying by drawing the ofHapPlayer inside a ofFbo and getting the texture from there.

1 Like

I actually did try this before, but the videos were tinted yellow. Might this be because the FBOs weren’t allocated correctly? Like the FBOs were some how incompatible with the way that the Hap textures were being drawn?

It looks this post recommends the same thing. I’ll try again and play around with the FBO allocation.

Don’t know what I was doing wrong before, but this time it worked hahaha.
Set up an FBO for each HapPlayer, draw the hap frames to that FBO, and then pass the texture of the FBOs into the shader.

1 Like

Hi, you could do it without all the fbo stuff, but it would require you to process the textures in the same way as ofxHapPlayer already does. It can be a bit more cumbersome but might improve performance. If performance is not an issue for you go with what you are doing.

you point out the following:
maskFbo.draw(0,0);//not entirely sure why I have to do this, but it's in an example and it doesn't work without it
In order for the shader to work it needs to have some vertices (think of it as a surface) over which to draw. calling maskFbo.draw(0,0); certainly will pass this surface but it is unnecesary. You can simply call ofDrawRectangle(x, y, w, h) instead.

Hi, thanks very much!
I’ve only tested this out with two videos at a time, but I’m planning to do this process with many videos concurrently, so performance may soon quickly become an issue. If you have any recommendations on readings/resources that might explain how to process the textures in the way that ofxhapplayer does, that would be very much appreciated.
Also thanks for explaining the note about the vertices, that does make more sense now.

I just read the ofxHapPlayer code. It is quite straight forward though. It seems to be a “color-correction”. Check its code, the shader code is embeded in ofxHapPlayer.cpp.
What seems like it could work is that ofxHapPlayer has a getShader() which returns a pointer to the shader it uses internaly, so you can call this method and load it with a different shader, which includes the code that ofxHapPlayer already uses, as well pass to it the textures of the other players. Check ofxHapPlayer::draw

Oh interesting, never would have thought to try that. I’ll see if I can make that work, that would be a convenient way of handling it.

I’ve been trying to get this method to work this week to no avail, but have recently discovered that unless the video file is in Hap Q or Hap Q Alpha format, ofxHapPlayer does NOT use a shader to draw the texture. The color conversion is only necessary for those files because they use a YCoCg colorspace. If the file is in standard Hap format, ofxHapPlayer.getShader() returns a nullptr, and it therefore doesn’t use a shader in its draw loop. I’ve confirmed this by drawing the textures from a standard Hap file directly like so:

void ofApp::update(){
    
    hapTexture = hap.getTexture(); //get a pointer to the texture in the player
}

//--------------------------------------------------------------
void ofApp::draw(){
    hapTexture->draw(0,0); //drawing without a shader, looks fine!
}

While this is helpful in that it means I likely won’t need to mess with color conversion, it means I’m still confused about why I can’t successfully pass these textures to a custom shader. As a test, the following very simple shader results in a black screen with textures from two ofxHapPlayer, but displays colors as expected with textures from two ofVideoPlayers:

//FRAGMENT SHADER

#version 150

uniform sampler2DRect tex0;
uniform sampler2DRect tex1;

in vec2 texCoordVarying;

out vec4 outputColor;

void main(){
    
    vec4 tex0Col = texture(tex0,texCoordVarying);
    vec4 tex1Col = texture(tex1,texCoordVarying);
    
    float rnew = mod(1.0 + (tex0Col.r-tex1Col.r),1.0);
    float gnew = mod(1.0 + (tex0Col.g-tex1Col.g),1.0);
    float bnew = mod(1.0 + (tex0Col.b-tex1Col.b),1.0);
    
    outputColor = vec4(rnew,bnew,gnew,1.0);
}


My draw loop still includes drawing of an FBO because drawing ofDrawRectangle, ofMesh, or ofPlanePrimitive all resulted in a single color being displayed for the entire screen rather than the expected output from the shader:

//if using ofVideoPlayers:
void ofApp::draw(){
    shader.begin();
    shader.setUniformTexture("tex0", video.getTexture(), 1);
    shader.setUniformTexture("tex1", video2.getTexture(), 2);
//    ofDrawRectangle(0,0,ofGetWidth(),ofGetHeight()); // doesn't work! behaves like one giant pixel
    fbo.draw(0,0);
    shader.end();

}

//if using ofxHapPlayers:
void ofApp::draw(){
    ofTexture * hapTexture = hap.getTexture();
    ofTexture * hap2Texture = hap2.getTexture();
    shader.begin();
    shader.setUniformTexture("tex0", *hapTexture, 1);
    shader.setUniformTexture("tex1", *hap2Texture, 2);
    maskFbo.draw(0,0);
    shader.end();

}

I see. Maybe you can post an issue in ofxHapPlayers github and get an answer from its developer.

For anyone following this, I’ve opened the following issue on GitHub: