OpenGL ES 2.0 image processing

Hi everyone

Very excited about OF 0.8.0, especially the of OpenGL ES 2.0 support. I’m trying to do some basic image processing but failing miserably :slight_smile: I’ve googled like a maniac the last couple of days but can’t seem to get started. I think the shader part should be straightforward enough, it’s the first hurdle at which I’m stuck - the OF / objective C++ bit.

I’m trying to use a shader to alter an image and draw the results to an FBO. I’d usually write something like this.

  
  
MyFbo.begin();    
ofClear(0, 0, 0);    
MyShader.begin();    
MyShader.setUniform1i("randomValue", randomValue);    
MyFbo.draw(0, 0);    
MyShader.end();    
MyFbo.end();    
  

After some research, it appears I need to pass the texture to the shader rather than draw it between the begin() and end() functions. It also seems to be the case that I need to specify the drawable region and probably the texCoords. Most examples seem to be written in Objective-C which mostly baffles me.

Here are my very simple test shaders to date. As a first step I’m just trying to do is display and make minor adjustments to the texture. Any help would be much appreciated.

TestShader.vert

  
  
attribute vec4 a_position;    
attribute vec2 a_texCoord;    
uniform mat4 u_MVPMatrix;    
varying mediump vec2 v_texCoord;    
    
void main()    
{    
    gl_Position = u_MVPMatrix * a_position;    
    v_texCoord = a_texCoord;    
}    
  

TestShader.frag

  
  
#ifdef GL_ES    
precision mediump float;    
#endif    
varying vec2 v_texCoord;    
uniform sampler2D s_texture;    
    
void main()    
{    
    vec4 col = texture2D(s_texture, v_texCoord);       
    if (col.r > 0.5)    
        col = vec4(1.0, 0.0, 0.0, 1.0);    
    gl_FragColor = col;    
}  
  

Hey James,

Hope this isn’t too scattered,/misleading I am not too familiar with the IOS particulars but have been working with OpenGL ES 2 shaders on the RPi and still figuring some of it out

Where are you drawing your texture? I see the MyFbo.draw(0,0) but can you do that in between MyFbo.begin(), MyFbo.end()?

usually it is something like this

  
ofTexture someTexture;   
ofLoadImage(someTexture, "whatever.png"); //or do whatever to get something in here  
  
MyFbo.begin();    
ofClear(0, 0, 0);    
MyShader.begin();    
MyShader.setUniform1i("randomValue", randomValue);    
someTexture.draw(0, 0);  
MyShader.end();    
MyFbo.end();    
  

I think if you change s_texture to tex0 it may automagically work (still figuring this out) or if you want to be explicit you can

  
shader.setUniformTexture("s_texture", someTexture, someTexture.getTextureData().textureID);  
MyShader.setUniform1i("randomValue", randomValue);   
ofRect(0, 0, someTexture.getWidth(), someTexture.getHeight());  

I haven’t gone through this completely myself but this is new
http://www.openframeworks.cc/tutorials/graphics/shaders.html

and for your other attribute names (a_position, a_texCoord, u_MVPMatrix) you may want to try the ones used here as they may come in for “free”

https://github.com/openFrameworks/openFrameworks/blob/master/examples/ios/iosES2ShaderExample/bin/data/shaders/noise.vert

Thanks for your suggestions Jason. Good spot on the mistake there. In my code I was actually drawing the texture to the FBO.

I changed the attribute names as you suggested and got a little closer. The FBO is now filled with the colour from the top left pixel of the loaded texture. I can change this colour by adding a normalised vec2 position to texCoordVarying:

  
  
vec4 col = texture2D(tex0, texCoordVarying + vec2(0.01, 0.01));  
  

Here’s my code:

testApp::draw()

  
  
ofTexture tex = catImage.getTextureReference();  
int texID = catImage.getTextureReference().getTextureData().textureID;  
  
myFbo.begin();  
ofClear(0, 0, 0);  
myShader.begin();  
myShader.setUniformTexture("tex0", tex, texID);  
ofRect(0, 0, tex.getWidth(), tex.getHeight());  
myShader.end();  
myFbo.end();  
      
myFbo.draw(0, 0);  
  

MyShader.vert

  
  
attribute vec4 position;  
attribute vec2 texCoord;  
uniform mat4 modelViewMatrix;  
uniform mat4 projectionMatrix;  
varying vec2 texCoordVarying;  
  
void main()  
{  
    gl_Position = projectionMatrix * modelViewMatrix * position;  
    texCoordVarying = texCoord;  
}  
  

MyShader.frag

  
  
#ifdef GL_ES  
// define default precision for float, vec, mat.  
precision mediump float;  
#endif  
  
varying vec2 texCoordVarying;  
uniform sampler2D tex0;  
  
void main()  
{  
    vec4 col = texture2D(tex0, texCoordVarying + vec2(0.01, 0.01));  
    if (col.r > 0.5) col = vec4(1.0, 0.0, 0.0, 1.0);  
    gl_FragColor = col;  
}  
  

It seems I do need to draw to an object (rather than wrap shader.begin()/end() around texture.draw()). I’ve tried drawing to an ofRectangle, ofPlanePrimitive and an ofMesh (with normalised and standard texCoords) all with similar results. I wondered if this might ARB issue but disabling/enabling makes no difference.

I am curious if this makes any difference…

try removing

  
myShader.setUniformTexture("tex0", tex, texID);    
ofRect(0, 0, tex.getWidth(), tex.getHeight());   

and replace with

  
catImage.draw(0, 0);  
  

Just tried and that produces the same results as in my previous post - fbo taking the colour of the top left pixel of the texture. Weird, I’m sure I checked this earlier and saw a black square where the image should have been.

It looks as though you might not need to pass the texture as an attribute and draw to a graphic after all.

is it possible these numbers are too small or too small for the precision?

vec2(0.01, 0.01)

I tried different precision levels and saw the same results. I’ve also tried altering this fragment offset vector by passing different values to the shader as uniform vars.

  
  
myShader.setUniform1f("normXPos", (float)ofGetMouseX() / (float)ofGetWidth());  
myShader.setUniform1f("normYPos", (float)ofGetMouseY() / (float)ofGetHeight());  
  

This allows my to change the FBO colour to the various pixels of the image by dragging across the screen. Weirdly I can scan the whole width of the 200 x 200px image within 0 - 0.78. Same for the height.

Hello

The same problem has also been observed me.

“The FBO is now filled with the colour from the top left pixel of the loaded texture.”

I tried to change the settings for the allocate FBO, texture settings related ofDisableArbTex or NormalizedTexCoords but problem did not resolve.

I think as there is a problem with the process of setUniformTexture the texture to the shader in iOS(OpenGL ES2). Please let me know if you have some people you know a solution.

Sorry I’m not good in English…

256 * 0.78 = ~200. Definitely sounds like a power of 2 issue somewhere. Perhaps all your fbos/textures are being allocated as POT?

I’m still seeing no difference after doing the following:

Update images/textures/fbos to power of two (256x256 in this case).
Set ofDisableArbTex().
Normalise the texCoords of the ofMesh I’m drawing to (using crazy numbers or removing textCoords altogether makes no difference)

You can download the project here:
http://jamesalliban.co.uk/storage/iosES2ShaderExampleSimple.zip

And the code is below.

testApp.h

  
  
#pragma once  
#include "ofMain.h"  
#include "ofxiOS.h"  
#include "ofxiOSExtras.h"  
  
class testApp : public ofxiOSApp{  
public:  
    void setup();  
    void draw();  
  
    ofShader shader;  
    ofFbo fbo;  
    ofImage image;  
    ofMesh mesh;  
};  
  

testApp.cpp

  
  
#include "testApp.h"  
  
void testApp::setup()  
{  
    ofBackground(60);  
    ofSetLogLevel(OF_LOG_VERBOSE);  
    ofDisableArbTex();  
      
    shader.load("shaders/SimpleShader.vert", "shaders/SimpleShader.frag");  
      
    //image.loadImage("image/cat_256.jpg");  
    image.loadImage("image/test_image_256.jpg");  
    image.setImageType(OF_IMAGE_COLOR);  
      
    ofFbo::Settings settings;  
    settings.width = image.width;  
    settings.height = image.height;  
    settings.internalformat = GL_RGB;  
    fbo.allocate(settings);  
      
    // set up mesh to draw to  
    mesh.addVertex(ofVec3f(0,0,0));  
    mesh.addVertex(ofVec3f(image.width,0,0));  
    mesh.addVertex(ofVec3f(0,image.height,0));  
    mesh.addVertex(ofVec3f(image.width,image.height,0));  
    mesh.addTexCoord(ofVec2f(0,0));  
    mesh.addTexCoord(ofVec2f(1,0));  
    mesh.addTexCoord(ofVec2f(0,1));  
    mesh.addTexCoord(ofVec2f(1,1));  
    mesh.addIndex(0);  
    mesh.addIndex(1);  
    mesh.addIndex(3);  
    mesh.addIndex(0);  
    mesh.addIndex(3);  
    mesh.addIndex(2);  
}  
  
void testApp::draw()  
{  
    ofTexture tex = image.getTextureReference();  
    int texID = image.getTextureReference().getTextureData().textureID;  
      
    fbo.begin();  
    ofClear(0, 0, 0);  
    shader.begin();  
    shader.setUniformTexture("tex0", tex, texID);  
    float normXPos = (float)ofGetMouseX() / (float)ofGetWidth();  
    float normYPos = (float)ofGetMouseY() / (float)ofGetHeight();  
    shader.setUniform1f("normXPos", normXPos);  
    shader.setUniform1f("normYPos", normYPos);  
      
    //printf("normXPos: %f, normYPos: %f \n", normXPos, normYPos);  
      
    //ofRect(0, 0, tex.getWidth(), tex.getHeight());  
    mesh.draw();  
      
    shader.end();  
    fbo.end();  
    fbo.draw(0, 0);  
      
    // draw texture beneath to point out the colour displayed when dragging   
    ofPushMatrix();  
    ofTranslate(0, image.height);  
    image.draw(0, 0, image.width * 0.5, image.height * 0.5);  
    ofPushStyle();  
    ofNoFill();  
    ofSetColor(255);  
    ofRect(ofMap(normXPos, 0, 1, 0, image.width * 0.5) - 2, ofMap(normYPos, 0, 1, 0, image.height * 0.5) - 2, 4, 4);  
    ofPopStyle();  
    ofPopMatrix();  
}