Developing "ofxVolumetrics"

[TimS]

Update - many changes since I last posted in this thread.
https://github.com/timscaffidi/ofxVolumetrics

*increased rendering quality
*simplified, optimized raycasting shader, fixed type casting issues
*fixed near and far clipping issues, clipping planes match external scene regardless of FBO resolution
*rendering from within the volume is now possible
*fixed face culling issue, automatically detects and handles both right and left handed coordinate systems and properly culls faces
*added Xcode project files
*example now uses ofEasyCam

*made a video of it in action:
http://vimeo.com/35720138

UPDATE

OK, I am done with the first version. Included is an example that renders the head seen below.

Let me know what you think and what changes you’d like to see!

I’m learning git, so I’ve moved this addon to my git repo, seems like this will make everything easier.

https://github.com/timscaffidi/ofxVolumetrics

recent changes:
* updated shader with Joshua Noble’s compatibility fixes.
* Fixed a strange freezing issue when rendering at certain angles, seemed like the shader was getting into a very long/infinite loop in rare occasions.

/UPDATE

I am in the process of cleaning up my volumetric renderer to release as an addon. Since I’m using 3d textures I figured I’d include an ofxTexture3d class to make things more modular and reusable.

Since ofTexture is designed for 2d only, it doesnt really make sense for me to just make a subclass, or at least I have trouble thinking about how to go about that properly, so instead I’m making a standalone class which mimics much of the interface that ofTexture uses.

While implementing my loadData functions, I saw this in ofTexture.cpp:

  
  
	/*if(glFormat!=texData.glType) {  
		ofLogError() << "ofTexture::loadData() failed to upload format " <<  ofGetGlInternalFormatName(glFormat) << " data to " << ofGetGlInternalFormatName(texData.glType) << " texture" <<endl;  
		return;  
	}*/  
	  
	if(w > texData.tex_w || h > texData.tex_h) {  
		ofLogError() << "ofTexture::loadData() failed to upload " <<  w << "x" << h << " data to " << texData.tex_w << "x" << texData.tex_h << " texture";  
		return;  
	}  
	  
	// update our size with the new dimensions  
	texData.width = w;  
	texData.height = h;  
  

I see that there used to be an error when trying to load data of a different glType, however that code is now commented out, and the new type overrides the old one. Also, as long as the new width and height are <= the current width and height, those are overwritten as well.
The image data is uploaded using glTexSubImage2d, may or may not replace the entire texture. The texture data on the gfx card is then essentially being cropped? Does openGL not really care if you upload data as a different type? For example if i allocate the texture using GL_RGB and then load data as GL_LUMINANCE using smaller dimensions than originally allocated, what happens to the data that was not updated? does it become unusable because they are in different formats?

I guess I am asking this so that I know weather or not to even allow loading a different format of data into an existing 3d texture, because it is useful to use glTexSubImage3d to only update a slice of the volume, yet still keep the entire texture’s dimensions the same. I would think that if you tried to upload data in a different format in this case, you may get some strange output.

Well, thorugh writing this post, I think I’ve come to the conclusion that I should not allow different formats than the allocated one, but maybe I’m wrong. I’ll post back here if I run into more problems/design issues for this addon.

[arturo]

so the idea is that when you create the texture you specify the internal format but when you upload data you specify the format which are different things, check the signatures of the allocate and loadData methods.

for example, you can specify the internal format as GL_RGB but then upload data as GL_BGR. probably there should be a check so you can’t upload types that are not the same size but we realize about this shortly before doing the release so we just removed the check.

[TimS]

Ah, I thought it was something like that. I’m not sure how I missed it. For volumetrics I’ll have to restrict the internal format and the data format to only the ones using an alpha channel.

The addon is coming along smoothly. I haven’t run into any real problems, so I should be able to release an initial version soon.

[TimS]

Ok I have another question. From a usability standpoint, it would be really cool if I could make it so you could do a bunch of transforms in your draw routine–rotates, translates, scale–and then call myVolume.draw(x,y,z,w,h,d) and it would just place the volume in the scene at the right point. The problem is I’m using an FBO to render the volume which needs to be drawn to the screen in a 2d compositing sort of way. So, how can I make the transforms done outside of the FBO carry into the FBO, and furthermore, how can I then load the default OF perspective to draw the FBO to the screen, then go back to the transformed view when the function returns?

Is this even possible? I have seen ways to save and load transformation matrices, which sounds promising, but I don’t know how to do it. Is there a way to reset the viewport to the default oF view, then draw the FBO and then reload the transformed matrix? This way it would be like calling any other shape.draw() function, it would do what you want and leave the transformations the way they were before it was called.

Here’s a nice preview image using the head.

Right now I am passing in a few parameters for rotation and zoom and handling everything inside the FBO, but I don’t like the interface at all.

[joshuajnoble]

Couldn’t you just copy the modelView (and maybe projection) like:

  
  
	GLfloat	modl[16];  
	glGetFloatv( GL_MODELVIEW_MATRIX, modl );  
  

setupScreenOrtho() to reset everything, then set it back like:

  
  
glLoadMatrixf(modl);  
  

That might be wrong, but I foggily remember this was how I did something similar a while ago.

[TimS]

Thanks Joshua! Works like a charm, this is a much more intuitive way of rendering now. Just a few more tweaks and I can post the first version

[TimS]

OK, so it is working, however after the draw function returns, I can’t seem to get anything else to draw at all. Does ofSetupScreenOrtho() wipe out the matrix stack or something?
With ofSetupScreenOrtho() in there, the FBO draws perfectly flat to the screen as it should, but then the ofDrawBitmapString I draw after drawVolume does not show up.
if I comment out that line, then ofDrawBitmapString shows up, but the fbo is drawn transformed.

I’m doing this in draw:

  
  
void testApp::draw()  
{  
    background.draw(0,0,ofGetWidth(),ofGetHeight());  
    ofSetColor(255,255,255,255);  
   
    ofPushMatrix();  
    ofTranslate(ofGetWidth()/2, ofGetHeight()/2, -100);  
    ofRotateX(mouseY*-360/(float)ofGetHeight());  
    ofRotateZ(mouseX*360/(float)ofGetWidth());  
   
    myVolume.drawVolume(0,0,0, ofGetHeight()*0.75, 0);  
    ofPopMatrix();  
   
    ofSetColor(255,255,255,255);  
    ofDrawBitmapString("volumeDimensions:\t" + ofToString(myVolume.getVolumeWidth()) + "x" + ofToString(myVolume.getVolumeHeight()) + "x" + ofToString(myVolume.getVolumeDepth()) + "\n" +  
                       "FBO resolution:\t" + ofToString(myVolume.getRenderWidth()) + "x" + ofToString(myVolume.getRenderHeight()) + "\n" +  
                       "XY/Z Quality (q/Q, z/Z):\t" + ofToString(myVolume.getXyQuality()) + "/" + ofToString(myVolume.getZQuality()) + "\n" +  
                       "Threshold (t/T):\t" + ofToString(myVolume.getThreshold()) + "\n" +  
                       "Density (d/D):\t" + ofToString(myVolume.getDensity()) + "\n",20,20);  
}  
  

and this in drawVolume() (simplified):

  
  
void ofxVolumetrics::drawVolume(float x, float y, float z, float w, float h, float d, int zTexOffset)  
{  
    updateRenderDimentions();  
  
    ofVec3f cubeSize = ofVec3f(w, h, d);  
  
    GLfloat modl[16];  
    glGetFloatv( GL_MODELVIEW_MATRIX, modl );  
    GLfloat proj[16];  
    glGetFloatv( GL_PROJECTION_MATRIX, proj );  
  
    /* render backface */  
    fboBackground.begin();  
    ofClear(0,0,0,0);  
  
    //load matricies from outside the FBO  
    glMatrixMode(GL_MODELVIEW);  
    glLoadMatrixf(modl);  
  
    // render the backface  
  
    fboBackground.end();  
  
    /* raycasting pass */  
    fboRender.begin();  
    volumeShader.begin();  
    ofClear(0,0,0,0);  
  
    //load matricies from outside the FBO  
    glMatrixMode(GL_MODELVIEW);  
    glLoadMatrixf(modl);  
  
    // do some volume rendering stuff  
  
    volumeShader.end();  
    fboRender.end();  
  
    ofPushMatrix();  
    glColor4f(1,1,1,1);  
    ofSetupScreenOrtho(ofGetWidth(), ofGetHeight(),OF_ORIENTATION_DEFAULT,false,0,1000);  
    fboRender.draw(0,0,ofGetWidth(),ofGetHeight());  
    ofPopMatrix();  
    // at this point I would expect the matrices to be the same as they were before entering the function  
}  
  

[TimS]

OK nevermind. I figured it out. ofSetupScreenOrtho doesn’t screw up the matrix stack, but it does modify both the modelview and projection matrices, so If you push both of them, it works!

  
  
    glMatrixMode(GL_MODELVIEW);  
    ofPushMatrix();  
    glMatrixMode(GL_PROJECTION);  
    ofPushMatrix();  
      
    glColor4f(1,1,1,1);  
    ofSetupScreenOrtho();//ofGetWidth(), ofGetHeight(),OF_ORIENTATION_DEFAULT,false,0,1000);  
    fboRender.draw(0,0,ofGetWidth(),ofGetHeight());  
  
    glMatrixMode(GL_PROJECTION);  
    ofPopMatrix();  
    glMatrixMode(GL_MODELVIEW);  
    ofPopMatrix();  
  

Almost done…

[TimS]

OK, I am done with the first version. Included is an example that renders the head seen above.

Let me know what you think and what changes you’d like to see!

http://timothyscaffidi.com/files/of/ofxVolumetrics.tar.gz

[joshuajnoble]

Nice work! I had to make some tweaks to the fragment shader to get it compile, just int to float stuff, nothing major.

  
  
#extension GL_ARB_texture_rectangle : enable  
varying vec4 pos;  
uniform sampler2DRect backface;  
uniform sampler3D volume_tex;  
uniform vec3 vol_d;  
uniform vec2 bg_d;  
uniform float zoffset;  
uniform float quality;  
uniform float threshold;  
uniform float density;  
  
void main()  
{  
  
    vec3 start = vec3(gl_TexCoord[0]);//  // the start position of the ray is stored in the texturecoordinate  
    vec4 back_position = texture2DRect(backface, (((pos.xy / pos.w)+1.) / 2.)*bg_d);//texture2DRect(backface, texc);  
    vec3 dir = vec3(0,0,0);  
    dir.x = back_position.x - start.x;  
    dir.y = back_position.y - start.y;  
    dir.z = back_position.z - start.z;  
    float len = length(dir.xyz);  
    float steps = len * length(vol_d.xyz) * quality;  
    float stepsize = len / steps;  
    vec3 delta_dir = normalize(dir) * stepsize;  
    float delta_dir_len = length(delta_dir);  
    vec3 vec = start;  
    vec4 col_acc = vec4(0,0,0,0);  
    float alpha_acc = 0.0;  
    vec4 color_sample;  
    float alpha_sample;  
    float aScale =  density * (1.0/quality);  
  
    //raycast  
    for(int i = 0; i < int(steps); i++)  
    {  
        color_sample = texture3D(volume_tex,vec+vec3(0.0,0.0,zoffset/vol_d.z));  
        if(color_sample.a > threshold) {  
            color_sample.a *= aScale;  
            col_acc.rgb = col_acc.rgb * col_acc.a + color_sample.rgb * color_sample.a * (1.0 - col_acc.a);  
            col_acc.a += color_sample.a * (1. - col_acc.a);  
            col_acc.rgb/=col_acc.a;  
            if(col_acc.a >= 1.0) {  
                break; // terminate if opacity > 1  
            }  
        }  
        vec += delta_dir;  
    }  
  
    // export the rendered color  
	gl_FragColor = col_acc;  
  
}  
  
  

[matthewpierreradford]

Hey there!

I work in the AAA videogame field and am currently developing a real time raymarching system. I have this bottle neck when it comes to 3d textures. Creating them is actually weirdly difficult. There is a plugin for photoshop by Nvidia which lets you export DDS textures and in theory supports the volume texture, but seems to be broken and unable to export them with more than one slice (though… maybe i haven’t found the magic button somewhere)

I’m curious about the Texture3D class you’ve created for your application. My goal is to essentially create 3d textures of high quality noise and export them as a DDS 3d texture (which our game engine and DirectX and OpenGL all support). Did you come up with a solution for exporting your 3D data or are you purely loading existing 3d textures and ray marching them ?

also cool to see this happening in OF, for the future you might want to implement diffuse lighting! It’s quite simple, we can chat about it. Finally another raymarching geek i can converse with!

Matt R