GLSL Texture Delays -> aka. 'Video Delays'

Hi guys, I am trying really hard to implement a decent looking texture delay or video delay system. I have seen working examples of this working in VVVV and Jitter but I cant seem to work out how to implement it in oF.

I have included links to some examples below of this working so you can see the aesthetic I am going for. I would love to be able to get this working in GLSL, only to keep this effect as light and quick as possible but am open to other suggestions.

If anyone has any projects/code lying around that does something like this in oF or GLSL or could just give me some tips so I can keep moving forward that would be amazing.

The Multix code in ofxPlaymodes has a nice delay but the addon has too many interconnected dependancies for simple projects -> https://github.com/arturoc/ofxPlaymodes

In Jitter I used to use this, I tried to implement it in openFrameworks but to no avail -> http://cycling74.com/2009/04/06/the-video-processing-system-part-3/

But the one that looks really nice is this HLSL VVVV effect called echo, I downloaded the HLSL code and tried to port it to GLSL but its just a little bit too complicated for me right now -> http://vvvv.org/documentation/kalle.shader

Thanks!!

Just a cheeky bump with a video that just got released on Vimeo that is doing this effect probably the best i’ve seen it done.

-> https://vimeo.com/35770492

This kinda technique would be super useful for more than just live video, surely the oF community would benefit from this kind of technique… anyone?

This doesn’t seem super hard to do if you don’t need it to look really good. Make an FBO, render your video frame to it, then when you get a new frame that you want, maybe 0.3 seconds later, add the values for each pixel together and divide by two:

  
gl_FragColor = mix(vec4(tex1,1.0),vec4(tex1,1.0),0.5);  

Then render that.

The trick I think would be in finding a good algorithm for differencing the frames so that it looks crisp and smooth the way the video does. Something primitive would be like:

  
if(distance (vec3(tex1.xyz),vec3(tex1.xyz)) > 1.0)   
{  
    gl_FragColor = vec4(tex1,1.0);  
}  
else  
{  
    gl_FragColor = vec4(tex1,1.0);  
}  
  

hi !

we used to make something similar in real time some years ago … as here :

https://vimeo.com/5081638
(check from minute 2 on …)

what we find to be the “nicest” way to draw things like this is to use a blending mode based on “maximum” values …
for using “maximum” as blending mode you requiere to have a dark background … if you want to achieve this in a bright background then you would use “minimum” function instead.
Basically this blend trick chooses the brightest pixel … and the result blend is quite nice .

one way would be to use :

glBlendEquation(GL_MAX);
(by default is GL_ADD_FUNC)

and the nicest way nowadays is to use max function in GLSL …

i’m quite sure that what you see on that video is a MAX function working … that’s why the background is dark …

e*

PD : related with this … with Arturo we’ve been some time developping this : https://github.com/eloimaduell/ofxPlaymodes … right now we’re stuck on some issues , but if you can compile and run it, you can play with the feedback parameter and you’ll get this effect live with cost almost 0 … all GPU …

oops i missed that you already mentione the Multix effect that is how we call it …

i’m sure you can take off the ofxPlaymodes dependencies from here and use it for your purpose …
basically here you’ve to give the shader a pointer to the realtime image (imageSource in ofxPlaymodes terminology) and a pointer to the first “copy” …

  
#include "VideoFeedbackGPU.h"  
using Poco::ScopedLock;  
  
#define STRINGIFY(...) #__VA_ARGS__  
  
static string fragmentFeedbackSrc =  
#ifdef TARGET_LINUX  
		"#version 140\n"  
		"#extension GL_ARB_texture_rectangle : enable\n"  
#endif  
        STRINGIFY(  
		uniform sampler2DRect tex0;  
        uniform sampler2DRect tex1;  
		uniform float feedback;		    
		uniform float opacityIn;		    
        void main (void){  
            vec2 pos = gl_FragCoord.xy;  
  
            vec4 color = vec4(max(texture2DRect(tex0, pos).r* opacityIn,texture2DRect(tex1, pos).r*(feedback)),max(texture2DRect(tex0, pos).g*opacityIn,texture2DRect(tex1, pos).g*(feedback)),max(texture2DRect(tex0, pos).b*opacityIn,texture2DRect(tex1, pos).b*(feedback)),1.0);  
  
            gl_FragColor = color;  
        }  
        );  
  
namespace ofxPm{  
VideoFeedbackGPU::VideoFeedbackGPU()  
:source1(0)  
,source2(0)  
,newFrame(false)  
{  
	// TODO Auto-generated constructor stub  
  
}  
  
VideoFeedbackGPU::~VideoFeedbackGPU() {  
	// TODO Auto-generated destructor stub  
}  
  
void VideoFeedbackGPU::setup(VideoSource & _source1, VideoSource & _source2){  
	source1 = &_source1;  
	source2 = &_source2;  
	front = VideoFrame::newVideoFrame(_source1.getNextVideoFrame());  
	back = VideoFrame::newVideoFrame(_source1.getNextVideoFrame());  
	back.setTextureOnly(true);  
	ofAddListener(source1->newFrameEvent,this,&VideoFeedbackGPU::newVideoFrame);  
	shader.unload();  
	shader.setupShaderFromSource(GL_FRAGMENT_SHADER,fragmentFeedbackSrc);  
	shader.linkProgram();  
  
}  
  
void VideoFeedbackGPU::removeListener()  
{  
	ofRemoveListener(source1->newFrameEvent,this,&VideoFeedbackGPU::newVideoFrame);  
	//stopThread();  
}  
	  
	  
VideoFrame VideoFeedbackGPU::getNextVideoFrame(){  
	return front;  
}  
  
void VideoFeedbackGPU::newVideoFrame(VideoFrame & frame){  
	//front = VideoFrame::newVideoFrame(frame);  
  
	if(source2->getNextVideoFrame()==NULL){  
		ofNotifyEvent(newFrameEvent,front);  
		return;  
	}  
  
  
	back.getFboRef().begin();  
	shader.begin();  
	shader.setUniformTexture("tex0",frame.getTextureRef(),0);  
	shader.setUniformTexture("tex1",source2->getNextVideoFrame().getTextureRef(),1);  
	shader.setUniform1f("feedback",feedback);  
	shader.setUniform1f("opacityIn",inputOpacity);  
	ofRect(0,0,frame.getWidth(),frame.getHeight());  
	shader.end();  
	back.getFboRef().end();  
  
	front = VideoFrame::newVideoFrame(back);  
  
	ofNotifyEvent(newFrameEvent,front);  
}  
  
float VideoFeedbackGPU::getFps(){  
	return source1->getFps();  
}  
	  
void VideoFeedbackGPU::setFeedback(float f)  
{  
	feedback = f;  
}  
	  
	void VideoFeedbackGPU::setInputOpacity(float f)  
	{  
		inputOpacity = f;  
	}  
	  
}  
  

this is the VideoFeedbackGPU.cpp present on my github …

hope it helps …

e*

Awesome, thanks for the info guys. really appreciate it! I have been coding away and have made a fairly decent one using the GL_MAX function in oF but I will spend the next bit looking at porting that over to your shader Eloi that uses the max function in GLSL to compare the results.

Thanks!

Sorry for reviving this long dead topic but I thought I’d share my experience with this in case anyone comes along looking to do this. I put together an example in webGL a couple years ago but only just now got around to trying it in oF.

In my iteration I just keep a vector of past textures around and have a few different cyclical indices evenly spaced throughout the vector. You are limited to whatever your max texture slots are per shader (in my case 16), but 8 textures seems to be plenty to get a nice effect.

I’d also be curious about developing a more advanced blending mode, or perhaps just doing a weighted sum based on the age of the texture. I know the shader is also very branchy for the different modes, but it seems performant enough. I think the real disadvantage of doing it this way is running out of memory when you try to allocate memory for like anything greater than a couple hundred fbos.

Lastly, one more approach might be to use a 3d texture, though having never used them, I can’t say for certain if this would give any performance gains.

precision highp float;

uniform sampler2D u_image;
uniform sampler2D u_image2;
uniform sampler2D u_image3;
uniform sampler2D u_image4;
uniform sampler2D u_image5;
uniform sampler2D u_image6;
uniform sampler2D u_image7;
uniform sampler2D u_image8;

uniform int mode;

varying vec2 tc;


void main() {
    vec3 tex_one = vec3(texture2D(u_image, tc));
    vec3 tex_two = vec3(texture2D(u_image2, tc));
    vec3 tex_three = vec3(texture2D(u_image3, tc));
    vec3 tex_four = vec3(texture2D(u_image4, tc));
    vec3 tex_five = vec3(texture2D(u_image5, tc));
    vec3 tex_six = vec3(texture2D(u_image6, tc));
    vec3 tex_seven = vec3(texture2D(u_image7, tc));
    vec3 tex_eight = vec3(texture2D(u_image8, tc));
    
    
    vec3 outCol = mode == 0 ? min(tex_one, tex_two) : max(tex_one, tex_two);
    outCol = mode == 0 ? min(outCol, tex_three) : max(outCol, tex_three);
    outCol = mode == 0 ? min(outCol, tex_four) : max(outCol, tex_four);
    outCol = mode == 0 ? min(outCol, tex_five) : max(outCol, tex_five);
    outCol = mode == 0 ? min(outCol, tex_six) : max(outCol, tex_six);
    outCol = mode == 0 ? min(outCol, tex_seven) : max(outCol, tex_seven);
    outCol = mode == 0 ? min(outCol, tex_eight) : max(outCol, tex_eight);
    
    gl_FragColor = vec4(outCol,1.0);
}
int maxFrames;
int textureIndex;
vector<ofFbo> fbos;
int index1, index2, index3, index4, index5, index6, index7, index8;
ofVideoGrabber grabber;

ofApp::setup(){
    multixShader.load("shaders/multix");
    grabber.setup(w, h);

    maxFrames = 128;
    textureIndex = 0;

    for(int i = 0; i<maxFrames; i++){
        ofFbo f;
        f.allocate(w, h, GL_RGB);
        fbos.push_back(f);
    }


    int numSplits = 8;
    index1 = 0;
    index2 = maxFrames/numSplits * 1;
    index3 = maxFrames/numSplits * 2;
    index4 = maxFrames/numSplits * 3;
    index5 = maxFrames/numSplits * 4;
    index6 = maxFrames/numSplits * 5;
    index7 = maxFrames/numSplits * 6;
    index8 = maxFrames/numSplits * 7;
}
ofApp::draw(){
      fbos[textureIndex].begin();
               grabber.draw(0,0);
      fbos[textureIndex].end();

      multixShader.begin();
            multixShader.setUniformTexture("u_image", fbos[index1].getTexture(), 0);
            multixShader.setUniformTexture("u_image2", fbos[index2].getTexture(), 1);
            multixShader.setUniformTexture("u_image3", fbos[index3].getTexture(), 2);
            multixShader.setUniformTexture("u_image4", fbos[index4].getTexture(), 3);
            multixShader.setUniformTexture("u_image5", fbos[index5].getTexture(), 4);
            multixShader.setUniformTexture("u_image6", fbos[index6].getTexture(), 5);
            multixShader.setUniformTexture("u_image7", fbos[index7].getTexture(), 6);
            multixShader.setUniformTexture("u_image8", fbos[index8].getTexture(), 7);
            multixShader.setUniform1f("mode", 0.0);
                fbos[index1].draw(0,0);
        multixShader.end();

    index1 = (index1 + 1) % maxFrames;
    index2 = (index2 + 1) % maxFrames;
    index3 = (index3 + 1) % maxFrames;
    index4 = (index4 + 1) % maxFrames;
    index5 = (index5 + 1) % maxFrames;
    index6 = (index6 + 1) % maxFrames;
    index7 = (index7 + 1) % maxFrames;
    index8 = (index8 + 1) % maxFrames;

    textureIndex = (textureIndex + 1) % maxFrames;
}
2 Likes