Offline Video Rendering

Hi!

I’m looking for an alternative to matlab for video processing. I’ve tried OpenCV and it works fine for this, but OF seems to be better designed, has a larger community and has better options if I want to expand into interactive real-time stuff or graphics.

OF seems to be mostly geared toward real-time applications however, and I can’t figure out how to process each frame in a video sequentially. This is the simple code I’ve written that is supposed to just read each frame in a video and save them:

void ofApp::setup()
{
	video.load("video.MP4");
	idx = 0;
}
void ofApp::update()
{
	idx++;
	if (idx == video.getTotalNumFrames())
	{
		OF_EXIT_APP(0);
	}
	video.setFrame(idx); // ofVideoPlayer::nextFrame() doesen't seem to work.
	video.update();

	ofImage img(video.getPixels());
	img.saveImage("test_" + std::to_string(idx) + ".png");
}

However, this just saves empty black images. video.getCurrentFrame() reports that the correct frame is set on the video object. If I call video.play() in ofApp::setup(), the images are correctly saved but it just selects whatever frame is ‘current’ in the real-time video stream, which results in tons of frames being skipped in between.

Is there any way of sequentially iterating through each frame in a video or is OF just the wrong tool for this kind of thing?

the default videoplayer in windows and osx doesn’t have options to process video in offline. if you are in linux or using the ofxGStreamer addon: https://github.com/arturoc/ofxGStreamer you can set a special mode to read the video frame by frame every time you call update on it.

Once you have included ofxGStreamer in your application (not needed in linux) just do:

#include ofGstVideoPlayer.h"

void ofApp::setup(){
    auto player = std::make_shader<ofGstVideoPlayer>();
    player.setFrameByFrame(true);
    video.setVideoPlayer(player);
}

Now everytime you call update on video (which is a normal ofVideoPlayer) it’ll grab the next frame

I found the solution, I just needed to pause the video.

void ofApp::setup()
{
	video.load("video.MP4");
	video.setPaused(true);
}
void ofApp::update()
{
	video.update();
	if (video.getIsMovieDone())
	{
		OF_EXIT_APP(0);
	}
	
	frame = (video.getPixels());
	frame.saveImage("test_" + std::to_string(video.getCurrentFrame()) + ".png");

	video.nextFrame();
}

However, if I downscale the frames from 4k to something like 360p, it sometimes dumps duplicate frames, but new frames are still in the correct place in sequence. I suspect that this is because writing to disk becomes much quicker which doesen’t allow ofVideoPlayer to fetch new pixel data quickly enough, so it just uses the data from previous successful fetch. Doing this seems to solve it:

void ofApp::setup()
{
	video.load("video.MP4");
	video.setPaused(true);
}
void ofApp::update()
{
	video.update();
	if (video.getIsMovieDone())
	{
		OF_EXIT_APP(0);
	}
	
	if (video.isFrameNew())
	{
		frame = (video.getPixels());
		frame.saveImage("test_" + std::to_string(video.getCurrentFrame()) + ".png");

		video.nextFrame();
	}
}

I suspect that doing all of this in ofApp::setup would be better for this kind of thing.

As i said in the previous post osx and windows won’t work for offline video, if you use a lower resolutuion it might be that changing a frame is fast enough so when you call getPixels the frame you asked for is already there but there’s really no warranties

Sorry, I was writing that reply before I saw your post. I’ll try ofxGStreamer, thanks for the suggestion.

The video I’m reading from is still 4k, i’m just downscaling the ofImage before saving it.

I finally got it to compile, neither the projectGenerator or the VS 2017 plugin included the gstreamer paths correctly, even though my environment variables are set correctly. When running the program nothing happens however, I get “exited with code -1073741819” when I allocate the ofGstVideoPlayer object.

Edit: Got it to work now. Importing via VS2017 plugin, then set the GSTREAMER_1_0_ROOT_X86_64 system variable ‘correctly’ (/ instead of \ to match what the importer outputs), then correcting the ‘%3b’ output in additional dependencies, then adding the \libs\openFrameworks\video\ofGst*.cpp files to the project. The projectGenerator basically tried to find every single dependency using relative path in addons\ofxGStreamer instead of the gstreamer root folder.

The only problem now is that the first 2-3 frames are usually jumbled in some way.

Just another idea but you can also use video software such as Adobe Premiere, finalcut, QuickTime pro 7, Compressor, etc. I normally use QuickTime pro 7 on osx for these conversion from video to image sequence. And then load images into oF and overlay another graphical element, render again, …

The program is just a simple test to see whether or not oF is able to process each video frame in sequence, the end goal is not to create a image sequence from the video. Your suggestion would circumvent the hassle of using video players though, but I’ve got it working now. Working with image sequences would just be an unnecessary extra step in the workflow.