Videos Playing out of Sync

Hi,

I am trying to play two locally stored, overlaid 4K videos in synchronization. However, I am experiencing a slight lag (2-3 frames) in one of them. Can someone kindly suggest me how I can achieve perfect sync.

My code is as follows:

void ofApp::setup(){
  ofBackground(255,255,255);
  ofSetVerticalSync(true);
  
  // HQ video
  movieHQ.load("horses.mp4");
  movieHQ.setLoopState(OF_LOOP_NORMAL);

  // LQ video
  movieLQ.load("horseslq.mp4");
  movieLQ.setLoopState(OF_LOOP_NORMAL);

  // Setting speeds equal 
  movieHQ.setSpeed(1.0);
  movieLQ.setSpeed(1.0);
  movieLQ.play();
  movieHQ.play();

  // videoSection is of type ofPixels
  videoSection.allocate(600, 300, OF_PIXELS_RGB);
 
  // these are defined as global var
  x_pos = 0;
  y_pos = 0;

}

//--------------------------------------------------------------
void ofApp::update(){
  ofPixels & pixels = movieHQ.getPixels();

  // Croping a fixed sized rectangle from HQ video with starting point as current mouse pointer position 
  pixels.cropTo(videoSection, x_pos, y_pos, 600, 300);

  // Taking entire 4K pixels of LQ video
  ofPixels & morepixels = movieLQ.getPixels();

  // Putting HQ pixel rectangle into LQ pixel 4K frame
  videoSection.pasteInto(morepixels, x_pos, y_pos);

  // videoTexture is of type ofTexture
  videoTexture.loadData(morepixels);
  movieLQ.update();
  movieHQ.update();  
}

//--------------------------------------------------------------
void ofApp::draw(){
	videoTexture.draw(0, 0, 3840, 2160);
}
//--------------------------------------------------------------
// updates global variables x_pos and y_pos with current mouse positions
void ofApp::mouseMoved(int x, int y ){
	if ((x_pos < 3840) && (x_pos >= 0))
		x_pos = x;
	else
		x_pos = 3840;
	if ((y_pos < 2160) && (y_pos >= 0))
		y_pos = y;
	else
		y_pos = 2160;
}

Typically adding keyframes will help syncing in h.264/mp4 files but at the expense of file size. However it’s still pretty unreliable

I often do another cheat, make a new movie with both together (this only works if you are not recording the movies in the app but have them before), so if it was 2 4K movies I would make a 7680*2160 file with one full movie on the left and one on the right and then draw the subsections or extract the pixels if you need. This way the sync is 100% locked. However, choosing another codec can help as well, try something with less compression like prores (you will get large file sizes but that is the tradeoff), and depending on your platform I have had great performance increases using the the Hap codec, more so using ofxDSHapVideoPlayer on windows.

Hey,

Thank you for the suggestions.

I managed to hack it together and play in sync.

I pre-processed the videos to make one video from two (eg. two 3840x2160 videos = one 3840x4320 video) and then use subsection command to display the desired videos portions.

Code is as follows:

void ofApp::setup(){
       // This prepossessed video is a combined video of upscaled LQ and HQ ie. 3840x4320 pixels
	video.load("horses.mp4");
	video.setLoopState(OF_LOOP_NORMAL);
	video.play();
	video.firstFrame();
	ofPixels & half = video.getPixels();
	videoFrame.allocate(half);
}

//--------------------------------------------------------------
void ofApp::update(){
	video.update();
}

//--------------------------------------------------------------
void ofApp::draw(){
        // fullFrame constitutes 3840x4320 pixels
        // ((0,0),(3840,0),(0,2160),(3840,2160)) --> HQ
        // ((0,2160),(3840.2160),(0,4320),(3840,4320)) --> LQ
	ofPixels & fullFrame = video.getPixels();

        // logic to adjust the width and height of overlaid HQ video as per pointer position
	int width = (x_pos + 1000 < 3840) ? 1000 : 3840 - x_pos;
	int height = (y_pos + 800 < 2160) ? 800 : 2160 - y_pos;
       
	videoFrame.loadData(fullFrame);
        // Drawing only LQ video subsection on the screen out of fullFrame
	videoFrame.drawSubsection(0, 0, 3840, 2160, 0, 2160);
        // Drawing a subsection from HQ part of fullFrame on top of LQ  
	videoFrame.drawSubsection(x_pos, y_pos, width, height, x_pos, y_pos);
}
//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y ){
	if ((x >= 0) && (x < 3840))
		x_pos = x;
	if ((y >= 0) && (y < 2160))
		y_pos = y;
}

Do let me know if someone finds a more optimal way of going about it.

Cheers,
Ashutosh