WebCam frame rate not set by VideoGrabber


#1

Hi,

I am new to open frameworks. Based on the suggestions from the forums, I choose OfxVideorecorder to record data from built camera of my Mac to an mp4 file. I have also added “GUI” buttons to start / stop my recording. Apparently, even after using setDesiredFrameRate of the ofVideoGrabber, I see lag in my webcam.

Here is my myApp.h file:

#pragma once

#include "ofMain.h"
#include "ofxVideoRecorder.h"
#include "ofxGui.h"
#include "PlayVideos.h"

class ofApp : public ofBaseApp{

	public:
		void setup();
		void update();
		void draw();

		void keyPressed(int key);
		void keyReleased(int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased(int x, int y, int button);
		void mouseEntered(int x, int y);
		void mouseExited(int x, int y);
		void windowResized(int w, int h);
		void dragEvent(ofDragInfo dragInfo);
		void gotMessage(ofMessage msg);
    
    ofVideoGrabber webcam;
    ofxVideoRecorder recorder;
    ofxPanel gui;
    ofxButton startRecording;
    ofxButton stopRecording;
    ofxButton playRecordedVideos;
    
    void record();
    void finishRecording();
    void play();
    
    bool videoSetupComplete;

    int index;
    PlayVideos playObj;
    
		
};

Here is my myApp.cpp file:

#include <sstream>
#include <string>
#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    
    index = 0;
    videoSetupComplete = false;
    
    startRecording.addListener(this, &ofApp::record);
    stopRecording.addListener(this, &ofApp::finishRecording);
    playRecordedVideos.addListener(this, &ofApp::play);
    
    
    // setup the gui
    gui.setup();
    gui.add(startRecording.setup("Start Recording"));
    gui.add(stopRecording.setup("Stop Recording"));
    gui.add(playRecordedVideos.setup("Play Recordings"));
    
    //set the app framerate to 60
    ofSetFrameRate(60);
    
    ofSetLogLevel(OF_LOG_VERBOSE);
    
    // set the webcame framerate to 30
    webcam.listDevices();
    webcam.setDeviceID(0);
    webcam.setDesiredFrameRate(30);
    webcam.setup(1280, 720);
    
}

void ofApp::record(){

    videoSetupComplete = false;
    
    ostringstream str1;
    str1 << index;
    
    std::string filename = str1.str() + ".mp4";
    
    std::cout<< "Starting to record ";
    
    // record at 30 fps
    recorder.setup(filename, webcam.getWidth(), webcam.getHeight(), 30);
    
    // start the recorder
    recorder.start();
    
}

void ofApp::finishRecording(){
    
    std::cout<< "Finished Recording ";
    
    index = index + 1;
    recorder.close();
    
}

void ofApp::play() {
    
    //close the webcam
    webcam.close();
    
    playObj.setup();
    //playObj.draw();
    
    videoSetupComplete = true;
}

//--------------------------------------------------------------
void ofApp::update(){
    
    webcam.update();
    
    // add frame to the recorder
    if(webcam.isFrameNew()){
        bool success = recorder.addFrame(webcam.getPixels());
        if (!success) {
            ofLogWarning("This frame was not added!");
        }
    }
    
    // Check if the video recorder encountered any error while writing video frame or audio smaples.
    if (recorder.hasVideoError()) {
        ofLogWarning("The video recorder failed to write some frames!");
    }
    
    if (videoSetupComplete) {
        playObj.update();
    }
    
    
}

//--------------------------------------------------------------
void ofApp::draw(){
    gui.draw();
    webcam.draw(100, 100);
    
    if (videoSetupComplete) {
        playObj.draw();
    }
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){
    
}

//--------------------------------------------------------------
void ofApp::keyReleased(int key){
}

//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y ){
    
}

//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button){
    
}

//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button){
    
}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button){
    
}

//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y){
    
}

//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y){
    
}

//--------------------------------------------------------------
void ofApp::windowResized(int w, int h){
    
}

//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg){
    
}

//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo){
    
}


Below is the console response of my app:

Starting to record [verbose] execThread: starting command: bash --login -c 'ffmpeg -y -an -r 30 -s 1280x720 -f rawvideo -pix_fmt rgb24 -i "/Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/ofxvrpipe0" -r 30  -vcodec mpeg4 -b 2000k -acodec pcm_s16le -ab 128k "/Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/0.mp4"' &
[verbose] execThread: command completed successfully.
[verbose] Recordi[nvge.r
b
ose] ofxVideoDataWriterThread: opening pipe: /Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/ofxvrpipe0
ffmpeg version 4.0 Copyright (c) 2000-2018 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.39.2)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.0 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --enable-videotoolbox --disable-lzma
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
[warning] ofxVideoDataWriterThread: got file descriptor 11
Input #0, rawvideo, from '/Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/ofxvrpipe0':
  Duration: N/A, start: 0.000000, bitrate: 663552 kb/s
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1280x720, 663552 kb/s, 30 tbr, 30 tbn, 30 tbc
Please use -b:a or -b:v, -b is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
Press [q] to stop, [?] for help
Output #0, mp4, to '/Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/0.mp4':
  Metadata:
    encoder         : Lavf58.12.100
    Stream #0:0: Video: mpeg4 (mp4v / 0x7634706D), yuv420p, 1280x720, q=2-31, 2000 kb/s, 30 fps, 15360 tbn, 30 tbc
    Metadata:
      encoder         : Lavc58.18.100 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1
frame=    4 fps=0.0 q=2.0 size=     256kB time=00:00:00.10 bitrate=20961.4kbits/s speed=0.175x    
frame=    6 fps=5.3 q=5.1 size=     256kB time=00:00:00.16 bitrate=12580.1kbits/s speed=0.147x    
frame=    8 fps=4.7 q=10.5 size=     256kB time=00:00:00.23 bitrate=8986.8kbits/s speed=0.136x    
frame=   10 fps=4.4 q=16.3 size=     256kB time=00:00:00.30 bitrate=6990.2kbits/s speed=0.132x    
frame=   12 fps=4.2 q=17.0 size=     256kB time=00:00:00.36 bitrate=5719.4kbits/s speed=0.129x    
frame=   14 fps=4.1 q=17.5 size=     256kB time=00:00:00.43 bitrate=4839.7kbits/s speed=0.127x    
frame=   16 fps=4.1 q=18.9 size=     256kB time=00:00:00.50 bitrate=4194.5kbits/s speed=0.127x    
frame=   18 fps=4.1 q=23.8 size=     256kB time=00:00:00.56 bitrate=3701.1kbits/s speed=0.128x    
frame=   20 fps=4.0 q=26.0 size=     256kB time=00:00:00.63 bitrate=3311.5kbits/s speed=0.128x    
frame=   22 fps=4.0 q=25.6 size=     256kB time=00:00:00.70 bitrate=2996.2kbits/s speed=0.128x    
Finished Recording [verbose] ofxVideoDataWriterThread: closing pipe: /Users/nitkatya/Kepler/of_v0.9.8_osx_release/apps/myApps/camRecorder/bin/data/ofxvrpipe0
frame=   23 fps=3.8 q=24.6 size=     256kB time=00:00:00.73 bitrate=2860.0kbits/s speed=0.122x    
frame=   23 fps=3.7 q=24.6 Lsize=     476kB time=00:00:00.73 bitrate=5319.5kbits/s speed=0.117x    
video:475kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.196626%

It seems like my frame rate is very low. How do I fix this?


#2

Hi, how do you notice lag?

The encoding is happening at less than real time, because of the fps=0.0 fps=5.3 fps=4.7 fps=4.4 .... But that should not be an issue.

If the light conditions are poor, the webcam may run at a lower frame rate, but if the produced video is fixed at 30 fps, the video will play too fast. If that is happening to you, you could run the program also at 30 fps (instead of 60) and then add frames to the recorder even if webcam.isFrameNew() is not true. That way you add 30 frames per second to the video, even if some frames are repeated, to keep the original timing.

Some video recorders allow specifying the timestamp for each video frame to make sure the playback time is the same as the original (in iOS for instance) even if the source frame-timing is irregular, as it may be with a webcam.

Is that the issue you are noticing?


#3

Thanks for the response.

The frame rates were dropped because of the poor lighting condition. Didn’t knew about that.