Accessing cv::VideoCapture from OpenFrameworks

georgeprofenza · July 18, 2012, 10:16pm

Hello,

I’m working on a basic cv app using an IP camera (unfortunately it’s not an Axis one).
I’ve started in pureC++ with OpenCV and have successfully got my stream using the VideoCapture class.

Now I’d like to use that stream in oF using ofxCv, but am having issues with linked libraries:

  
  "_CVPixelBufferUnlockBaseAddress", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  ".objc_class_name_QTCaptureDecompressedVideoOutput", referenced from:  
  
  
      literal-pointer@__OBJC@__cls_refs@QTCaptureDecompressedVideoOutput in opencv.a(cap_qtkit.o)  
  
  
  "_QTMediaTypeVideo", referenced from:  
  
  
      _QTMediaTypeVideo$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMediaTypeVideo$non_lazy_ptr)  
  
  
  "_QTMovieNaturalSizeAttribute", referenced from:  
  
  
      _QTMovieNaturalSizeAttribute$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieNaturalSizeAttribute$non_lazy_ptr)  
  
  
  "_QTMovieDurationAttribute", referenced from:  
  
  
      _QTMovieDurationAttribute$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieDurationAttribute$non_lazy_ptr)  
  
  
  "_CVBufferRetain", referenced from:  
  
  
      -[CaptureDelegate captureOutput:didOutputVideoFrame:withSampleBuffer:fromConnection:] in opencv.a(cap_qtkit.o)  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  "_QTMovieFrameImageTypeCVPixelBufferRef", referenced from:  
  
  
      _QTMovieFrameImageTypeCVPixelBufferRef$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieFrameImageTypeCVPixelBufferRef$non_lazy_ptr)  
  
  
  "_QTMovieFrameImageType", referenced from:  
  
  
      _QTMovieFrameImageType$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieFrameImageTypeCVPixelBufferRef$non_lazy_ptr, _QTMovieFrameImageType$non_lazy_ptr )  
  
  
  ".objc_class_name_QTMovie", referenced from:  
  
  
      literal-pointer@__OBJC@__cls_refs@QTMovie in opencv.a(cap_qtkit.o)  
  
  
  ".objc_class_name_QTCaptureDevice", referenced from:  
  
  
      literal-pointer@__OBJC@__cls_refs@QTCaptureDevice in opencv.a(cap_qtkit.o)  
  
  
  "_QTFormatDescriptionVideoCleanApertureDisplaySizeAttribute", referenced from:  
  
  
      _QTFormatDescriptionVideoCleanApertureDisplaySizeAttribute$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTFormatDescriptionVideoCleanApertureDisplaySizeAttribute$non_lazy_ptr)  
  
  
  "_QTMovieCurrentTimeAttribute", referenced from:  
  
  
      _QTMovieCurrentTimeAttribute$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieCurrentTimeAttribute$non_lazy_ptr)  
  
  
  "_CVBufferRelease", referenced from:  
  
  
      -[CaptureDelegate captureOutput:didOutputVideoFrame:withSampleBuffer:fromConnection:] in opencv.a(cap_qtkit.o)  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  ".objc_class_name_QTCaptureSession", referenced from:  
  
  
      literal-pointer@__OBJC@__cls_refs@QTCaptureSession in opencv.a(cap_qtkit.o)  
  
  
  "_CVPixelBufferLockBaseAddress", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  "_CVPixelBufferGetHeight", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  "_CVPixelBufferGetBaseAddress", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  "_kCVPixelBufferWidthKey", referenced from:  
  
  
      _kCVPixelBufferWidthKey$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _kCVPixelBufferWidthKey$non_lazy_ptr)  
  
  
  "_kCVPixelBufferPixelFormatTypeKey", referenced from:  
  
  
      _kCVPixelBufferPixelFormatTypeKey$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _kCVPixelBufferPixelFormatTypeKey$non_lazy_ptr)  
  
  
  ".objc_class_name_QTCaptureDeviceInput", referenced from:  
  
  
      literal-pointer@__OBJC@__cls_refs@QTCaptureDeviceInput in opencv.a(cap_qtkit.o)  
  
  
  "_CVPixelBufferGetWidth", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
  "_kCVPixelBufferHeightKey", referenced from:  
  
  
      _kCVPixelBufferHeightKey$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _kCVPixelBufferHeightKey$non_lazy_ptr)  
  
  
  "_QTMovieLoopsAttribute", referenced from:  
  
  
      _QTMovieLoopsAttribute$non_lazy_ptr in opencv.a(cap_qtkit.o)  
  
  
     (maybe you meant: _QTMovieLoopsAttribute$non_lazy_ptr)  
  
  
  "_CVPixelBufferGetBytesPerRow", referenced from:  
  
  
      CvCaptureFile::retrieveFramePixelBuffer()       in opencv.a(cap_qtkit.o)  
  
  
      -[CaptureDelegate updateImage] in opencv.a(cap_qtkit.o)  
  
  
ld: symbol(s) not found  
  
  
collect2: ld returned 1 exit status

Unfortunately I’m not very experienced with c++, but as far as I can tell
the compiler looks in opencv.a for VideoCapture and I presume the class isn’t there.
I’ve made a separate group and copied all the OpenCv 2 dylib files I have in /opt/local/lib,
but am still getting the errors

How can I can I use VideoCapture with oF and ofxCv ?

Thanks!

georgeprofenza · July 19, 2012, 3:05pm

Ok,

I’ve figured out that the VideoCapture class is in the highgui library.
I’ve tried to recompile OpenCV 2.4.0 from source for i386, but for VideoCapture
to work with the IP camera OpenCV needs to be build with ffmpeg.

I’ve tried that but I have the 64bit version of ffmpeg. I’ve downloaded the ffmpeg sources
and tried to compile but there are other dependencies that also need to be compiled for i386.

I just want to grab a frame from an IP camera and this whole 32bit/64bit hassle is causing more
problems than expected. Any hints/tips ?

georgeprofenza · July 19, 2012, 8:50pm

Based on this thread (http://forum.openframeworks.cc/t/07-64-bit-build–/7195/15) I’ve managed get get ofxCv working with it as well. Also, I’ve removed the reference to opencv_highgui.a from ofxOpenCv and added my own libopencv_highgui.2.4.1.dylib compiled with ffmpeg support to stream from the IPCamera.

To display the image(cv::Mat) I’ve used a conversion function from ofxCv Utilities to store into ofPixels then plugged that into an ofImage:

  
  
//image is cv::Mat, pix's type is ofPixels and frame's type is ofImage  
toOf(image,pix);  
frame.setFromPixels(pix.getPixels(), 640, 384, OF_IMAGE_COLOR, false);

Is this the recommended/fastest way to deal with this ?
Also, in most ofxCv examples there is an isFrameNew() call - how would I deal with this in my case ?

gregab · July 23, 2012, 4:54pm

You could also just pass the ofImage’s ofPixelsRef directly into toOf(), like this:

toOf(img, frame.getPixelsRef());
frame.update();

You could use frame.getPixelsRef() anywhere you’re currently using that separate ofPixels object to eliminate the intermediary.

georgeprofenza · July 23, 2012, 5:14pm

Cool!

Thanks, this seems tidier and I wouldn’t need to pass width/height (which might change with a different stream).

In other ofxOpenCv examples I noticed isFrameNew() calls to either the grabber or the video player.
Currently I do a check like so:

  
  
if(!image.empty()){  
  toOf(image,frame.getPixelsRef());  
  frame.update();  
  //etc  
}

Do I need to do anything extra ? Is this how I check if there are new pixels ?

Thanks!

gregab · July 23, 2012, 6:43pm

Are you using your IP camera with OF’s ofVideoGrabber? If so, you can use grabber.isFrameNew() to determine if there are new pixels.

georgeprofenza · July 23, 2012, 7:02pm

No, I’m using loading an H264 stream from an IP Camera via RTSP using OpenCV’s VideoCapture class.

Before doing the image.empty() check I fetch a new frame via VideoCapture’s read() method