OpenCV for Processing

bindings
#1

Howdy.

I’m working with Stéphane Cousot this week on integrating OpenCV into Processing. We were re-compiling OpenCV directly from the source when I remembered that OF has already done all this. Since we would like to keep OF & Processing as coherent as possible, we want to copy the OF+CV examples and library names for the Processing examples and library names. So it looks like we’re going to use the OF implementation as our starting blocks.

Any reason anyone can think of to avoid this strategy?

#2

no - by all means go for it, our aim is to wrap things in a simple way and if it’s useful for p5ers, cool. (although we would encourage them to try OF too for CV stuff)

if you do wind up using our wrappers, or writing code based on our wrapper, please just be sure to credit OpenFrameworks and the folks that debugged / tested it:

Zach Lieberman
Theo Watson
Carl-Johan Rosén
Stefan Hechenberger
Chris Sugrue
Chris O’Shea

(anyone else I forget to mention? Please jump into this thread - my bad)

if you have questions feel free to post…

thanks for checking!
zach

#3

Ok, for the credits.

We’ve been having some troubles compiling for JNI. Something about dynamic libraries, can’t remember all the details, it’s early :shock: :shock:

#4

yeah the jni stuff is tricky
ben fry showed me once but i couldn’tget my head around getting data back and forth in a simple way…

the problem you mention is covered in the forum, see the sticky (top) post in forum > extend

http://forum.openframeworks.cc/t/bugs-with-opencv-addon-w-of-0.03±,-readme-/329/0

and:

http://forum.openframeworks.cc/t/computervision-weirdness/271/0

basically, once we added destructors for everything (like ofTexture) some pass by copy c++ default behavior was tripping us up… we’ve posted the fixed code on that sticky link, and we’ve fixed the default equals and copy behavior on the svn for ofTexture and ofImage which should also alleviate the issue.

take care!
zach

#5

Great. I didn’t see that, thanks. It looked like a classic dangling pointer problem but I couldn’t figure out how, since it didn’t crash, and just gave the white frames. Now I see why, it’s the way you draw() in OF. I’m only just now really starting to work with your frameworks. I’ll be using them a lot more in the second semester when we start exploring it more officially in my atelier.

As for JNI it’s not all that bad once you’ve banged your head a few times against the wall. Also Stéphane’s a pro so I’m a little spoiled. So far we’ve figured out the JNI transfer back and forth. Processing stores data in a single int, whereas openCV stores them as 3 successive unsigned char, so we have to convert everything (ugh) as well as convert the ints to jints via JNI, etc. It makes me a little concerned for speed. On the question of speed, I was wondering why you had so many pass-by-copies in there. Usually one passes pointers when dealing with video, in order to maintain speed. You can’t do that with Java (obviously) but you should be able to do it in C++. Maybe I’m missing something?

Another question (I’m starting to nag here): have you tried the native OpenCV grabber instead of the Quicktime/VfW ones? I’m thinking of functions like: cvGrabFrame(), cvRetrieveFrame(), etc. which are designed for working with video quickly. It seems to me that passing video from Quicktime to OpenCV might slow things down a bit.

Yet another question: I don’t understand why you do a memcopy on each line of the video frame, as opposed to copying the whole block. I’m thinking explicitly of this line:

  
for (int i = 0; i < height; i++){  
		memcpy(pixels + (i * width * 3), cvImage->imageData + (i * cvImage->widthStep), width*3);  
}

Again, I’m sure I just haven’t understood something.

Anyway, as for JNI. We’ve figured it all out, and we’re compiling fine using the sourceforge CVS, so it’s now just a question of making the JNI wrappers. OpenFrameworks has been great for that part of the project. Even if we won’t use all the code directly (obviously), it’s definitely a part of the process.

But OpenCV is so huge, it’s hard to know where to start. Contours is a great start, face tracking is probably a good next step, but there’s so much in there. Any suggestions on what you were planning next?

#6

hi, some answers:

a) the pass by copy was a mistake. please look at the sticky post I sent you too, it’s all explained there in italics and in the zip linked there **we have recoded it as pass by reference **. You can download that zip and copy that code in. is that clear or do you need me to explain it again another time, in more detail :slight_smile: ?

b) we need to use memcpy because opencv images do not have contiguous memory - and we get data in and out as contiguous memory. For example, if you have an image that is 30 pixel by 30 pixels, internally, opencv might store it as:

32 pixels wide x 30 pixels

this is because the step size needs to be a multiple of 16, etc. since we thought folks would have trouble with that, we created functions that help. We have done that also for ofImage, ofVidepPlayer (quicktime based), since it’s not the esiest thing to do and since our ofTexture expect contiguous memory

sometimes, we hve chose to write code which is more helpful then optimized, as in this case. but honestly, the penalty in c++ to copy a blck of pixels is not that huge. if you want to take that out, do so - we don’t care - but then your users will have to know about “step size” being different then width, which is not always easy to explain or implement to the end user.

c) we don’t care about opencv’s movie loading functionality because we have our own movie loader. I don’t think the penalty is that bad. This is c++ and it doesn’t hurt to pass pointer around, etc. opencv is doing the same thing anyway, so I’m not sure what difference it makes. Second, we haven’t implemented all of opencv because - well - life is short. In a way that would be kind of ridiculous - we don’t, for example, rewrite all of quicktimes functionality. We give you access to the stuff we think you need (play a movie, grab video) in a way that is easy to use and clear. We’ve created some helper classes to get people access - like the ability pass around images, a way of doing operations on images (ie, a class which wraps two opencv images as one seamless, since many operations need a second image to pass the results in to) and do contour detection. If you want to add face detection, there is code in the forums about how to do that. if you want to wrap other stuff, be our guest and have fun.

hope that helps -
zach

#7

Great answers to all my questions. All clear now.

I agree on simplifying the basics and letting others dig deeper for whatever else they need. Unfortunately, due to JNI, we’re going to have to map everything people want to do. We can’t just connect up the library. Ugh.

-Douglas

#8

have you thought about JNA as opposed to JNI ?

https://jna.dev.java.net/

it’s supposedly much much better and easier to get started with !

-z

#9

Oh holy crap, indeed we really should have gone that route. I’ll talk about it with Stéphane. Thanks for the tip.

I should mention that we’ve ended up really abandoning (for the moment) code compatibility with OpenFrameworks. Our function call names turned out to be so much easier using the Processinging naming style (updatePixels, image, etc). I absolutely understand some of the OF choices for names, but it made the Processing .pde examples almost unreadable. I’ll see if I can’t reformulate some of these observations at a later date. These are mostly about coding style, but also the overall accessibility of your project – which is frikkin’ brilliant by the way, I thought I’d just remind you of that ;-).

#10

antojames,

please post only those related to the forum topic. I’m not a forum admin, but I love OF (my business and consulting depends on OF) so I do my best to keep things tidy, not just for my own reading but for others as well as a sign of respect. tnx…