How to manipulate an image using the mesh vertices you've detected?

I have detected a mesh of vertices (joined by OF_PRIMITIVE_LINES) based on the luminosity of each pixel.

I am animating this mesh by adding a simple jitter movement. In the screenshot below, I have placed the original image behind the mesh so you can also see the background.

When the jitter movement increases you can start seeing the lines of mesh. What I would like to do instead is use the vertices in the mesh point to locations on my image and manipulate the pixels on the image (or texture of my image?)

I am reviewing the OpenGL basics but meanwhile it would be great if you can point me to the right direction to achieve this.


Hi there!

I would like to help you, but first, can you explain a bit better what is your objective?

Are you trying to do a glow/jitter effect? Either way, why are creating a mesh if what you want to do is manipulate the pixels directly?

Hi, I’ll be glad to explain :smile:

My objective to create animations in the image mapped to sound. I am interested in detecting interesting elements/objects in the image and then apply some effects to them jitter/glow/bulge/flow/spiral/etc/etc that will make sense to a listener. This is what I have been able to achieve so far. (click to see the gif)

I think I am using the mesh to segregate the objects of interest so I can apply effects to those areas but my mesh is separate from the image. Thus, this question.

Also, do you think I should look into some other methods to achieve this? (OpenCV?)

Thank you so much for helping me out!

I see… Well, this is quite a complex question, since it envolves a lot of steps and possible solutions. So I will try to give you a few suggestions…

Regarding the detection of interesting elements, openCV is a possibility. But I believe it’s more focused on clear features like smiles, faces, contours, et cetera.

Since you are starting already with abstract paintings, you could define different criteria. Just like when you when with luminosity, maybe tint, kernel area, contrast, and so on. Another ideia would be, prior to playing around in OF, creating masks for your images (in Photoshop or similiar), and then use those masks to play around with your image.

About the mesh and texture manipulation, and since you want to play with both in realtime, if your window size starts to get a bit larger, you’ll probably go out gas pretty soon. With you go only with CPU, of course.

So I would advise you to go GPU and check this tutorial. Especially the Textures as Data (e.g. Displacement). But instead of using a noise texture, you could create an ofFloatImage, based on sound input.

That’s a treasure trove of info! Thank you so much!
I’m gonna hop right on and try these and will get back to you right here :slight_smile: