Sending images from python to openFrameworks

beginners
#1

I’m working on a project to generate images in python based on the viewer’s position found using a kinect in oF. I’ll transfer the coordinates from openFrameworks to python and send the corresponding image from python back to openFrameworks. So, looking at the specs for the sending the images:

  • Right now 200x200 RGB images, although could go up to 1000x1000 RGB
  • stored as pixels in python
  • there’s a one to one mapping between coordinates and resulting image
  • I want as high of a framerate as possible

Considering the above, the solutions I’ve looked into are:

  • OSC
  • TCP
  • FIFO
  • saving images locally and loading them with oF (this way image results could be cached)

Which solution would be best for my needs? Thanks!

#2

Why are you sending to python, considering that it generates these based on data coming from OF.
What would give you the best performance would definitely leave python out and do all the processing in OF, probably using a different thread (look at the ofThreadedChannelExample)

#3

I collect the data with OF but the images are generated with a GAN built using pytorch, so it unfortunately has to be generated with python.

#4

You can use spout for python

#5

I use:

  1. Spout for Python but use this lib for python 3.7 http://eps.here.ru/work/SpoutSDK37.zip
    Also I had to change this line in the spout receiver example to make it work.

spoutReceiver.pyReceiveTexture(receiverName, spoutReceiverWidth, spoutReceiverHeight, textureReceiveID, GL_TEXTURE_2D, False, 0)

must become:

spoutReceiver.pyReceiveTexture(receiverName, spoutReceiverWidth, spoutReceiverHeight, np.uintc(textureReceiveID).item(), GL_TEXTURE_2D, False, 0)

In my case the textureReceiveID needed to be casted from numpy to ‘regular’ python type.

  1. I also made this https://github.com/wearenocomputer/ofxZmqWindows and I used to send video to Python.
#6

I see. Although I think that the largest bottlenecks are the kinect framerate and the GAN itself, so I would not worry too much about the way to send it, as long as it is not absurd. I think that going for a TCP solution can give you the flexibility to move the GAN processing to a different computer, and have it networked. @bakercp might have a good idea on what would perform the best.
cheers

#7

All of this really depends on your requirements for size, latency, etc. Whether you are compressing / decompressing (e.g. jpg, png, etc) on either end, the method doesn’t matter as you’ll just be sending a byte array in the end.

If you want to send data back and forth as fast as possible and you don’t care too much about about ordering or a delivery guarantees, big UDP packets on a local network are very fast. OSC is UDP based, but does have some overhead and potentially limitations you probably don’t need.

If you require temporal and data continuity – and you can handle a little potential latency, TCP works well. On a local wired network, this would be extremely fast and reliable.

From the user side, TCP and UDP are pretty much the same. You send an array of bytes and then end up at the other side. If you need meta data, it will be encoded as header data in a compressed image (e.g. jpg) or you can make up your own packet format.

1 Like