Best way to communicate data from OF and a node browser app using tf.js?

beginners
#1

I am using a pose estimation model from tensorflow.js that takes a camera stream and estimates my ‘pose’ then draws a pose skeleton on top of the camera feed in a browser. What I want to do is take the coordinates of each keypoint and stream them over to my OF app for further processing

PoseNet: https://github.com/tensorflow/tfjs-models/tree/master/posenet Try a Demo of it here.

Im sadly not too familiar with web development and all the different communication protocols. At first I thought I could try and make the browser a virtual serial device and send the data over serial, but didnt look like that was going to work.

I thought maybe I could use OSC and send the data over UDP but It looks like there is something significant that blocks browsers from sending/receiving UDP packets?

Im looking into websockets right now and the approach looks quite complex. I just wanted to know if there were any reliable AND low-latency communication options that I could use before I decided to dive into websockets?

Thanks for any help!

#2

As far as I know, Websockets is the fastest when dealing with a browser, it’s not instant but in most cases it’s good enough. Have a look at https://github.com/robotconscience/ofxLibwebsockets it has a lot of examples and is pretty straight forward to integrate in your OF project, you would setup your OF app as a Websocket server.

On the client side (browser) you need to implement some javascript to send your data over to the server. There is a basic example in /ofxLibwebsockets/example_server_echo/bin/data/web which can get you started. You could probably send all the coordinates as a string separated by spaces (or any other separator).

Good luck !

1 Like
#3

hello,
usually websockets work quite well and i think it is the way to go.
websockets are very well supported across the major browsers, and nodejs. they are easy to use in the web. it might be a bit harder to get them running in c++. ofxLibWebsocket has not been updated for a while and might not run out of the box.

if you are more looking for a pub/sub solution, dont mind running a communication server, then i can recommend mqtt. it runs on top of websocket or tcp and you need a mqtt broker.

also check out Chris’ addons, e.g.

there is also an electron app which stream postnet data via osc:

hope that helps

1 Like