Grabbing the "canvas" using ofFbo, ofTexture or ofImage and sending through ofxNDI


#1

Hi,

I’m trying to stream my application output using NDI on a mac inside xcode. I’m running of_v10 and I’ve tried different approaches and ofxNDI repositories on github. I’ll try to summarize the whole experience and I’ll hope to get some help to solve the problem.

Well, lets say I have a circle that moves inside the application screen, bouncing on the edges:

//setup()
x = ofGetWidth()/2;
y = ofGetHeight()/2;
vx = 5;
vy = 5;

// update()
x += vx;
y += vy;
if (x < 0) vx = 5;
if (y < 0) vy = 5;
if (x > ofGetWidth()) vx = -5;
if (y > ofGetHeight()) vy = -5;

// draw()
ofBackground(0);
ofSetColor(255);
ofDrawCircle(x, y, 50);

So, I want to send this frame using Newtek NDI to a remote application (mad mapper) through TCP/IP. I don’t even need to draw the circle inside my application window. I just need to send the frame (min 30 fps) to the remote application.

There are two main questions to solve this problem:

  1. How to get the “canvas” pixel array.
  2. How to send the frame using NDI (the method send() receives ofPixels &pixels).

In order to solve it, my first attempt was using https://github.com/fabianmoronzirfas/ofxNDI and “grab the screen” using ofFbo:

#include "ofApp.h"
#include "ofxNDISender.h"

int x, y, vx, vy;

ofxNDISender ndiSender;
ofFbo fbo;
ofPixels pixels;

//--------------------------------------------------------------
void ofApp::setup(){
    x = ofGetWidth()/2;
    y = ofGetHeight()/2;
    vx = 5;
    vy = 5;
    
    ndiSender.setMetaData("ofxNDISender Test 1", "send frames", "ofxNDI", "0.0.0", "", "", "");
    fbo.allocate(ofGetWidth(), ofGetHeight()), GL_RGB;
}

//--------------------------------------------------------------
void ofApp::update(){
    x += vx;
    y += vy;
    if (x < 0) vx = 5;
    if (y < 0) vy = 5;
    if (x > ofGetWidth()) vx = -5;
    if (y > ofGetHeight()) vy = -5;
    
    fbo.readToPixels(pixels);
    ndiSender.send(pixels);
}

//--------------------------------------------------------------
void ofApp::draw(){
    fbo.begin();
    ofBackground(0);
    ofSetColor(255);
    ofDrawCircle(x, y, 50);
    fbo.end();
    fbo.draw(0,0);
}

  • Problems:
    – I get
    [ error ] ofxNDISender:
    – It seems that the ofPixel format generated by ofReadToPixels() is not adequate
    – It is ultra-slow (maybe because it’s printing the error at the console)

Second attempt was still using fabianmoronzirfas/ofxNDI but grabbing the pixels using img.grabScreen

void ofApp::draw(){
    
    ofBackground(0);
    ofSetColor(255);
    ofDrawCircle(x, y, 50);
    
//This could be inside update, doesn't matter
    img.grabScreen(0, 0, ofGetWidth(), ofGetHeight()); 
    ndiSender.send(img.getPixels());
}

Problem:

  • Ultra slow;

I’ve also tried to use https://github.com/leadedge/ofxNDI but couldn’t get it running. Installed the NDI SDK, copied libs to addon/ofxNDI/libs/NDI/Libs as stated in readme. Got 20+ erros in ofxNDIutils.cpp

Last attempt was with thomasgeissl/ofxNDI but it was sending absolutely nothing though NDI. No errors, just pure silence.

Any ideas?


Network Device Interface (NDI) - Video over IP
#2

Grabbing screen pixel examples based in this topic: FASTEST way of reading FBO pixels or screen pixels, please read!


#3

Hi,
Does somthing like this work?

int w = ofGetWidth();
int h = ofGetHeight();

ofTexture tex;
tex.allocate(w, h, GL_RGBA);

tex.loadScreenData(0, 0, w, h);
ofPixels pixels;
  tex.readToPixels(pixels);
 

//Here send the pixels through HDI.


tex.clear();

#4

you might want to make some benchmarking in order to figure out where is the bottleneck. There is one in the textrure to pixels transfer, but there might be some others that can be worse.
I recommend you to use ofxTimeMeasurements which will allow you to measure easily the time it take for each chunk of code.
hope this helps


#5

Hi, Roy! Thanks for your answer!

I’ve tried also this approach but ofxNDI generates this error message:
[ error ] ofxNDISender: pixel type 4 is not supported yet

As I’ve stated before, it seems that the ofPixel format generated by ofReadToPixels() is not the adequate format. I can’t even benchmark it because it doesn’t work. I get the same error when I use ofFbo readToPixels(). The only approach that works is using ofImage through grabScreen() but even with this simple scenario it renders so slow that its not viable in a real application.

Any idea of how to get a specific ofPixelFormat out of readToPixels()?

Edit: It seems that img.getPixels() generate ofPixelFormat type 2 (supported). ofFbo and ofTexture format type 4 (not supported).


#6

I dont know, but maybe you get this error because the alpha channel? What about allocating the texture with GL_RGB


#7

Yeah, it did the trick of changing the ofPixelFormat, but I get 12FPS while sending the frame.

The ofxNDI example grabs a frame from the webcam using videoGrabber, sends it using getPixels() and you feel no lag at all (25~30fps). I can’t understand why ofxNDI has such distinct performance on both cases: sending a texture and sending a camera grabbed video frame.


#8

downloading a texture from the graphics card is really slow, using a pbo can hide the latency allowing the main thread to run in parallel while you download the texture to pixels. There’s some examples in the gl folder on how to use pbos and also some addons that make it easier to use them like https://github.com/NickHardeman/ofxFastFboReader


#9

I’ve imported ofxFastFboReader and tried to use it instead of default ofFbo readToPixels().

Considering the “bouncing ball” scenario:

ofxFastFboReader (sync mode): 11~13FPS;
ofxFastFboReader (async mode): 11~13FPS;
fbo.readToPixels(): 11~13FPS;

Not much of a difference… :confused:


#10

I think ofxFastFboReader does everything in the main thread which could slow down things a bit but it should be better than ofFbo::readPixxels

This example shows how to read from the pbo on a different thread which could accelerate things a bit: https://github.com/openframeworks/openFrameworks/tree/patch-release/examples/gl/threadedPixelBufferExample

Also check you are running in release mode which shouldn’t really make much difference in terms of downloading the texture but could make other things in the addon much faster

Also this addon https://github.com/arturoc/ofxTextureRecorder does PBO download using threads. is very specialized on saving to images but it could be modified to do something else


#11

Ok, Thanks! I’ll take a look on both links and get back soon…


#12

Ok, just to get it clear, the bottleneck is at ofxNDI ofxNDISender.cpp lines 91~99

for(int y = 0; y < pixels.getHeight(); y++){
			for(int x = 0; x < pixels.getWidth(); x++){
				auto index = (x*4 + y*pixels.getWidth()*4);
				_frame.p_data[index] = pixels.getColor(x, y).b;
				_frame.p_data[index+1] = pixels.getColor(x, y).g;
				_frame.p_data[index+2] = pixels.getColor(x, y).r;
				_frame.p_data[index+3] = 255;
			}
		}

More specifically at pixels.getColor()

_frame.p_data[index] = pixels.getColor(x, y).b;

The way data is stored or how it is read from a FBO is slow. I’ll get into Arturo’s links and threads but is there a faster way to access this data?


#13

yeah get color is super slow running in release mode should make it faster if you are not doing that already but also this should make it much faster:

int index = 0;
for(auto line: pixels.getLines()){
    for(auto pixel: line.getPixels()){
           _frame.p_data[index] = pixel[2];
           _frame.p_data[index+1] = pixel[1];
           _frame.p_data[index+2] = pixel[0];
           _frame.p_data[index+3] = 255;
           index += 4;
    }
}

#14

Yeah!!! I was already running in release mode, but after using getLines() and getPixels() it’s running 29~30FPS!

I still get a little delay (maybe 2 or 3 frames delay) but its already twice faster. I believe that this delay can only be minimized with threads, right?