ofxGstreamer on M1 Mac

Huh wow - I might have found a fix for the video player at least.
Try changing this line:

from:

GstElement * gstPipeline = gst_element_factory_make("playbin","player");

to:

GstElement * gstPipeline = gst_element_factory_make("playbin3","player");

I am going to do some more testing on Linux and will update the relevant Github but I was getting the same errors on macOS with ofxGStreamer and this change fixed it.

More on playbin3 here. I think an error I got when I made the player threaded pointed to an incomplete graph which playbin3 ( which auto builds graphs might address )

1 Like

@theo I tried your fix but still no luck on my end. Just to be sure we’re on the same page here is what I’m running and the error I get for ofGstVideoPlayer.

I think the issue I’m facing for ofxGstreamer is related to ofGstUtils. I commented out a bunch of things and traced the errors back to this line:

From what I can tell the gstreamer pipelines are ok, I tried a bunch of combinations and ideas pulled from searching around until I was able to stop the gstreamer specific errors. My bet is something to do with matching pad templates + capabilities before it gets to OF to be in the correct codex/format. (Basic tutorial 6: Media formats and Pad Capabilities). The line where I hit errors is related to the callback functions so maybe the data isn’t ready to be accessed when we’re hitting it with these functions? I’m a bit deeper into the source than I really understand but I’ve been commenting & googling my way through the errors. Thoughts? Ideas to try next?

@nlavella - thanks for this reply!

EDIT: so turns out I had 1.18 when it worked. I was messing around with versions and that is why it worked fine for me. 1.20+ is totally broken and playerbin3 has not affect. Sorry for the confusion.

Weird that playbin3 fixed it for me. I think maybe the only difference is that I used the GStreamer 1.20.3 Installer ? And maybe 1.21.0 had a change that broke that.

Either way it doesn’t work for Linux and so I think I just got lucky there.

There is a Github thread that we have been using to discuss the issue here:

The gst_app_sink_set_callbacks you mentioned is interesting as I think when I was messing with Ubuntu and using export GST_DEBUG=4 to get error messages ( It mentioned something about callbacks were no longer thread safe and seem related to that line. )

I think you can get the error messages on macOS by adding:

setenv("GST_DEBUG", "4", 1);

at the top of ofApp::setup to see the extra messages.

@arturo was suggesting it is something with the code not support hardware decoding and that now hardware decoding might be the default so there might be something with the callback or caps to add hardware decoding support?

There is def more discussion on this though here:

One thing that could be helpful @nlavella is if we compare the following output from the simple app:

//--------------------------------------------------------------
void ofApp::setup(){
    
    setenv("GST_DEBUG", "4", 1);

    player.setPlayer(ofPtr<ofGstVideoPlayer>(new ofGstVideoPlayer));
	player.loadMovie("fingers.mov");
	player.play();
}

//--------------------------------------------------------------
void ofApp::update(){
	player.update();

}

//--------------------------------------------------------------
void ofApp::draw(){
	player.draw(0,0);
}

Here is a gist of a woking gstreamer app output that renders video ( Gstreamer 1.18 ):

@theo bummer that was 1.18 but makes sense. setenv("GST_DEBUG", "4", 1); didn’t work right away, I’m troubleshooting but any ideas?

@theo some success!

In trying to track through the callback issue I noticed it was being setup that way because of the bFramebyFrame variable. I flipped this from false to true and the 1.21.0 stream came through on my M1 Mac! this line

So I imagine there are reasons this was disabled but I ran a test on a fresh install and it is 100% the blocker, at least for ofGstUtils. I didn’t see anything obvious in ofGstVideoPlayer that was comparable but if you know if something happy to test on that end.

I’m thinking there will be performance or other issues but this at least gives us a good place to start!

3 Likes

Yay @nlavella - that’s awesome!
It definitely points to the callbacks as the issue.

I also found out via some debug stuff that both (working) 1.18 and (broken) 1.20 are using hardware decoding so I don’t think it is that in the end.

Will dig in more, thanks for helping track this down!!

2 Likes

More progress :slight_smile:

Try moving the callbacks block under the switch (ret) statement.
( you can set bFramebyFrame to false now )

    if(isAppSink){
		ofLogVerbose("ofGstUtils") << "startPipeline(): attaching callbacks";
		// set the appsink to not emit signals, we are using callbacks instead
		// and frameByFrame to get buffers by polling instead of callback
		g_object_set (G_OBJECT (gstSink), "emit-signals", FALSE, "sync", !bFrameByFrame, (void*)NULL);
		// gst_app_sink_set_drop(GST_APP_SINK(gstSink),1);
		// gst_app_sink_set_max_buffers(GST_APP_SINK(gstSink),2);

		if(!bFrameByFrame){
			GstAppSinkCallbacks gstCallbacks;
			gstCallbacks.eos = &on_eos_from_source;
			gstCallbacks.new_preroll = &on_new_preroll_from_source;
#if GST_VERSION_MAJOR==0
			gstCallbacks.new_buffer = &on_new_buffer_from_source;
#else
			gstCallbacks.new_sample = &on_new_buffer_from_source;
#endif
			gst_app_sink_set_callbacks(GST_APP_SINK(gstSink), &gstCallbacks, this, NULL);
		}
	}

Just above these lines:

	// wait for paused state to query the duration
	if(!isStream){
		bPlaying = true;
		bLoaded = true;
	}

Now I just get a crash on exit which looks similar but I think is again thread related.

Edit:

Commenting out this line fixes the crash on exit but I see a similar crash when the video ends playback ( looping set to false ). So I think there is a threading issue relating to the callbacks.

void ofGstUtils::close(){
	if(bPlaying){
		if(!bIsMovieDone && !bPaused && !isStream){
			std::unique_lock<std::mutex> lck(eosMutex);
			closing = true;
			gst_element_send_event(gstPipeline,gst_event_new_eos());
//			if(eosCondition.wait_for(lck,std::chrono::milliseconds(5000))==std::cv_status::timeout){
//				ofLogWarning("ofGstUtils") << "didn't received EOS in 5s, closing pipeline anyway";
//			}
			closing = false;
		}
	}
	stop();
1 Like

Hey what did you use for this? I’d like to learn but not sure where to start. I’d like to be able to work on problems like this someday.

@TimChi

I set this at the top of ofApp::setup
setenv("GST_DEBUG_DUMP_DOT_DIR", "/Users/theo/Documents/CODE/PROJECTS/OF_0.11.2/apps/myApps/gstreamerTest/bin/gstDump", 1);

The folder gstDump needs to exist.

Then I added this call to the end ofGstUtils::startPipeline ( before the return )
GST_DEBUG_BIN_TO_DOT_FILE((GstBin *)(gstPipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline");

That produces a .dot file which you can turn into a pdf which shows the graph structure.

You can also add to ofApp::setup:
setenv("GST_DEBUG", "4", 1);
To set the level of debug messages printed out by gstreamer. 4 is quite overwhelming and 3 is more critical stuff.

2 Likes

Solved the crash on exit issue with using playbin3 ( this time it works!! ) along with the other changes mentioned above.

Have a wip PR @nlavella - let me know if it works for you?
Hoping this fixes it for Linux too but it seems to be the same issue.

2 Likes

yay, progress! still get a coredump, but the app window now shows up and briefly plays the videos! Sometimes the video plays for a couple of seconds, sometimes I only get the first frame before the example crashes - but it seems this is definitely the right direction! :partying_face:

edit: just realized: it crashes as soon as my mouse moves across the window. If I start the player with the mousepointer outside the app window, the video plays until the end & only then crashes.

This is on Fedora 36 with Intel Iris Xe embedded graphics - I’ll give this a shot on another laptop with discrete NVidia graphics as well.

(btw - would you prefer I comment here, or on the pull request?)

Ah yes - I get the crash with the mouse too. I think it is because the example is setting the speed with mouseMovement. Nice that it the same on mac and linux :slight_smile:

Btw- I think this approach might be a lot more stable for you, but my goal is to get the callback based system working again

Thanks!
Theo

Yes - consistency is good. Even if it’s a consistent crash :wink:
I’ll be following your progress closely. Please let me now if there is anything I can do to help - I have several versions of Fedora on NVIDIA & Iris XE-based hardware available for testing!
Thanks for digging into this issue!

Video playback now works on my Debian 11 Testing machine with Intel graphics. I also get a Seg fault when moving the mouse over the window.

Thanks a lot @theo for all your work on solving this!

I have GStreamer 1.20.3.

@theo thanks for pulling together a fix! Mixed results with gstreamer 1.21.0 & M1 Mac:

ofGstVideoGrabber fix works! Video plays through and no crash on exit with the bare bones example we were originally testing.

ofGstUtils fix still crashes for me with the same errors we were originally facing even with the new callback location. Enabling bFrameByFrame still does the trick for now but callbacks are still going to need something.

Hoping to dig in a bit more over the weekend to see if I can get a little deeper into the issue.

1 Like

These issues seem relevant on their issue tracker.
Says it was merged in for 1.20.3 but I wonder if there is another similar bug that wasn’t caugh which our usage is exposing:

The crash I am getting now seems directly related to the flushing command when doing setSpeed / setPositon etc

Hi all,

thanks for the effor. I wonder whether I am doing something wrong or missing a modification in the ofGstUtils. I get the following with the simplified video Player example with the change proposed by @theo:

[warning] ofGstVideoUtils: update(): ofGstVideoUtils not loaded

no video is shown.

by the way I am in fedora 35 and Gstreamer 1.20.0.

thanks

Posting this update here too.
This PR / Fix works for me on Ubuntu 22 and macOS with GStreamer 1.20.3

More notes here.

2 Likes

Hey way to go Theo! That is a pretty involved patch. I do wonder if this bug is on the gstreamer side. Thanks for working so hard on this. I’m sure it required a lot of effort.