ofxGstreamer on M1 Mac

Greetings - I’m setting up a new machine for a project which requires using ofxGstreamer to access IP camera video feeds for openCV tracking.

I have a working code base from two years ago but am attempting to convert everything over to work on a new Mac Studio with the M1 Ultra chip.

Has anyone had any luck getting ofxGstreamer to work on an M1 Mac? Here are the steps I’ve taken so far and info I’ve found, still unable to get a stream to run but I feel like it’s close! Some of this is a bit above my skillset but happy to help troubleshoot and publish a solution if anyone has a better grasp on the final steps to get this over the finish line.

Thanks!

  1. Confirmed gstreamer works with IP cameras & m1 chip via homebrew installation and “gst-launch-1.0” CLI which was able to display feed
  2. After reviewing this thread (Installing ofxGstreamer dependencies on OS X with Homebrew) I removed homebrew and attempted a direct install from gstreamer (Download GStreamer)
  3. Installed 1.20.x to support ARM architecture which got me pretty far before receiving this same error from the gstreamer binary as seen here (Installing ofxGstreamer dependencies on OS X with Homebrew - #6 by batchku)
  4. Dug around for a bit to find this post (Broken macOS build of 1.20.0 on gitlab.freedesktop.org (binaries contain links to /Users/matt/Projects/cerbero) (#362) · Issues · GStreamer / cerbero · GitLab) outlining the issue and discerned it would be best to build the binary local on my machine to fix the path issue which includes "Matt"s machine reference
  5. Following this documentation I was able to build the binary local and get past the path issue when building my OF project (Building from source using Cerbero)
  6. Still hitting errors as I try to open a stream BUT the app builds

At this point I think it could be a few things:

  • The instructions for building the binary local were for the next, currently unstable, release of gstreamer (1.21). I attempted to build 1.20 but it won’t run the full script, not sure if it is a device issue or from having build 1.21 already.
  • I only installed gstreamer-1.0 & gstreamer-1.0-devel (fyi you need -devel package to fix gst/gst.h OF error) but it built a ton more packages. I tried also installing them all which didn’t do it so went back to the core two packages.
  • Maybe gstreamer is working correctly and the next step would be to get ofxGstreamer to be compatible with gstreamer 1.20.x?

Any advise or thoughts are greatly appreciated! I’ll continue to document here incase anyone else is also fighting with this one.

Hi @nlavella , have a look at the following about gstreamer 1.20 (for linux); you may want to stick with gstreamer 1.18.

  1. gstreamer 1.20.0 breaks video on Linux · Issue #6871 · openframeworks/openFrameworks · GitHub
  2. Segmentation fault on examples/computer_vision/opencvExample - #15 by theo
  3. Va-api ofVideoPlayer error intel and nvidia Linux - #6 by bernard_Del

Thanks @TimChi, the issue here is 1.18 is not compatible with the M1 chip, see screenshot.

Filling in some more background on a fresh test with gstreamer 1.21.0. I’m able to get the OF app to build but I error out once I set a pipline (correction, fails on startPipeline()) in the setup with this error:

@nlavella - could you share where the Thread 1 is at the point it crashes?

I am curious where in the main thread code it is as it could point to a need for a mutex or something else to make things thread safe.

We are having issues currently with GStreamer > 1.16, but solving it for macOS / ofxGstreamer could lead to a fix for all the other issues we are having.

Thanks!
Theo

@theo is this what you’re after? Happy to share more and run tests as best I can!

Thanks yes!!
In this case it doesn’t really provide any clues, but I might try replicating here as its much easier for me to debug in Xcode vs Linux with Makefiles and Valgrind :slight_smile:

For sure! Let me know if there are any test or ideas I can run down. I’m typically not this deep in the source but happy to give it a try!

Huh wow - I might have found a fix for the video player at least.
Try changing this line:

from:

GstElement * gstPipeline = gst_element_factory_make("playbin","player");

to:

GstElement * gstPipeline = gst_element_factory_make("playbin3","player");

I am going to do some more testing on Linux and will update the relevant Github but I was getting the same errors on macOS with ofxGStreamer and this change fixed it.

More on playbin3 here. I think an error I got when I made the player threaded pointed to an incomplete graph which playbin3 ( which auto builds graphs might address )

1 Like

@theo I tried your fix but still no luck on my end. Just to be sure we’re on the same page here is what I’m running and the error I get for ofGstVideoPlayer.

I think the issue I’m facing for ofxGstreamer is related to ofGstUtils. I commented out a bunch of things and traced the errors back to this line:

From what I can tell the gstreamer pipelines are ok, I tried a bunch of combinations and ideas pulled from searching around until I was able to stop the gstreamer specific errors. My bet is something to do with matching pad templates + capabilities before it gets to OF to be in the correct codex/format. (Basic tutorial 6: Media formats and Pad Capabilities). The line where I hit errors is related to the callback functions so maybe the data isn’t ready to be accessed when we’re hitting it with these functions? I’m a bit deeper into the source than I really understand but I’ve been commenting & googling my way through the errors. Thoughts? Ideas to try next?

@nlavella - thanks for this reply!

EDIT: so turns out I had 1.18 when it worked. I was messing around with versions and that is why it worked fine for me. 1.20+ is totally broken and playerbin3 has not affect. Sorry for the confusion.

Weird that playbin3 fixed it for me. I think maybe the only difference is that I used the GStreamer 1.20.3 Installer ? And maybe 1.21.0 had a change that broke that.

Either way it doesn’t work for Linux and so I think I just got lucky there.

There is a Github thread that we have been using to discuss the issue here:

The gst_app_sink_set_callbacks you mentioned is interesting as I think when I was messing with Ubuntu and using export GST_DEBUG=4 to get error messages ( It mentioned something about callbacks were no longer thread safe and seem related to that line. )

I think you can get the error messages on macOS by adding:

setenv("GST_DEBUG", "4", 1);

at the top of ofApp::setup to see the extra messages.

@arturo was suggesting it is something with the code not support hardware decoding and that now hardware decoding might be the default so there might be something with the callback or caps to add hardware decoding support?

There is def more discussion on this though here:

One thing that could be helpful @nlavella is if we compare the following output from the simple app:

//--------------------------------------------------------------
void ofApp::setup(){
    
    setenv("GST_DEBUG", "4", 1);

    player.setPlayer(ofPtr<ofGstVideoPlayer>(new ofGstVideoPlayer));
	player.loadMovie("fingers.mov");
	player.play();
}

//--------------------------------------------------------------
void ofApp::update(){
	player.update();

}

//--------------------------------------------------------------
void ofApp::draw(){
	player.draw(0,0);
}

Here is a gist of a woking gstreamer app output that renders video ( Gstreamer 1.18 ):

@theo bummer that was 1.18 but makes sense. setenv("GST_DEBUG", "4", 1); didn’t work right away, I’m troubleshooting but any ideas?

@theo some success!

In trying to track through the callback issue I noticed it was being setup that way because of the bFramebyFrame variable. I flipped this from false to true and the 1.21.0 stream came through on my M1 Mac! this line

So I imagine there are reasons this was disabled but I ran a test on a fresh install and it is 100% the blocker, at least for ofGstUtils. I didn’t see anything obvious in ofGstVideoPlayer that was comparable but if you know if something happy to test on that end.

I’m thinking there will be performance or other issues but this at least gives us a good place to start!

3 Likes

Yay @nlavella - that’s awesome!
It definitely points to the callbacks as the issue.

I also found out via some debug stuff that both (working) 1.18 and (broken) 1.20 are using hardware decoding so I don’t think it is that in the end.

Will dig in more, thanks for helping track this down!!

2 Likes

More progress :slight_smile:

Try moving the callbacks block under the switch (ret) statement.
( you can set bFramebyFrame to false now )

    if(isAppSink){
		ofLogVerbose("ofGstUtils") << "startPipeline(): attaching callbacks";
		// set the appsink to not emit signals, we are using callbacks instead
		// and frameByFrame to get buffers by polling instead of callback
		g_object_set (G_OBJECT (gstSink), "emit-signals", FALSE, "sync", !bFrameByFrame, (void*)NULL);
		// gst_app_sink_set_drop(GST_APP_SINK(gstSink),1);
		// gst_app_sink_set_max_buffers(GST_APP_SINK(gstSink),2);

		if(!bFrameByFrame){
			GstAppSinkCallbacks gstCallbacks;
			gstCallbacks.eos = &on_eos_from_source;
			gstCallbacks.new_preroll = &on_new_preroll_from_source;
#if GST_VERSION_MAJOR==0
			gstCallbacks.new_buffer = &on_new_buffer_from_source;
#else
			gstCallbacks.new_sample = &on_new_buffer_from_source;
#endif
			gst_app_sink_set_callbacks(GST_APP_SINK(gstSink), &gstCallbacks, this, NULL);
		}
	}

Just above these lines:

	// wait for paused state to query the duration
	if(!isStream){
		bPlaying = true;
		bLoaded = true;
	}

Now I just get a crash on exit which looks similar but I think is again thread related.

Edit:

Commenting out this line fixes the crash on exit but I see a similar crash when the video ends playback ( looping set to false ). So I think there is a threading issue relating to the callbacks.

void ofGstUtils::close(){
	if(bPlaying){
		if(!bIsMovieDone && !bPaused && !isStream){
			std::unique_lock<std::mutex> lck(eosMutex);
			closing = true;
			gst_element_send_event(gstPipeline,gst_event_new_eos());
//			if(eosCondition.wait_for(lck,std::chrono::milliseconds(5000))==std::cv_status::timeout){
//				ofLogWarning("ofGstUtils") << "didn't received EOS in 5s, closing pipeline anyway";
//			}
			closing = false;
		}
	}
	stop();
1 Like

Hey what did you use for this? I’d like to learn but not sure where to start. I’d like to be able to work on problems like this someday.

@TimChi

I set this at the top of ofApp::setup
setenv("GST_DEBUG_DUMP_DOT_DIR", "/Users/theo/Documents/CODE/PROJECTS/OF_0.11.2/apps/myApps/gstreamerTest/bin/gstDump", 1);

The folder gstDump needs to exist.

Then I added this call to the end ofGstUtils::startPipeline ( before the return )
GST_DEBUG_BIN_TO_DOT_FILE((GstBin *)(gstPipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline");

That produces a .dot file which you can turn into a pdf which shows the graph structure.

You can also add to ofApp::setup:
setenv("GST_DEBUG", "4", 1);
To set the level of debug messages printed out by gstreamer. 4 is quite overwhelming and 3 is more critical stuff.

2 Likes

Solved the crash on exit issue with using playbin3 ( this time it works!! ) along with the other changes mentioned above.

Have a wip PR @nlavella - let me know if it works for you?
Hoping this fixes it for Linux too but it seems to be the same issue.

2 Likes

yay, progress! still get a coredump, but the app window now shows up and briefly plays the videos! Sometimes the video plays for a couple of seconds, sometimes I only get the first frame before the example crashes - but it seems this is definitely the right direction! :partying_face:

edit: just realized: it crashes as soon as my mouse moves across the window. If I start the player with the mousepointer outside the app window, the video plays until the end & only then crashes.

This is on Fedora 36 with Intel Iris Xe embedded graphics - I’ll give this a shot on another laptop with discrete NVidia graphics as well.

(btw - would you prefer I comment here, or on the pull request?)