ofxOpenNI installation randomly(?) crashes with crazy kids

Hi there- I need your help. I’ve installed an ofxOpenNI interactive piece at the boston children’s museum and am having difficulty tracking down a fugly bug.

I run this code all day long at my house, but given the fact I dont’ have a 100 children running in and out of frame, it’s tough to get the crash to happen. Generally, it seems to result from resetting the user tracker -I’ll get either a memory error or a crash in one of the threads

The basic gist is : run a movie as fast or as slow as the child’s hand moves between two points. When the kid walks away, time out and then reset the movie. B/c OpenNI hangs on to the user objects too long, I’m resetting the tracker by removing/adding the generator(s).

My question is- should I have to remove all the generators? I have tried removing/adding just the UserGenerator, but I think that is partially if not entirely responsible for the random crashes…

Here’s my reset code:

void testApp::reset(){
    
    ofLog() << "***RESETTING***";
    ofLog() << ofGetTimestampString();
    

   
        introMode = RESET_MODE;
        beatCounter = 0;
       
        bTimer.stop();
        if (movie.isPlaying())
                movie.stop();
        if (attractLoopMovie.isPlaying())
            attractLoopMovie.setPaused(true);
        
        if (attractLoopMovie.isLoaded())
            attractLoopMovie.setPosition(0);
        LIVE_HAND = false;
        LIVE_USER = false;
        
        vol = .2;
        
        bLessThan = false;
        bGreaterThan = false;
        counter = 0;
        
        ofLog() << "reset music samp and stretches"; //maxmillian lib
        //reset users
        samp.reset();
        current = 0;
        stretches[current]->position = 0;
        speed = 1;
        
        
        if (movie.isLoaded())
            movie.setPosition(0);
        
        timeoutCounter = 0;
        
        //deleting hand after music reset cuases problems... do
        
     

        ofRemoveListener(openNIDevice.handEvent, this, &testApp::handEvent);
        ofRemoveListener(openNIDevice.userEvent, this, &testApp::userEvent);
    
    
        ofLog() << "about to reset users/openni";
        //reset users
        openNIDevice.setPaused(true);  
//seems necessary to avoid crashes.. but still occasionally crashing!!!
  /* 
   do I need to do this? or can I just remove the user generator??
   openNIDevice.removeImageGenerator();
    openNIDevice.removeDepthGenerator();
    openNIDevice.removeHandsGenerator();
    */
 
    openNIDevice.removeUserGenerator();
        sleep(5);
        while  (openNIDevice.isUserOn()){}
        
        openNIDevice.addUserGenerator();
    /*if removed, then would need to re-add
     openNIDevice.addImageGenerator();
     openNIDevice.addDepthGenerator();
     openNIDevice.setRegister(true);
     openNIDevice.setMirror(true);
     openNIDevice.addUserGenerator();
     openNIDevice.addHandsGenerator();
     openNIDevice.addAllHandFocusGestures();
     */ß
    
    sleep(3);
        openNIDevice.setPaused(false);

        ofLog() << "about to start attractloop";
        
        
        bInit = true;  //at the bottom here b/c when we do a reset, we need time to regen
    //attractLoopMovie.setSpeed(1);   maybe take out...
      //  attractLoopMovie.play();

   
}

Is there any way you can run it in debug to see the stack trace of the crash?
Also, have you looked at the app’s memory consumption (activity viewer, etc) to see if there’s any kind of leak?

What is this for ? Have you removed some code in the while loop ? It does’t seem the greatest idea to me to run an empty while loop.

Maybe you should try to stand in front of your camera and force the reset every 5 or 10 second and see if you can cause the crash to happen. (There might be a sweet spot where you can reproduce the bug by yourself in a timely manner without needing to have 100 kids around)

EDIT:

Actually if you look at the source code of ofxOpenNI the add/remove function you’re calling should return true/false on success/failure. I think what you want to do is more something like :

 while(!openNIDevice.removeUserGenerator()) {
        sleep(5);}
   //     while  (openNIDevice.isUserOn()){}
if (!openNIDevice.addUserGenerator()) { Log() << "Something went wrong" ; }

It may or may not be your fix though. (Best approach is first find a way to reproduce your bug in a more efficient way, and then try to fix it…)

thanks, good suggestion to check the ret val. Re: reproducing: I had put a glove on a stick to trigger the hand/userCreation/timeout for my local build but no crashes occurred over 24 hours!. Re: The while loop was my attempt -other than pausing the device- to try to get the threads a chance to die nicely so that when the addUserGenerator which would call a g_User.Release() wouldn’t cause an alloc crash b/c the user obj had already been swept…

Zach- I’ll post a sample of the error logs shortly…

i would check the sound, seems like you are using maximilian? which will run in a different thread, the error that you are describing not happening every time and only when there’s much more interaction than when debugging sounds like the tipical error produced by not thread safe code

arturo, yes, I am using ofxMaxim… and I am watching the sample position and if it reaches the end of the music, stop, reset the sample and the movie as well. i have gotten a few random threads dying in the QTVideoDecoder thread but none indicating the music lib AFAIK…

eg:
I check before and after the for loop due to my inability to accurately capture the music end at the exact end when the audio requested event fires

void testApp::audioRequested 	(float * output, int bufferSize, int nChannels){
    if (introMode == HANDS_USER_STEADY_MODE){
if (stretches[current]->position >= samp.length -1000){
        ofLog() << " music finished ";
        movie.stop();
        attractLoopMovie.stop();
        ofRemoveListener(openNIDevice.handEvent, this, &testApp::handEvent);
        ofRemoveListener(openNIDevice.userEvent, this, &testApp::userEvent);
introMode = RESETTING_MODE;
        reset();
    }

    for (int i = 0; i < bufferSize; i++){
     
		wave = stretches[current]->play(speed, rate, 0.1, 2, 0);    
		if (fft.process(wave)) {
			oct.calculate(fft.magnitudes);
		}
//play result
        
		mymix.stereo(wave, outputs, 0.5);
		lAudioOut[i] = output[i*nChannels    ] = outputs[0] * vol; 
		rAudioOut[i] = output[i*nChannels + 1] = outputs[1] * vol;
	}

//and I check again here too  :(

}

okay: sample errors:

here’s one memory error:

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
*** error for object 0x69986400: pointer being freed was not allocated
Thread 6 Crashed:
0   libsystem_kernel.dylib        	0x96a3aa6a __pthread_kill + 10
1   libsystem_c.dylib             	0x98ac3b2f pthread_kill + 101
2   libsystem_c.dylib             	0x98afa4ec abort + 168
3   libsystem_c.dylib             	0x98adce03 free + 428
4   libXnVFeatures_1_5_2.dylib    	0x00f6846a std::_Deque_base<float, std::allocator<float> >::_M_destroy_nodes(float**, float**) + 42
5   libXnVFeatures_1_5_2.dylib    	0x00f67eea LBS::reset_body_measures() + 218
6   libXnVFeatures_1_5_2.dylib    	0x00f70802 LBS::update_user_body_params() + 4962
7   libXnVFeatures_1_5_2.dylib    	0x00f68252 LBS::Update(xn::SceneMetaData const&, int, Vector3D<int>, Box3D<int>) + 482
8   libXnVFeatures_1_5_2.dylib    	0x00e50044 FeatureExtractor::UpdateLBS(LBSOutputData&, LBS*) + 820
9   libXnVFeatures_1_5_2.dylib    	0x00e50dfc FeatureExtractor::ApplyLBS(SensorTime, int) + 76
10  libXnVFeatures_1_5_2.dylib    	0x00e53b52 FeatureExtractor::Run(int, SensorTime, int) + 50
11  libXnVFeatures_1_5_2.dylib    	0x00e54cb5 FeatureExtractor::Update(int, SensorTime, int) + 101
12  libXnVFeatures_1_5_2.dylib    	0x00eceee9 MultiUserFeatureExtractor::Update(xn::DepthGenerator const&, xn::DepthMetaData const&, xn::SceneMetaData const&) + 201
13  libXnVFeatures_1_5_2.dylib    	0x00d4a64b XnVSkeletonGenerator::UpdateFeatureExtractor() + 363
14  libXnVFeatures_1_5_2.dylib    	0x00d4c7c6 XnVSkeletonGenerator::UpdateUsers() + 1862
15  libXnVFeatures_1_5_2.dylib    	0x00d43ced XnVSkeletonCrossHandsPose::UpdateUsers() + 29
16  libXnVFeatures_1_5_2.dylib    	0x00d5792d XnVSkeletonSelectionPoseGenerator::UpdateUsers() + 29
17  libXnVFeatures_1_5_2.dylib    	0x00d48841 XnVSkeletonGenerator::UpdateData() + 97
18  libOpenNI.dylib               	0x00abda36 xnUpdateDataImpl(XnInternalNodeData*) + 70
19  libOpenNI.dylib               	0x00abddfc xnUpdateTreeImpl(XnNodeInfo const*) + 156
20  libOpenNI.dylib               	0x00abe32f xnWaitAndUpdateData + 399
21  com.yourcompany.openFrameworks	0x000f17b9 ofxOpenNI::updateGenerators() + 329
22  com.yourcompany.openFrameworks	0x000f7e48 ofxOpenNI::threadedFunction() + 40
23  com.yourcompany.openFrameworks	0x002ae1a5 ofThread::run() + 197
24  com.yourcompany.openFrameworks	0x00121f4b Poco::ThreadImpl::runnableEntry(void*) + 121
25  libsystem_c.dylib             	0x98ac25b7 _pthread_start + 344
26  libsystem_c.dylib             	0x98aacd4e thread_start + 34

and here’s the VideoDecodeThread crash

Crashed Thread:  33  com.apple.quicktime.VideoDecodeThread

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
*** error for object 0x837a2c04: incorrect checksum for freed object - object was probably modified after being freed.

Thread 33 Crashed:: com.apple.quicktime.VideoDecodeThread
0   libsystem_kernel.dylib        	0x96a3aa6a __pthread_kill + 10
1   libsystem_c.dylib             	0x98ac3b2f pthread_kill + 101
2   libsystem_c.dylib             	0x98afa4ec abort + 168
3   libsystem_c.dylib             	0x98ae4227 szone_error + 443
4   libsystem_c.dylib             	0x98ae5482 free_list_checksum_botch + 50
5   libsystem_c.dylib             	0x98ae9fb5 small_malloc_from_free_list + 659
6   libsystem_c.dylib             	0x98ae99a5 szone_malloc_should_clear + 3067
7   libsystem_c.dylib             	0x98adea9e szone_malloc + 24
8   libsystem_c.dylib             	0x98adc5ab malloc_zone_malloc + 75
9   libsystem_c.dylib             	0x98adcfe7 malloc + 53
10  QuickTimeH264.scalar          	0x16000521 0x15cf0000 + 3212577
11  QuickTimeH264.scalar          	0x15e9e3c2 0x15cf0000 + 1762242
12  QuickTimeH264.scalar          	0x15ec3942 0x15cf0000 + 1915202
13  QuickTimeH264.scalar          	0x15ec3cfc 0x15cf0000 + 1916156
14  QuickTimeH264.scalar          	0x15cf5747 JVTDecoDrawBand + 1296
15  com.apple.CoreServices.CarbonCore	0x92218adb callComponentStorage_44 + 25
16  com.apple.CoreServices.CarbonCore	0x92209abf CallComponentFunctionCommonWithStorage(char**, ComponentParameters*, long (*)(), unsigned long) + 45
17  com.apple.CoreServices.CarbonCore	0x92209aff CallComponentFunctionWithStorageProcInfo + 30
18  com.apple.QuickTimeH264.component	0x96aa20d9 JVTDecoComponentDispatch + 64
19  com.apple.CoreServices.CarbonCore	0x92183ade CallComponent + 151
20  com.apple.CoreServices.CarbonCore	0x92183b38 CallComponentDispatch + 29
21  com.apple.QuickTimeComponents.component	0x930c9c21 0x92538000 + 12131361
22  com.apple.QuickTimeComponents.component	0x92c2a789 0x92538000 + 7284617
23  com.apple.QuickTime           	0x94631de0 VideoDecodeThread + 128
24  libsystem_c.dylib             	0x98ac25b7 _pthread_start + 344
25  libsystem_c.dylib             	0x98aacd4e thread_start + 34

and thanks to Arturo mentioning it, I found this one too

Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
*** error for object 0x81b7dc04: incorrect checksum for freed object - object was probably modified after being freed.
 

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x96a3aa6a __pthread_kill + 10
1   libsystem_c.dylib             	0x98ac3b2f pthread_kill + 101
2   libsystem_c.dylib             	0x98afa4ec abort + 168
3   libsystem_c.dylib             	0x98ae4227 szone_error + 443
4   libsystem_c.dylib             	0x98ae5482 free_list_checksum_botch + 50
5   libsystem_c.dylib             	0x98ae9fb5 small_malloc_from_free_list + 659
6   libsystem_c.dylib             	0x98ae99a5 szone_malloc_should_clear + 3067
7   libsystem_c.dylib             	0x98adea9e szone_malloc + 24
8   libsystem_c.dylib             	0x98adc5ab malloc_zone_malloc + 75
9   libsystem_c.dylib             	0x98adcfe7 malloc + 53
10  com.apple.CoreServices.CarbonCore	0x92140577 CSMemNewPtr + 50
11  com.apple.CoreServices.CarbonCore	0x921b9b98 CSMemReallocateHandle + 82
12  com.apple.QuickTimeComponents.component	0x92b7304e 0x92538000 + 6533198
13  com.apple.CoreServices.CarbonCore	0x922187c1 callComponentStorage_4444 + 39
14  com.apple.CoreServices.CarbonCore	0x92209abf CallComponentFunctionCommonWithStorage(char**, ComponentParameters*, long (*)(), unsigned long) + 45
15  com.apple.CoreServices.CarbonCore	0x92209aff CallComponentFunctionWithStorageProcInfo + 30
16  com.apple.QuickTimeComponents.component	0x92b60aab STMediaComponentDispatch + 81
17  com.apple.CoreServices.CarbonCore	0x92183ade CallComponent + 151
18  com.apple.CoreServices.CarbonCore	0x92183b38 CallComponentDispatch + 29
19  com.apple.QuickTimeComponents.component	0x930d1967 0x92538000 + 12163431
20  com.apple.QuickTimeComponents.component	0x92b949c6 0x92538000 + 6670790
21  com.apple.CoreServices.CarbonCore	0x922187c1 callComponentStorage_4444 + 39
22  com.apple.CoreServices.CarbonCore	0x92209abf CallComponentFunctionCommonWithStorage(char**, ComponentParameters*, long (*)(), unsigned long) + 45
23  com.apple.CoreServices.CarbonCore	0x92209aff CallComponentFunctionWithStorageProcInfo + 30
24  com.apple.QuickTimeComponents.component	0x92b8c9eb Sound3ComponentDispatch + 99
25  com.apple.CoreServices.CarbonCore	0x92183ade CallComponent + 151
26  com.apple.CoreServices.CarbonCore	0x92183b38 CallComponentDispatch + 29
27  com.apple.QuickTimeComponents.component	0x930d1967 0x92538000 + 12163431
28  com.apple.QuickTimeComponents.component	0x92b95ab5 0x92538000 + 6675125
29  com.apple.QuickTimeComponents.component	0x92b969cb 0x92538000 + 6678987
30  com.apple.CoreServices.CarbonCore	0x9221861c callComponentStorage_44444 + 47
31  com.apple.CoreServices.CarbonCore	0x92209abf CallComponentFunctionCommonWithStorage(char**, ComponentParameters*, long (*)(), unsigned long) + 45
32  com.apple.CoreServices.CarbonCore	0x92209aff CallComponentFunctionWithStorageProcInfo + 30
33  com.apple.QuickTimeComponents.component	0x92b8c9eb Sound3ComponentDispatch + 99
34  com.apple.CoreServices.CarbonCore	0x92183ade CallComponent + 151
35  com.apple.CoreServices.CarbonCore	0x92183b38 CallComponentDispatch + 29
36  com.apple.QuickTime           	0x9461e03f MediaMoviesTask + 61
37  com.apple.QuickTime           	0x9461d1a7 TaskMovie_priv + 3453
38  com.apple.QuickTimeComponents.component	0x92bdb9ca 0x92538000 + 6961610
39  com.apple.QuickTimeComponents.component	0x92be7d05 0x92538000 + 7011589
40  com.apple.QuickTimeComponents.component	0x92bd9f75 0x92538000 + 6954869
41  com.apple.CoreServices.CarbonCore	0x92209abf CallComponentFunctionCommonWithStorage(char**, ComponentParameters*, long (*)(), unsigned long) + 45
42  com.apple.CoreServices.CarbonCore	0x92209aff CallComponentFunctionWithStorageProcInfo + 30
43  com.apple.QuickTimeComponents.component	0x92bd8a02 _MCComponentDispatch + 81
44  com.apple.CoreServices.CarbonCore	0x92183ade CallComponent + 151
45  com.apple.CoreServices.CarbonCore	0x92183b38 CallComponentDispatch + 29
46  com.apple.QuickTime           	0x9461c0c3 MCIdle + 37
47  com.apple.QuickTime           	0x9461add2 QTOMovieObject::SendCommand(unsigned long, void const*) + 2152
48  com.apple.QuickTime           	0x94603594 DispatchQTMsg(void const*, unsigned long, unsigned long, unsigned long, QTOGenericObject*) + 866
49  com.apple.QuickTime           	0x946031a4 QTObjectTokenPriv::SendMessageToObject(QTMessagePriv*, unsigned long) + 178
50  com.apple.QuickTime           	0x94602b85 QTObjectTokenPriv::DispatchMessage(QTMessagePriv*, void const*, __CFAllocator const*, unsigned long, unsigned long, ComponentMsgParam*, unsigned long, unsigned long) + 1445
51  com.apple.QuickTime           	0x9461a55f QTSendToObject + 99
52  com.apple.QTKit               	0x973d08b1 QTObjectTokenExecuteCommand + 82
53  com.apple.QTKit               	0x9739492f -[QTMovie_QuickTime idle] + 311
54  com.apple.QTKit               	0x97394660 +[QTMovie_QuickTime idleAllMovies:] + 340
55  com.apple.Foundation          	0x93e99cd4 __NSFireTimer + 117
56  com.apple.CoreFoundation      	0x90189416 __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 22
57  com.apple.CoreFoundation      	0x90188db5 __CFRunLoopDoTimer + 709
58  com.apple.CoreFoundation      	0x9016dbc2 __CFRunLoopRun + 1842
59  com.apple.CoreFoundation      	0x9016d02a CFRunLoopRunSpecific + 378
60  com.apple.CoreFoundation      	0x9016ce9b CFRunLoopRunInMode + 123
61  com.apple.HIToolbox           	0x9936ef5a RunCurrentEventLoopInMode + 242
62  com.apple.HIToolbox           	0x9936ebf5 ReceiveNextEventCommon + 162
63  com.apple.HIToolbox           	0x9936eb44 BlockUntilNextEventMatchingListInMode + 88
64  com.apple.AppKit              	0x9154c9aa _DPSNextEvent + 724
65  com.apple.AppKit              	0x9154c1dc -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 119
66  com.apple.glut                	0x0090200d -[GLUTApplication _runMainLoopUntilDate:autoreleasePool:] + 221
67  com.apple.glut                	0x00902240 -[GLUTApplication run] + 192
68  com.apple.glut                	0x00918ca1 glutMainLoop + 977
69  com.yourcompany.openFrameworks	0x0026491d ofRunApp(ofBaseApp*) + 301
70  com.yourcompany.openFrameworks	0x000f35c8 main + 88
71  com.yourcompany.openFrameworks	0x000c44a5 start + 53

I am the first one to say that it has been a while since I’ve had to track down threads, and the lock_mutex call scares me… thanks for any and all suggestions…

I’m not sure what this error is in particular, but OpenNI is very finicky about losing Skeletons. I’m taking a guess that this:

libXnVFeatures_1_5_2.dylib    	0x00f6846a std::_Deque_base<float, std::allocator<float> >::_M_destroy_nodes(float**, float**) + 42

looks an awful lot like something blowing up when a Skeleton is lost. Are you sure that you’re being very careful with any references to or pointers into the OpenNI skeleton data?

I’m not super familiar with the ofxOpenNI plugin but do you have events listening (or does the addon have events) for the LostUser event? I don’t have the code on this machine but that’s what I was using earlier this year when I had lots of users entering and exiting an OpenNI frame.

as a side note and as a general debugging strategy, is there a way to capture input to openNI (ie some sort of recorder?) I wonder if it’s possible to get some sort of recorded input that would crash the system? It’s usually the best strategy since you can reproduce the crash and study it.

also, have you checked memory usage on the app over time?

I was having similar issues while reseting users by removing and adding the user generator. In my case at least it was related to ofxOpenNI::updateGenerators() being called from openNI thread while I was removing the user generator. Something like this works for me:

 openNIDevice.waitForThread(true);
 openNIDevice.setPaused(true);
 openNIDevice.removeUserGenerator();
 openNIDevice.addUserGenerator();
 openNIDevice.setPaused(false);
 openNIDevice.startThread();

This pull request in github is also interesting about reseting user number in the addon:

Hope this helps.