trying to catch and restart locked up webcams automatically --- my idea

I’ve got a piece that uses five webcams, some of them HD. The display only shows one of their feeds at a time, but keeps them active continuously so it can switch between them.

I can usually get it to work for a few hours but eventually one or two cameras lock up. It’s never the same cam or the same USB port. I’m using PCIe USB cards in addition to the built-in ports to try and spread it all out, but it failes regardless. And I need it to run indefinitely.

So now I’m working on trying to detect when a camera is having problems so I can automatically reinitialize it. I’ll just have the app avoid that camera while it’s doing the reinit. Using isFrameNew(), I’ve found that I get more failed frames the higher my frame rate. However, those failed frames are intermixed with successfully grabbed frames. Therefore the trick is to look for a succession of failed frames in a row. Once they’re over some threshold I should be able to presume the camera has locked up. I also have to deal with the fact that isFrameNew() sometimes returns true on the first frame requested even though the cam isn’t up and running yet (a typical startup on my built-in iSight is to get one true from isFrameNew(), followed by about 50 false, then the camera seems live).

So below is a simple example that seems to be doing the job for one camera. However, since one camera never locks up I’ll have to expand it out for the five to really test it. I’m a bit worried about the overhead of all this if/else checking.

I’m including it here to see if anyone else has alternative ideas on how to deal with this issue? Any ideas? Any suggestions for improving my approach?

Thanks!

  
#include "testApp.h"  
  
#define BAD_FRAME_THRESHOLD 5  
  
//--------------------------------------------------------------  
void testApp::setup(){  
	  
	camWidth 		= CAPTURE_WIDTH;	// try to grab at this size.   
	camHeight 		= CAPTURE_HEIGHT;  
      
    ofSetFrameRate(30);  
    ofSetVerticalSync(true);  
       
    failed = 0;  
    good = 0;  
    startupcount = 0;  
	starting = false;  
      
	vidGrabber.setVerbose(true);  
	bool done = vidGrabber.initGrabber(camWidth,camHeight);  
      
    if(done) {   
        printf("init done\n");   
        starting = true;   
    }  
  
}  
  
  
//--------------------------------------------------------------  
void testApp::update(){  
	  
//	ofBackground(100,100,100);  
	  
	vidGrabber.grabFrame();  
      
    // got a new frame  
    if(vidGrabber.isFrameNew()) {  
          
        // for some reason isFrameNew returns true on the first frame sometimes  
        // even though it's not yet capturing.  this bit allows me to track  
        // whether we're starting up and and to toss out a first frame glitch  
        // once we're beyond some threshold of failed frames (40) then we can presume  
        // the next good frame is a real one  
          
        if(starting && startupcount > 40) {  
            starting = false;  
            startupcount = 0;  
            printf("startup is finished\n");  
        }  
          
        // if this new frame immediately followed a failed frame then we want to   
        // start checking to see if the camera has locked up  
        if(failed) {  
              
            // if the number of failed frames is over a threshold, then we presume   
            // the camera really is locked up.  close the connection and reinit.  
            if(failed > BAD_FRAME_THRESHOLD) {  
                printf("num failed frames before good: %d\n", failed);  
                // trigger close and reinit routines here  
            }  
              
            // reset our failed frame tracker  
            failed = 0;  
        }  
          
        // increment our good frame tracker  
        good ++;  
    }  
      
    // our request to get a new frame failed  
    else {  
          
        // if this failed request immediately followed a successful request  
        // then just reset our good frame counter because it's probably just  
        // a case of the camera not keeping up with update  
        if(good) {  
            good = 0;  
        }  
          
        // if we're not in a camera startup situation, then we need to increment  
        // our failed frame counter  
        if(!starting) {  
            failed ++;  
        }   
          
        // otherwise keep track of how many frames into the startup we are so we can   
        // easily get past that first frame glitch from isFrameNew  
        else {  
            startupcount++;  
        }  
    }  
	  
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
  
	vidGrabber.draw(0,0);  
  
}  
  

the general idea looks ok, but i would recommend two things:

  • wrap it up into a class that extends ofVideoGrabber instead of writing it in the testApp like that, it will make it much cleaner
  • it’s better to debug what the problem is than to try and work around it. if you have a problem with cameras locking up, you should address it rather than hacking something like this. though if you’re using multiple PCIe cards i’m not sure what the problem is…

thanks for the feedback. the class idea is a good one.

as for fixing the core problem, i’m not sure what else to try. i’ve tried multiple PCs, many different configurations of varying brands of webcams, different brands of PCIe cards (both USB 2 and USB 3), etc. the machine I’ve settled on is a screamer: an asrock p67 extreme 6 with a core i7 2600k CPU. each cam gets its own USB bus (four cards, each with one cam, one on the internal bus). there’s really no reason bandwidth-wise that I can think of why it wouldn’t work (task manager shows the machine nowhere near its capacity). I keep each camera active and simply change which camera I’m showing at a particular moment.

i get longer periods of stability if I reduce the resolution of the cameras or if i reduce the number of cameras. it would probably work if i could live with every camera being 640x480, but i need some of them to run at higher res (1280x720). the symptom is that eventually one or more cameras will just show a frozen image. it can take many hours for this to start happening.

That’s on win7 with the latest oF from github (although I had tried this with previous versions earlier). I’ve tried on a Mac Pro as well, but it can’t come close to handling all those cameras at once.

if anyone has a suggestion for other things to try for fixing the core problem i’d really appreciate the advice! i agree that the best solution would be to not have to deal with the instability.

If one camera never locks up, does two? does three? when does this start happening?

If you post your 5-camera code I’ll take a look to see if I can find anything that might be causing this. I’ve done some multi-cam work before and I can probably give you some feedback.

I don’t think two will ever lock up. It’s been a while since I did the testing (it can take hours just to trigger the problem so it’s a slow process) but I believe 3 will eventually trigger the lockup.

You can see the code below. I’ve cut out a bunch of code that just deals with making calculations that affect the position parameters to draw(), but the base code is all there. I’ve recently learned I can draw the camera mirrored with by adjusting those same parameters to draw() but haven’t gotten around to changing it here yet. Regardless, I don’t think that would affect the lockup.

Any suggestions are appreciated. Thanks for taking a look!

  
#include "testApp.h"  
  
  
void testApp::setup(){  
	camWidth 		= 1280;	// try to grab at this size.  
	camHeight 		= 720;  
 	screenWidth    = 1920;  
	screenHeight     = 1200;  
	maxX            = 2020;  
	maxY            = 1300;  
  
	oX = 0;  
	oY = 0;  
    zoomCnt         = 0;  
  
    camchoice = 1;  
    lastCam = 1;  
    testing = false;  
  
    drawHeight = screenHeight;  
    drawWidth = screenWidth;  
  
	cam1.setVerbose(true);  
	cam1.setDeviceID(0);  
	cam1.initGrabber(camWidth,camHeight);  
	cam1.listDevices();  
  
    cam2.setVerbose(true);  
	cam2.setDeviceID(1);  
	cam2.initGrabber(camWidth,camHeight);  
  
    cam3.setVerbose(true);  
	cam3.setDeviceID(2);  
	cam3.initGrabber(camWidth,camHeight);  
  
    cam4.setVerbose(true);  
	cam4.setDeviceID(3);  
	cam4.initGrabber(camWidth,camHeight);  
  
    cam5.setVerbose(true);  
	cam5.setDeviceID(4);  
	cam5.initGrabber(camWidth,camHeight);  
  
	videoInverted 	= new unsigned char[camWidth*camHeight*3];  
	videoTexture.allocate(camWidth,camHeight, GL_RGB);  
  
    ofHideCursor();  
  
	camchoice = 1;  
	movement = FIVE;  
	duration = 30;  
    frameCount = 0;  
  
}  
  
  
//--------------------------------------------------------------  
void testApp::update(){  
  
	// then we're starting or in a segment  
	if(testing) { }  
	else if(frameCount != duration+1) { frameCount++; }  
	else {  
	    // it's a new event coming  
        // pick the camera, call functions to calc   
        // adjustments to oX, oY, drawWidth and drawHeight  
          
	    newEvent = true;  
	    lastCam = camchoice;  
	    camchoice = rand() % 4 + 1;  
        movement = rand() % 5 + 1;  
  
	    if(rand() % 10 == 1) {  
            duration = rand() % 30 + 20;  
            movement = FIVE;  
	    }  
	    else {  
            duration = rand() % 14 + 10;  
	    }  
  
	    frameCount = 0;  
  
        if(movement == ONE)        { setupOne(); }  
        else if(movement == TWO)   { setupTwo(); }  
        else if(movement == THREE) { setupThree(); }  
        else if(movement == FOUR)  { setupFour(); }  
        else if(movement == FIVE)  { setupFive(); }  
        else { printf("movement error in update."); }  
	}  
  
  
    if(camchoice == 1) {  
        cam1.grabFrame();  
    } else if(camchoice == 2) {  
        cam2.grabFrame();  
    } else if(camchoice == 3) {  
        cam3.grabFrame();  
    } else if(camchoice == 4) {  
        cam4.grabFrame();  
    } else if(camchoice == 5) {  
        cam5.grabFrame();  
    } else if(camchoice == 6) {  
        cam6.grabFrame();  
    } else if(camchoice == 7) {  
        cam7.grabFrame();  
    }  
  
    if(newEvent) {  
         if(lastCam == 1) {  
            cam1.grabFrame();  
        } else if(lastCam == 2) {  
            cam2.grabFrame();  
        } else if(lastCam == 3) {  
            cam3.grabFrame();  
        } else if(lastCam == 4) {  
            cam4.grabFrame();  
        } else if(lastCam == 5) {  
            cam5.grabFrame();  
        } else if(lastCam == 6) {  
            cam6.grabFrame();  
        } else if(lastCam == 7) {  
            cam7.grabFrame();  
        }  
    }  
  
	if(camchoice == 1) {  
		if (cam1.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam1.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
	  
	if(camchoice == 2) {  
		if (cam2.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam2.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
	if(camchoice == 3) {  
		if (cam3.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam3.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
	if(camchoice == 4) {  
		if (cam4.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam4.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
	if(camchoice == 5) {  
		if (cam5.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam5.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
	if(camchoice == 6) {  
		if (cam6.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam6.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
	if(camchoice == 7) {  
		if (cam7.isFrameNew()){  
			int totalPixels = camWidth*camHeight*3;  
			unsigned char * pixels = cam7.getPixels();  
			for (int i = 0; i < totalPixels; i++){  
				videoInverted[i] = 255 - pixels[i];  
			}  
			videoTexture.loadData(videoInverted, camWidth,camHeight, GL_RGB);  
		}  
	}  
  
}  
  
//--------------------------------------------------------------  
void testApp::draw(){  
  
    // do some calculations that affect oX, oY,   
    // drawWidth, and drawHeight for panning  
    // and zooming on the frame (not shown)  
  
	if(newEvent) {  
	    newEvent == false;  
		if(lastCam == 1) {  
	        cam1.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 2) {  
		    cam2.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 3) {  
		    cam3.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 4) {  
		    cam4.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 5) {  
		    cam5.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 6) {  
		    cam6.draw(oX,oY,drawWidth,drawHeight);  
		} else if(lastCam == 7) {  
		    cam7.draw(oX,oY,drawWidth,drawHeight);  
		}  
	}  
	  
	else {  
		if(camchoice == 1) {  
	        cam1.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 2) {  
		    cam2.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 3) {  
		    cam3.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 4) {  
		    cam4.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 5) {  
		    cam5.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 6) {  
		    cam6.draw(oX,oY,drawWidth,drawHeight);  
		} else if(camchoice == 7) {  
		    cam7.draw(oX,oY,drawWidth,drawHeight);  
		}  
	}  
  
}  
  
  

A side note which doesn’t answer your question: it should be possible to use an array of camera objects and operate with cams[camchoice] etc, thus making code cleaner and easier to maintain, no?

A comment about initGrabber says: “It is good to check what the actual size is before you start processing the pixels.”

You use the same size for videoInverted array. Imagine that one of the cameras doesn’t support the requested resolution. Then you might be operating beyond boundaries of the arrays returned by getPixels().

Again, it might be that this is not the root cause of locking cameras.

I would recommend calling the grabFrame() function on all cameras at all times. at one point I had camera lockup issues which seemed to be caused by not updating the cameras even if I wasn’t going to use them for anything. Give it a try and let me know what happens.

I made this change and the app has run without locking a camera for more than 12 hours! That’s definitely the longest it’s run so far. I’ll keep it running for another two or three days and report back. Thanks so much for this suggestion!

My goal in the way I had it was to not waste time grabbing frames from cameras when they weren’t needed. I do notice a slight slowdown now that I’m grabbing them all all the time. A lag is probably the best way to describe it—a lag between my movement captured by the camera and when that movement shows up on screen. There always was one but it just seems a tiny bit longer now. I’m guessing there’s not much that can be done about it, as I’ve got a superfast PC. I’m running four of my five cams at 720p (the other at 640x480) so that’s probably adding to the slowness.

well, unfortunately one of the cameras locked up sometime between 12 hours and 48 hours.

any other ideas?

i’m trying different configurations of cameras and where they’re plugged in. I think next I may try a different motherboard altogether. the one i’m using has a mix of PCI and PCIe slots, so my USB cards are also a mix. i’ll look for a board with all PCIe slots so i can have all PCIe usb cards in case that helps.

are you running on windows? You may want to check out your DPC Latency.
http://www.thesycon.de/deu/latency-check.shtml

Sometimes things like wireless drivers can cause spikes which will drop out your video.

yes, windows. i presume i should only be checking the latency while NOT running my program? when I run it the latency kicks way up into the red.

try running it without the program to see if you can find any random spikes. then try with only one camera. if there are any spikes, try disabling devices in device manager that you dont need. I think there is another program, called Rat or something, that will track where the DPCs are coming from if you can’t figure it out.

Good luck!

thanks for the leads. i have been trying every configuration I can imagine over the last week and just cannot get anything to run for more than 36 hours without locking up a camera.

i get no spikes when running with one camera. as I add each camera the latency increases until at 5 cameras i’m averaging around 5k µs. it will stay stable for as long as i watch, but usually it will at some point get a spike up as high as 18k µs. i have no idea where in a 24-36 hour run this happens. and these spikes are with every conceivable device disabled in device manager. there’s basically nothing left to disable that I can find.

i got more PCIe USB cards and filled all my slots with them (I have four slots). this allowed me to put one camera on one of the motherboard’s unused USB busses, while putting each of the other cameras on its own PCIe USB bus. no improvement. also, i see that each PCIe card increases the base latency just a bit (about 25 µs per card). i also tried running everything off of the built-in busses. seems to run about as long.

i knocked down the resolution on the cameras to 800x448. this got the average DPC latency into the green, around 500 µs. i can then see some occasional spikes (say into 1k µs) but again, with everything disabled in devmanager i’m not sure what else to do.

i tried RATT but it doesn’t run on win7. i found another tool called latencymon (win7 compatible) that i’ll try next.

any other ideas?? i really appreciate your suggestions! if i find anything with latencymon i’ll write back.

good luck with the latency monitor, I hope you find the culprit. I suppose if nothing else works you could try linux, but no garuntee that your cameras would work out of the box, and you might have freezing issues there too.

is the linux videoGrabber performance as good as windows? could be worth a try. any particular flavor of linux preferred? i see ubuntu is supported, but do i need to go with 32-bit for webcam driver compatibility or can I use 64-bit?

I’ve never tried any HD cameras in linux, but the PS3 Eye cameras work really well. As far as I can tell, there isnt any reason not to go with 64bit. I haven’t had any compatibility issues with that in a long long time.