ofxKinectForWindows2 background substraction

Goodmorning @elliotwoods ,
i’m totally new on this topic (OOP, openframeworks/ computer vision ecc…),
I tried to write a code for background substraction using Kinect v2 and openCV in OF.
I used some code sample like “opencvExample” and “exampleBodyIndexColor”.
My code works but sometimes I get this error

my code for ofApp.h:

	ofxKFW2::Device kinect;
	ICoordinateMapper* coordinateMapper;

	vector<ofVec2f>			colorCoords;
	ofxCvColorImage			colorImg;
	ofxCvGrayscaleImage 	grayImage;
	ofxCvGrayscaleImage 	grayBg;
	ofxCvGrayscaleImage 	grayDiff;

	ofxCvContourFinder 	contourFinder;

	int 				threshold;
	bool				bLearnBakground;

ofApp.cpp:

void ofApp::setup() {

kinect.open();
kinect.initDepthSource();
kinect.initColorSource();
kinect.initInfraredSource();
kinect.initBodySource();
kinect.initBodyIndexSource();

if (kinect.getSensor()->get_CoordinateMapper(&coordinateMapper) < 0) {
	ofLogError() << "Could not acquire CoordinateMapper!";
}

colorCoords.resize(DEPTH_WIDTH * DEPTH_HEIGHT);

colorImg.allocate(320, 240);
grayImage.allocate(320, 240);
grayBg.allocate(320, 240);
grayDiff.allocate(320, 240);

bLearnBakground = true;
threshold = 80;

}
void ofApp::update() {
ofBackground(100, 100, 100);

bool bNewFrame = false;

kinect.update();
bNewFrame = kinect.isFrameNew();

if (bNewFrame) {
	/*
	ofxCvColorImage currImg;
	currImg.allocate(320, 240);

	auto& p = kinect.getColorSource()->getPixels();
	p.resize(320, 240);
	currImg.setFromPixels(p);
	// currImg.resize(320, 240);



	colorImg = currImg;
	*/

	auto& colorPix = kinect.getColorSource()->getPixels();
	auto& depthPix = kinect.getDepthSource()->getPixels();

	coordinateMapper->MapDepthFrameToColorSpace(DEPTH_SIZE, (UINT16*)depthPix.getPixels(), DEPTH_SIZE, (ColorSpacePoint*)colorCoords.data());

	// Loop through the depth image
	for (int y = 0; y < DEPTH_HEIGHT; y++) {
		for (int x = 0; x < DEPTH_WIDTH; x++) {
			int index = (y * DEPTH_WIDTH) + x;
			colorImg.getPixels().setColor(x, y, ofColor::white);

			// For a given (x,y) in the depth image, lets look up where that point would be
			// in the color image
			ofVec2f mappedCoord = colorCoords[index];

			// Mapped x/y coordinates in the color can come out as floats since it's not a 1:1 mapping
			// between depth <-> color spaces i.e. a pixel at (100, 100) in the depth image could map
			// to (405.84637, 238.13828) in color space
			// So round the x/y values down to ints so that we can look up the nearest pixel
			mappedCoord.x = floor(mappedCoord.x);
			mappedCoord.y = floor(mappedCoord.y);

			// Make sure it's within some sane bounds, and skip it otherwise
			if (mappedCoord.x < 0 || mappedCoord.y < 0 || mappedCoord.x >= COLOR_WIDTH || mappedCoord.y >= COLOR_HEIGHT) {
				continue;
			}

			// Finally, pull the color from the color image based on its coords in
			// the depth image
			colorImg.getPixels().setColor(x, y, colorPix.getColor(mappedCoord.x, mappedCoord.y));

		}
	}

	grayImage = colorImg;
	if (bLearnBakground == true) {
		grayBg = grayImage;		// the = sign copys the pixels from grayImage into grayBg (operator overloading)
		bLearnBakground = false;
	}

	// take the abs value of the difference between background and incoming and then threshold:
	grayDiff.absDiff(grayBg, grayImage);
	grayDiff.threshold(threshold);

	// find contours which are between the size of 20 pixels and 1/3 the w*h pixels.
	// also, find holes is set to true so we will get interior contours as well....
	contourFinder.findContours(grayDiff, 20, (340 * 240) / 3, 10, true);	// find holes
}

}

void ofApp::draw() {

// draw the incoming, the grayscale, the bg and the thresholded difference
ofSetHexColor(0xffffff);
colorImg.draw(20, 20);
grayImage.draw(360, 20);
grayBg.draw(20, 280);
grayDiff.draw(360, 280);

// then draw the contours:

ofFill();
ofSetHexColor(0x333333);
ofDrawRectangle(360, 540, 320, 240);
ofSetHexColor(0xffffff);

// we could draw the whole contour finder
//contourFinder.draw(360,540);

// or, instead we can draw each blob individually from the blobs vector,
// this is how to get access to them:
for (int i = 0; i < contourFinder.nBlobs; i++) {
	contourFinder.blobs[i].draw(360, 540);

	// draw over the centroid if the blob is a hole
	ofSetColor(255);
	if (contourFinder.blobs[i].hole) {
		ofDrawBitmapString("hole",
			contourFinder.blobs[i].boundingRect.getCenter().x + 360,
			contourFinder.blobs[i].boundingRect.getCenter().y + 540);
	}
}

// finally, a report:
ofSetHexColor(0xffffff);
stringstream reportStr;
reportStr << "bg subtraction and blob detection" << endl
	<< "press ' ' to capture bg" << endl
	<< "threshold " << threshold << " (press: +/-)" << endl
	<< "num blobs found " << contourFinder.nBlobs << ", fps: " << ofGetFrameRate();
ofDrawBitmapString(reportStr.str(), 20, 600);

}

void ofApp::keyPressed(int key) {

switch (key) {
case ' ':
	bLearnBakground = true;
	break;
case '+':
	threshold++;
	if (threshold > 255) threshold = 255;
	break;
case '-':
	threshold--;
	if (threshold < 0) threshold = 0;
	break;
}

Thanks in advance
forgive me for the bad English.

hey ire!

i’m not sure which specific line of code your error is coming from
so if you can narrow it down to where the error is happening, then i can help more

Hunting where error messages are coming from

pause the debugger

try and hit pause in the debugger, generally it’ll (by coincidence) pause whilst it’s printing the error message
then you can check the Call Stack to see where in the program the error is being emitted from
(if it paused in the wrong place, just hit play and pause again)

remove parts of your program

try removing/commenting out the contents of the draw function and running
then remove parts of the update function bit by bit (starting at the bottom of update)
until you find the offending lines of code and the error message goes away

step through your program

add breakpoints throughout your program and run it
then visual studio will stop at the breakpoints, and you can step through manually
then you can find where the error occured

make sure you’re running in Debug not Release

once you know where the error is coming from, then hopefully you’ll know quite quickly why it might be causing an error
otherwise report back here

thank you
elliot

HI @elliotwoods!!

I tried to follow your instructions . I added some breakpoints and memory allocation error returned was : violation of read access.


what does it mean?
Thanks for your help!
Ire

Hey!

Again, you’re not showing where in your code this error is happening.
I.e. What line in your source (based on what you posted above) triggers the error message to happen?
Please step through break points until you see the error printed to the console, and let me know where in your code triggers the error.

Also, are you saying that this is a new error now with the vector?
What did you change?

Elliot

hey @elliotwoods
I think I’ve finally found the error.
inside the loop FOR (in update) I have these two lines of code.

colorImg.getPixels().setColor(y, x, ofColor::white);
// for colored every pixels of the colorImg in white.
colorImg.getPixels().setColor(y, x, colorPix.getColor(mappedCoord.y, mappedCoord.x));
//pull the color from the color image based on its coords in the depth image

commenting on these lines of code the output I get is this:

do not hate me for being a little hard of thinking :wink:
thank you very much for helping.

thanks so much for tracking it down!

a couple of things come to mind

auto& p = kinect.getColorSource()->getPixels();
p.resize(320, 240);

this takes the pixels INSIDE the color source (by reference, not by copy)
and then resizes them
this is bad. you’re editing data that you shouldn’t be editing
(the only reason that i return a non-const reference is to support ofBaseHasPixels)

MapDepthFrameToColorSpace

this function presumes you have a 1920x1080 image
possibly you’re outside the bounds

finally
is that the ofPixels object inside ofxCvColorImage might not work well with the setColor function
I don’t use either ofxOpenCv or setColor,
you might want to try some simple tests on those together to see if you get expected results.

once you’ve identified the offending lines
it’s always a good idea to make a seperate simple project and play around with those functions
to see if they’re working like you expect them to work