Kinect Drawing Extra Vertices/Indices


#1

Hi All,

I am working on an OF kinect project to essentially draw people covered in fur. I’m creating a mesh and getting the normals, and then drawing the vertices and indices to create the fur effect. However, it seems to be drawing extra vertices/indices around the bodies it detects. Does anyone know why this is? Been struggling with this for a few days now. Here are some pictures to show what is being drawn:

As you can see, there is this weird extra layer of lines around my body. Anyone have an idea about how to get rid of these?

Also - the background of this project is a video but I’ve excluded that for the sake of simplicity for this post.

Code here:
ofApp.h:

#pragma once

#include "ofMain.h"
#include "ofxKinect.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"

using namespace cv;
using namespace ofxCv;

class ofApp : public ofBaseApp {

	public:
		void setup();
		void update();
		void draw();
    
    void updateNormals();

		void keyPressed(int key);
		void keyReleased(int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased(int x, int y, int button);
		void mouseEntered(int x, int y);
		void mouseExited(int x, int y);
		void windowResized(int w, int h);
		void dragEvent(ofDragInfo dragInfo);
		void gotMessage(ofMessage msg);
    
    ofxKinect kinect;
    
    ofVideoPlayer backgroundLoop;
    ofVideoPlayer backgroundDetect;
    
    ofSoundPlayer backgroundSound;
    
    bool blobDetected;
    
    ofMesh mesh;
    ofMesh normalLines;
    
    ofxCv::ContourFinder contourFinder;

    ofxCvGrayscaleImage grayImage;
    
    ofEasyCam cam;
    
    float nearThresh, farThresh;
    float grayFarThresh, grayNearThresh;
    float normalLength;
    int angle;
    int skip = 2;
    
    glm::vec3 getTriangleCenter(glm::vec3 a, glm::vec3 b, glm::vec3 c);
		
    ofTrueTypeFont myfont;
};

ofApp.cpp:

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup(){
    ofBackground(0);
    
    myfont.load("arial.ttf", 32);
    
    //    ofSetFrameRate(30);
    
    //ambient background video
    backgroundLoop.load("backgroundLoop.mp4");
    backgroundLoop.setLoopState(OF_LOOP_PALINDROME);
    //    backgroundLoop.play();
    
    //triggered background video
    backgroundDetect.load("backgroundDetect.mp4");
    backgroundDetect.setLoopState(OF_LOOP_NONE);
    
    //background sound
    backgroundSound.load("backgroundNoise.mp3");
    backgroundSound.play();
    backgroundSound.setLoop(true);
    
    blobDetected = false;

    
    kinect.setRegistration(true);
    kinect.init();
    kinect.open();
    
    grayImage.allocate(kinect.width, kinect.height);
    
    angle = 20;
    kinect.setCameraTiltAngle(angle);
    
    
    grayFarThresh = 145;
    grayNearThresh = 255;
    
    farThresh = -800;
    nearThresh = 1000;
    
//    farThresh = 800;
//    nearThresh = 500;
    
    contourFinder.setMinArea(50);
    contourFinder.setThreshold(30);
    contourFinder.getTracker().setPersistence(10);


//    farThresh = 1000;
//    nearThresh = 50;
}

//--------------------------------------------------------------
void ofApp::update(){
    mesh.setMode(OF_PRIMITIVE_TRIANGLES);
    kinect.update();
    
    backgroundLoop.update();
    if(blobDetected == true){
        backgroundDetect.update();
    }
    
    if(kinect.isFrameNew()){
        mesh.clear();
        grayImage.setFromPixels(kinect.getDepthPixels());
        ofPixels & pix = grayImage.getPixels();
        int numPixels = pix.size();
        for(int i=0; i<numPixels; i++){
            if(pix[i] < grayNearThresh && pix[i] > grayFarThresh){
                pix[i] = 255;
            } else{
                pix[i] = 0;
            }
        }
        
        //update the cv images
        grayImage.flagImageChanged();
        
        // find contours which are between the size of 20 pixels and 1/3 the w*h pixels.
        // also, find holes is set to true so we will get interior contours as well...
//        contourFinder.findContours(grayImage, 50, (kinect.width*kinect.height)/2, 5, false);
//         cout << "number of blobs: " << contourFinder.nBlobs << endl;
        
        contourFinder.findContours(grayImage);
        
        int width = kinect.width/skip;
        int height = kinect.height/skip;
        for(int y=0; y<height; y++){
            for(int x=0; x<width; x++){
//                if(kinect.getDistanceAt(x, y) > 0){
                glm::vec3 vertex = kinect.getWorldCoordinateAt(x *skip, y*skip);
                ofColor kinCol = kinect.getColorAt(x*skip, y*skip);
                glm::vec3 normal = glm::vec3(0,0,0);
              
                vertex.z *= -1;
                vertex.z += 1000;
                
                if(vertex.z > nearThresh || vertex.z <farThresh){
                    kinCol.a = 0;
                }
                
                    mesh.addVertex(vertex);
                    mesh.addNormal(normal);
                    mesh.addColor(kinCol);
//        }
        }
        }
    for(int y = 0; y<height-1; y++){
        for(int x = 0; x<width-1; x++){
            
            mesh.addIndex(x+y*width);         // 0
            mesh.addIndex((x+1)+y*width);     // 1
            mesh.addIndex(x+(y+1)*width);     // 10
            
            mesh.addIndex((x+1)+y*width);     // 1
            mesh.addIndex((x+1)+(y+1)*width); // 11
            mesh.addIndex(x+(y+1)*width);     // 10
        }
    }
        
    }

    updateNormals();
    
}

//--------------------------------------------------------------
void ofApp::draw(){
    
    if(blobDetected == false){
        backgroundDetect.setPaused(true);
        backgroundLoop.play();
        backgroundLoop.draw(0, 0, ofGetWidth(), ofGetHeight());
    }
    else if(blobDetected == true){
        //        backgroundLoop.setPaused(true);
        backgroundLoop.stop();
        backgroundDetect.play();
        backgroundDetect.draw(0,0, ofGetWidth(), ofGetHeight());
        if(backgroundDetect.getIsMovieDone()){
            backgroundDetect.setPaused(true);
        }
    }
    ofBackground(0);
    
//    ofPushStyle();
//    ofBackground(0, 0, 0);
//    grayImage.draw(10, 320, 400, 300);
//    ofScale(-1,-1,1);
//    contourFinder.draw(0, 0, ofGetWidth(), ofGetHeight());
//    ofPopStyle();

    ofPushMatrix();
    ofPushStyle();
    cam.begin();
    ofEnableDepthTest();
    ofTranslate(0, 0, 0);
    ofScale(-1,-1, 1);
//    mesh.drawFaces();
    
    //dark brown
    ofSetColor(78, 54, 41);
    
    //forest green
//    ofSetColor(33, 68, 2);

    if (contourFinder.size() > 0){
        blobDetected = true;
    }
    else{
        blobDetected = false;
    }
    
//    for(int j = 0; j< contourFinder.size(); j++){
//        int id = contourFinder.getLabel(j);
//        ofPushStyle();
//        ofSetColor(0);
//        ofFill();
//        mesh.drawFaces();
//        ofPopStyle();
//        normalLines.draw();
//    }
    
    if (contourFinder.size()){
        normalLines.draw();
    }
    
    ofPopStyle();
    cam.end();
    ofDisableDepthTest();
    ofPopMatrix();
    
}

//--------------------------------------------------------------
void ofApp::keyPressed(int key){
    switch(key){
        case OF_KEY_UP:
            angle++;
            if(angle>30) angle=30;
            kinect.setCameraTiltAngle(angle);
            break;
            
        case OF_KEY_DOWN:
            angle--;
            if(angle<-30) angle=-30;
            kinect.setCameraTiltAngle(angle);
            break;
    }
    
    if(key == 'f'){
        ofToggleFullscreen();
    }
    
    //threshold values
    if(key == 'q'){
        farThresh +=50;
    }
    else    if(key == 'w'){
        farThresh -=50;
    }
    
    else    if(key == 'e'){
        nearThresh +=50;
    }
    
    else    if(key == 'r'){
        nearThresh -=50;
    }
    
    cout<<nearThresh<<endl;
    cout<<farThresh<<endl;
    

}

//--------------------------------------------------------------
void ofApp::updateNormals(){
    
    
    auto& verts = mesh.getVertices();
    normalLines.clear();
    normalLines.setMode(OF_PRIMITIVE_LINES);
    int normalIndex = 0;
    
    
    //generate vertex normals:
    //based on ofxMeshUtils addon:
    // https://github.com/ofZach/ofxMeshUtils/blob/master/src/ofxMeshUtils.cpp#L32-L58
  
    
    // reset current normals
    mesh.clearNormals();
    mesh.addNormals( vector<glm::vec3>(verts.size()) );
    // 1 normal per vertex
    
    // loop through the triangles
    for( int i=0; i+2 < mesh.getIndices().size(); i+=3 )
    {
        // 3 vertices per triangle
        const int va = mesh.getIndices()[i];
        const int vb = mesh.getIndices()[i+1];
        const int vc = mesh.getIndices()[i+2];
    
        ofFloatColor ca = mesh.getColors()[va];
        ofFloatColor cb = mesh.getColors()[vb];
        ofFloatColor cc = mesh.getColors()[vc];
        
        if (ca.a > 0 && ca.a > 0 && ca.a > 0){
 
        // calculate triangle face normal:
        // cross product of two edges
        glm::vec3 e1 = verts[va] - verts[vb];
        glm::vec3 e2 = verts[vc] - verts[vb];
        glm::vec3 nml = glm::normalize(glm::cross(e2,e1));
    
    
        // depending on your clockwise / winding order, you might want to reverse the e2 / e1 above if your normals are flipped.
        
        // distribute face normal to 3 vertex normals:
        mesh.getNormals()[va] += nml;    // *add* face normal
        mesh.getNormals()[vb] += nml;    // to each vertex -
        mesh.getNormals()[vc] += nml;    // averages faces
    
        // extra --
        // store the face normal as a line
        // in normalLines mesh for drawing
        // -- normalLines mesh uses OF_PRIMITIVE_LINES mode
        
        glm::vec3 faceCenter = getTriangleCenter(verts[va],verts[vb],verts[vc]);
        
        normalLength = ofRandom(10,40);
        glm::vec3 normalEnd = faceCenter + nml * normalLength;
       
        // 1 line = 2 vertices, 2 indices
        normalLines.addVertex(faceCenter);
        normalLines.addVertex(normalEnd);
        
        normalLines.addIndex(normalIndex);
        normalLines.addIndex(normalIndex+1);
            
        normalIndex += 2;   // increment
        
        }
    }
}
//--------------------------------------------------------------

glm::vec3 ofApp::getTriangleCenter(glm::vec3 a, glm::vec3 b, glm::vec3 c)
{
    // center of a triangle:
    // 2/3 from point to center of opposite edge
    glm::vec3 edgePt = (b + c) * 0.5;     // middle b_c edge
    return a * .333f + edgePt * .667f;    // lerp 2/3 a --> edgePt
}

//--------------------------------------------------------------

void ofApp::keyReleased(int key){

}

//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y ){

}

//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button){

}

//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y){

}

//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y){

}

//--------------------------------------------------------------
void ofApp::windowResized(int w, int h){

}

//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg){

}

//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo){ 

}

#2

at first glance it looks like the IR shadow of the subject onto the background and errors in mesh triangulation at the edges where there is null data returned…
if I get time I will run up the code and look through and see if I can find the issue…


#3

Yeah I agree with danb.

One thing I found useful with using the Kinect was to simplify the openCV image by using .erode() and .blur() - it results in a less exact blob, but definitely a smoother one that can feel more natural


#4

Hi, I think that the problem has to do with the mesh indexing. I’d rather create a ofVboMesh, set mode to OF_PRIMITIVES_LINES and initialize it with grid of lines, which needs to be done just once.

mesh.setMode(OF_PRIMITIVE_LINES);
for(int y=0; y<kinect.height; y++){
    for(int x=0; x<kinect.width; x++){
        mesh.addVertex({x, y , 0});
        mesh.addVertex({x, y , 1});
    }
}

Then pass this mesh to a vert shader, along with the kinect depth image, and calculate the normal from the depth image, which you can also threshold there. This should be much faster and easier to deal with.