Creating a polygon shape in Box2D with Face outline

Hey all,

I’m trying to create a polygon body in Box2D using the Face outline that I get from ofxFaceTracker2. But I’m unable to map the texture of the face correctly. My strategy has been this:

1: Get the FACE_OUTLINE polyline using the landmarks object.
2: Create a polygon object in Box2D with this polyline’s vertices.
3: Create a new mesh with texture coordinates from the polyline in Step 1
4: Use the points of the Polygon obtained in Step 2 to update the points of the mesh.

I had to change the value of the constant #define b2_maxPolygonVertices 100 in box2D code, else the getPoints() on the polygon object returned only limited points to map the mesh vertices to.

Here is the code. My texture seems to not be mapped correctly, but the mesh is successfully created. Any thoughts on how I can get the face’s outline mapped to a polygon object properly.

Here are some images.

image image image

ofApp.h

class ofApp : public ofBaseApp{

	public:
		void setup();
		void update();
		void draw();

		void keyPressed(int key);

    ofVideoGrabber grabber;
    ofxFaceTracker2 tracker;
    std::vector<ofRectangle> boundingBoxes;
    ofMesh mesh; 
  
    bool takeSnapshot;
    bool createBody;
    bool hideGrabber; 
  
    ofImage dst;
    ofPixels crop;
    ofxBox2d box2d;
  
    ofPolyline facePoly; 
    std::vector<std::shared_ptr<ofxBox2dPolygon>> polys;
  
    std::vector<ofMesh> bodyMesh; 

};

ofApp.cpp

//--------------------------------------------------------------
void ofApp::setup(){
  ofBackground(0);
  grabber.setup(200, 200);
  tracker.setup();
  
  box2d.init();
  box2d.setGravity(0, 3);
  box2d.createGround();
  box2d.setFPS(60);
  
  takeSnapshot = false;
  createBody = false;
  hideGrabber = false;
}

//--------------------------------------------------------------
void ofApp::update(){
  box2d.update();
  
  grabber.update();
  
  if (grabber.isFrameNew()) {
    boundingBoxes.clear();
    tracker.update(grabber);
    
     // Get the bounding boxes.
    if (tracker.size() > 0){
      auto& instances = tracker.getInstances();
      for (auto &i : instances) {
        auto rect = i.getBoundingBox();
        boundingBoxes.push_back(rect);
      }
      
      ofxFaceTracker2Landmarks landmarks = instances[0].getLandmarks();
      auto points = landmarks.getCvImagePoints();
      mesh = landmarks.getMesh(points);
      
      facePoly = landmarks.getImageFeature(ofxFaceTracker2Landmarks::Feature::FACE_OUTLINE);
//      std::cout << "Original:" << facePoly.getVertices().size() << "\n";
//      facePoly = facePoly.getSmoothed(ofRandom(1, 10), ofRandom(0, 1));
//      std::cout << "Resampled:" <<facePoly.getVertices().size() << "\n";
//      facePoly = facePoly.getResampledBySpacing(10);
    }
  }
  
  if (createBody && facePoly.getVertices().size() > 0) {
      // Create mesh and also soft polygon shape.
      auto vertices = facePoly.getVertices();
      cout << "Size of vertices: " << vertices.size() << "\n";
      auto polyShape = std::make_shared<ofxBox2dPolygon>();
    
      polyShape -> addVertices(vertices);
      polyShape -> setPhysics(0.2, 0.4, 0.8);
      polyShape -> create(box2d.getWorld());
      polys.push_back(polyShape);
    
      // Create empty mesh and push
      ofMesh m;
      m.setMode(OF_PRIMITIVE_TRIANGLE_STRIP);

      // Add polyline centroid
      auto center = polyShape -> getCentroid2D();
      m.addTexCoord(center);
      // Add polyline vertices
      for (auto v : vertices) {
        m.addTexCoord(v); // Add tex coordinates for the mesh.
      }
      bodyMesh.push_back(m);
    
      createBody = false;
  }
}

//--------------------------------------------------------------
void ofApp::draw(){
  
  // Grabber hiding.
  if (!hideGrabber) {
    grabber.draw(0, 0);
    for (auto& b : boundingBoxes) {
      ofPushStyle();
        ofNoFill();
        ofSetColor(ofColor::red);
        ofDrawRectangle(b.getX(), b.getY(), b.getWidth(), b.getHeight());
      ofPopStyle();
    }
    if (mesh.getVertices().size() > 0) {
      ofSetColor(ofColor::white);
      mesh.drawWireframe();
    }
  }
  
  // Mesh from Polygon
  if (polys.size() > 0) {
    for (int i = 0; i < polys.size(); i++) {
      bodyMesh[i].clearVertices();
      
      // Create mesh from box2d polygon
      auto &vertices = polys[i]->getPoints();
      ofPoint center = polys[i]->getCentroid2D();

      for (int j = 0; j < vertices.size(); j++) {
        bodyMesh[i].addVertex(center);
        bodyMesh[i].addVertex(vertices[j]);
      }

      bodyMesh[i].addVertex(center);
      bodyMesh[i].addVertex(vertices.front());
    
      // Draw the new mesh here. 
      grabber.getTexture().bind();
        bodyMesh[i].drawWireframe();
      grabber.getTexture().unbind();
    }
  }
}

Your texture coordinates are wrong, you should map them to the image of the face inside red square.the left bottom corner of the image has
coordinates 0,0, the top right 1,1 (assuming you are working with normalized coordinates).

Thanks @edapx, for your response. I understand why that would be a good way to go. I guess I wanted to know if it’s possible to map them to the texture coordinates coming from the face outline of the face’s mesh.