Collision Detection with (and other questions about) ofxBox2d

Before I changed anything it was “sfx/”+ofToString(i)+".mp3);

and I got the same error, except

could not load "data\sfx/0.mp3"

so I changed the fowardslash to a backslash so that both slashes would at least be pointing in the same direction, lol. Not sure what I’d need to edit to change the first backslash to a fowardslash.

Also just now noticed that SoundData is a class defined in the header file- whoops.

the strange thing is that it is prepending the “data/” to it. try the following

auto filepath = ofToDataPath(“sfx/”+ofToString(i)+".mp3", true);
cout << filepath << endl;// Just to check what is the filepath
sound[i].load(filepath);

if it still does not work try instead the following

auto filepath = ofToDataPath(ofFilePath::join("sfx", ofToString(i)+".mp3"),true);

let me know if this works

1 Like

Hey, that helped, thank you. When importing the file into the oF project launcher I had messed up something with the file naming and path. Got it working now!

And for the record, when I print the file path it displays everything with backslashes, ie.

C:\Users\selli\openFrameworks\addons\ofxBox2d\example-ContactListener\bin\data\sfx\0.mp3
1 Like

@roymacdonald So it’s working now, but I have some more questions and figured I should put them here instead of making a new thread for each one.

  1. Why is it that when accessing a circle the syntax is ofxBox2dCircle->getVelocity() but when I’m accessing it via the ContactListener it ends up being ofxBox2dContactArgs->GetBody()->GetLinearVelocity()?

  2. I’d like to find a way to transfer momentum from a contour edge into shapes that are touching the edge- as would happen when throwing or launching an object in the physical world. I guess I would try to ofxBox2dCircle->setVelocity(). ofxBox2dEdge doesn’t seem to have any velocity values that I could use for that though. So I guess I could calculate the velocity of the circle manually based on the circle’s movement up until the contact with the edge is ended and then set this to be the circle’s new velocity… This seems like it’d be janky though because of a little bit of contact still counts as contact. Any tips before I try that?

Hi, so ofxBox2D uses internaly the box2D library, which has a lot of different objects within it. (you might want to take a look inside of each of the ofxBox2D classes). So ofxBox2dCircle->getVelocity() is a function of the ofxBox2dCircle class which has that syntax, while ofxBox2dContactArgs->GetBody()->GetLinearVelocity() is accessing an internal box2d object of the ofxBox2dContactArgs class thus it uses the syntax of box2d.

Box2d has several internal objects as fixtures, bodies, shapes, etc which all play a different role and ofxBox2d has made some classes that already use these in order to make it easier to use, yet, as you still can access all of Box2d’s classes and functions you can use it to its full extent and you are not limited to what ofxBox2d has. Take a look at box2d’s documentation. it is quite good and has good examples.

In box 2d, among other classes you have fixtures, bodies and shapes.
A fixture contains a body and a shape.
A body is which contains the objects physical properies, like mass, speed, etc.
A shape has the shape of the object, which will define how it gets draw.

So I would guess that the way to transfer momentum is through the contact listeners. the ofxBox2dContactArgs class only contains two pointers, pointing to each fixtures being in contact. If you have multiple objects touching the contact callback function will get triggered each time a pair of objects make contact.

So, In order to transfer momentum, on each contact callback you can access the each fixture’s body and from it get its mass and velocity (linear) and from that work out the math, as momentum = mass*velocity. and that should be it.

1 Like

Ah, I see! I didn’t know about Box2d even though I was using it. If they use the Box2d syntax for contacts, I wonder why they didn’t just use it for everything, i.e. instead of creating the getVelocity() method… I guess it makes the syntax cleaner to be able to just type ofxBox2dCircle->getVelocity()

So the contour edges always have 0 velocity and the reason for that is that, every frame, I’m creating a new ofxBox2dEdge based on input of my camera (Kinect depth camera) and deleting the old edge:

void ofApp::createContourEdge(ofPolyline polyline) {
	delete m_contourEdge;
	m_contourEdge = new ofxBox2dEdge();
	m_contourEdge->addVertexes(polyline);
	m_contourEdge->create(m_box2d.getWorld());
	m_contourEdge->resize(m_size);
	m_contourEdge->draw();
}

Now that I think about it, I’m not even sure how to calculate the velocity of the edge manually, since the position of the edge isn’t changing that much- only the shape is (when I move my arms up and down).

Maybe using ofxKinectForWindows2 like you suggested to me before would allow me to detect arms as contours separate from the rest of the body? Then I could calculate their velocity better… maybe.

Hi,
I see, I guess that the edge will not have a velocity, specially if you are setting it on each frame.
Although I think that collisions would still work although without you being able to find the momentum, yet momentum transfer would be imposible. What I think would allow you to find the velocity is to keep the previous frame body contour as an ofPolyline. Then when a collision happens you find the point at where it happened and call in the current polyline [getClosestPoint](https://openframeworks.cc//documentation/graphics/ofPolyline/#!show_getClosestPoint) then you can call the same function but in the previous frame polyline, and you should get something like the distance traveled in a frame, thus velocity.

Using kinect v2 would automatically give you the contour for a body but not sure if it is isolated from the rest of the body.

Yeah, that’s a good idea. The only problem is that, because of the erraticness of the kinect camera’s image, each new polyline’s points are in a different location. I tested it by drawing a circle every frame

ofVec2f point = ofVec2f(polyline.getPointAtIndexInterpolated(0.0).x, polyline.getPointAtIndexInterpolated(0.0).y);
	ofSetColor(ofColor::green);
	ofDrawCircle(point, 10);

Generally, the circle stays in the same general area, erratically moving in a smaller area, but sometimes it makes big crazy leaps too. This’ll probably mess up the velocity measurements, but I’ll try it anyway and see how it goes.

sure. then you will need to smooth it out.
First and easiest way is to use ofPolyline´s function getResampledByCount. The trick is to always resample by the same amount so then you can have a 1 to 1 relationship between each frame´s polyline points.
You can then also try ofPolyline´s getSmoothed
An important thing to do is to have a consistent first point of the polylines. It could be either, the lowest point or get the first point of the current polyline and find in the previous frame´s polyline which point is the closest to that one. You will need to try which performs best.

Last but not least, you will need to apply a temporal filter, which should smooth out the shakiness of the shapes. What you do is, instead of saving just the previous frame polyline you save a certain amount of previous frames polylines, say 5 or 10. the more frames, the smoother it will be but at the same time it will feel less responsive. you need to find the sweet spot.
As you already found the first vertex for each polyline there will be a relationship between all of this, so you can just loop through all. What you do is that you just get the average point of all of these, which becomes your new point. You could also do a weighted average, where the current point is multiplied by a certain factor and the remaining ones by another factor. Or you can have a running average. There are lots of options. you need to test which works best, but my bet is that the simplest, just averaging should work nicely.

tip: instead of using an std::vector to store each past frame polyline you can use an std::deque which behaves in a similar way as the vector but allows you to insert and remove elements from both begining and end of it, as opposed to vector which only allows you to do so at its end.

Thus, it should look something like.


std::deque<ofPolyline> polys;//class member

// on setup or when you want to change the amount of polylines stored
polys.resize(10);

// on each frame
ofPolyline newFramePoly;// the raw contour from kinect.
newFramePoly = newFramePoly.getResampledByCount(100);

// find the first point on newFramePoly.
size_t first_index = 0;
float min_dist =  std::numeric_limits<float>::max();// start as the larges number that a float can represent. A hard coded large number should work too, but this covers all possible cases
for(size_t i = 0; i < newFramePoly.size(); i++){
    //implement here how to find that point.
// use distance2 to save a sqrt and make it run faster. 
// as the contour comes from an image, I assume that its vertices are in pixel "units" thus all are positive. We will simply find the closest one to the origin.
// This might not be an ideal algorith but it should work. you should try different methods.
    auto d = glm::distance2(newFramePoly[i], {0,0,0});
    if(d < min_dist){
        min_dist = d;
        first_index = i;
    }
}

if(first_index > 0){
    vector<glm::vec3> rearangedVerts;
    rearangedVerts.insert(rearangedVerts.begin(),  newFramePoly.begin() + first_index, newFramePoly.end());
    rearangedVerts.insert(rearangedVerts.end(), newFramePoly.begin(), newFramePoly.begin() + first_index);

    ofPolyline rearangedPoly;
    rearangedPoly.addVertices(rearangedVerts);
    polys.push_front(rearangedPoly);// add the new frame at the beginning, which moves all the others one position. 
}else{
    polys.push_front(newFramePoly);// add the new frame at the beginning, which moves all the others one position. 
}


polys.pop_back();// removes the last one, so the number of elements remains the same.

ofPolyline averaged;
//numVertices is the number you passed to getResampledByCount
for(size_t i = 0; i < numVertices; i++){
    glm::vec3 avg;
    for(size_t j= 0; j < polys.size(); j++){
        avg += polys[j][i];
    }
    avg /= polys.size();
   averaged.addVertex(avg);
}

then you use averaged with box2d.
I think that this should work. I just coded it here in the forum text box out of the top of my head so there might be some syntax errors, but the main idea is there.

I hope this is useful.
Let me know how it works

Hey, thanks for the detailed write-up and code example. I worked through it today to understand everything and, when I finally thought I understood the logic of it, I went to test it and would immediately get a crash with this error message in a pop-up window. I don’t know how to set up debugging in Visual Studio in a more helpful way (tried before with Unity and couldn’t figure it out), so I just commented things out until I found the line that was causing the problem, which was

avg += m_polys[j][i];

To fix this I prepopulated all the deque’s polylines with vertices in start():

for (int i = 0; i < m_polys.size(); i++)
	{
		for (int j = 0; j < m_polylinePointCount; j++)
		{
			m_polys[i].addVertex(0, 0, 0);
		}
	}

and when I ran this, instead of crashing immediately, it ran for a second or two before crashing with the same error. Confused, I printed the size of the ofPolyline derived from the Kinect after resampling

polyline = polyline.getResampledByCount(m_polylinePointCount);
cout << polyline.size() << endl;

and this showed that, despite use of getResampledByCount(), the amount of points varied- it would sometimes be one less than m_polylinePointCount. I don’t know what to do about that, so as a hack, I just set it up to ignore the last point during the averaging loops.

The result is… really weird! Here’s a video of it.

Red curve: polyline drawn directly from the ContourFinder
Green curve: “smoothed” contour used for the ofBox2dEdge
Green circle: 0th point of the smoothed polyline

Also, here are the draw(), createContourEdge() and smoothPoly() functions of my code- the latter of which should closely resemble your code, unless I made some dumb mistake… Not sure how to proceed.

void ofApp::draw() {
	ofPushMatrix();
		ofScale(m_size, m_size);
		ofSetRectMode(OF_RECTMODE_CORNER);
		ofSetColor(255);

		m_kinect.draw(0, 0);

		for (ofPolyline const& p : m_contour.getPolylines()) 
		{
			ofSetLineWidth(5);
			ofSetColor(ofColor(255,0,0));
			ofNoFill();
			p.draw();

			ofSetLineWidth(1);
			ofSetColor(0, 0, 0);
			ofFill(); 
			//Tried using p.getSmoothed() here, but that doesn't seem to address the main problems.
			ofPolyline smoothedPoly = smoothPoly(p); 			
			createContourEdge(smoothedPoly);
		}		
		ofSetColor(255);
		for (auto circle : m_circles)
		{
			circle->draw();
		}
        panel.getPosition().y / size);
	ofPopMatrix();
	m_panel.draw();
}
//...
ofPolyline ofApp::smoothPoly(ofPolyline polyline)
{
	polyline = polyline.getResampledByCount(m_polylinePointCount + 1); //the +1 is the hack
	//cout << polyline.size() << endl;
	size_t first_index = 0;
	float min_dist = std::numeric_limits<float>::max();
	for (size_t i = 0; i < polyline.size(); ++i)
	{
		auto d = glm::distance2(polyline[i], { 0,0,0 });
		if (d < min_dist)
		{
			min_dist = d;
			first_index = i;
		}
	}

	if (first_index > 0)
	{
		vector<glm::vec3> rearrangedVerts;
		rearrangedVerts.insert(rearrangedVerts.begin(), polyline.begin() + first_index, polyline.end());
		rearrangedVerts.insert(rearrangedVerts.end(), polyline.begin(), polyline.begin() + first_index);
		ofPolyline rearrangedPoly;
		rearrangedPoly.addVertices(rearrangedVerts);
		m_polys.push_front(rearrangedPoly);
	}
	else
	{
		m_polys.push_front(polyline);
	}
	m_polys.pop_back();

	ofPolyline averaged;
	for (size_t i = 0; i < m_polylinePointCount; ++i)
	{
		glm::vec3 avg;
		for (size_t j = 0; j < m_polys.size(); ++j)
		{
			avg += m_polys[j][i];
		}		
		avg /= m_polys.size();
		averaged.addVertex(avg);
	}
	return averaged;
}

void ofApp::createContourEdge(ofPolyline polyline) {

	delete m_contourEdge;
		
	ofVec2f point = ofVec2f(polyline.getPointAtIndexInterpolated(0.0).x, polyline.getPointAtIndexInterpolated(0.0).y);
	ofSetColor(ofColor::green);
	ofDrawCircle(point, 10);
	m_contourEdge = new ofxBox2dEdge();
	m_contourEdge->addVertexes(polyline);
	m_contourEdge->create(m_box2d.getWorld());
	m_contourEdge->resize(m_size);
	m_contourEdge->draw();
}

Hi,
Why don’t you add an if statement to check the polyline’s size as it follows?

for(size_t i = 0; i < numVertices; i++){
    glm::vec3 avg;
    for(size_t j= 0; j < polys.size(); j++){
        if(i < polys[j].size()){
            avg += polys[j][i];
        }
    }
    avg /= polys.size();
   averaged.addVertex(avg);
}

running the averaging and smoothing in the draw function. It should be done during update, right after you receive the kinect data and you get m_contour updated. It might not be the cause of the problems but it will improve performance.

The weird behaviour in the video you sent might be because of the first vertex being jumpy and the way that the polylines are getting averaged in time. Can you try not averaging and see what happens? maybe a running average might work better.

also, in the following code:

auto d = glm::distance2(polyline[i], { 0,0,0 });

it might be better if the distance is checked agains the first vertex of the last frame instead of a fixed value.

glm::vec3 first_vert = {0,0,0};
if(m_polys.size() > 0 && m_polys[0].size() > 0){
first_vert = m_polys[0].getVertices()[0];
}
size_t first_index = 0;
	float min_dist = std::numeric_limits<float>::max();
	for (size_t i = 0; i < polyline.size(); ++i)
	{
		auto d = glm::distance2(polyline[i],first_vert);
		if (d < min_dist)
		{
			min_dist = d;
			first_index = i;
		}
	}

I think that the sca;ing behaviour is because you prealocated m_polys. because of it, the average includes several vertices at 0,0,0. dont do the pre allocation.

Hey @s_e_p I recalled about this

take a look at what is going on in https://github.com/ofZach/sfpcContourProject/blob/master/contourApp/src/input/contourTracker.cpp
It has a scene manager but it probably already does what you are trying to achieve.

Hello, sorry for the late reply. I was busy and then got COVID… so coming back to this after four weeks… hope you’re still around.

Attached is are the src files for the project. I know it’s better form to share only the smallest section necessary of a code, but you seem capable of navigating it quickly and you might find some problem I’d have neglected to include in the reduced version. If it really is too big, let me know and I’ll cut it down.

I tried the suggestions in your penultimate post in smoothPoly(). These didn’t seem to help. I then used the code from the ofZach code you linked in your last post as as analyze(). The two first things I don’t understand about this code are

  1. why the polyline is resampled first to 180 and then to 100.

  2. what does this line of code do:

avgLen += (resampled[(j + i) % 180] - m_prevFrame[j]).length() / 180.0;

I’m unclear especially on what the ‘-’ here does.

The result of using analyze() is an accurate contour that corresponds to my body shape (no weird shrinking or anything), but the green circle representing the 0th polyline-point jumps around and it still goes to my hand if I raise my hand above my head.

running the averaging and smoothing in the draw function. It should be done during update, right after you receive the kinect data and you get m_contour updated. It might not be the cause of the problems but it will improve performance.

I was starting to move the the averaging and smoothing to the update function, when I realized that I’m using a for loop to go through all detected contours in draw(). Despite this, I’m only using one vector (m_polys) for everything. Is this an issue (except when there’s only one detected contour)? Should I be using a vector of vectors?

Sorry if I’m not making sense or have forgotten to implement something you previously wrote.
Contour_src.zip (4.7 KB)

Hi,
I am sorry to hear that you got COVID. I hope you have recovered well.
Good to know that zach´s code works.

I dont know. those are just arbitrary numbers, most probably found by trial and error.
There is a lot of room for improvement there.

This part is trying to find the first vertex by comparing to the previous frame and find the combination that give the lowest difference. the ‘-’ sign is substracting the vertex of the previous frame to the current in order to find the distance between these.
as you can see there, it uses a nested for loop, which is what makes it slow. It can be improved somehow in order to find the first point.

is this what you want?

what is not working on your current code?
I dont have a kinect in ordet to test it

I am sorry to hear that you got COVID. I hope you have recovered well.

Thank you! I feel kind of tired still… but that may just be normal for me.

This part is trying to find the first vertex by comparing to the previous frame and find the combination that give the lowest difference. the ‘-’ sign is substracting the vertex of the previous frame to the current in order to find the distance between these.

But if that returns a vertex, how is it being +='d to a float value?

is this what you want?

what is not working on your current code?

No, I’d want the points to stay in a similar position (on the body) as the last frame. So if the 0th point is on the head, it’d stay there even the hand is raised above the head.

Hi,

lol. If it is of any comfort for you, I have narcolepsy (to a quite severe degree)

I run your code using some silhoute videos (from the same repo that had zach’s code i shared) and there are several problems.

There is a problem there with the vector class, when it changed from ofVec3f to glm::vec3 as the default. The latter does not have much member functions, but it happens to have a length() function which always returns 3.

Then when you moved the analize() code into your app, you missed using the correct ofPolyline.

In short, the code in analize() that you have in your app does nothing at all, but it is due to some small errors in the copypaste process.


#pragma once
#include "ofMain.h"
#include "ofxBox2d.h"
#include "ofxGui.h"

#include "ofxCv.h"



class ofApp : public ofBaseApp {

public:
	void setup();
	void update();
	void draw();
	void exit();

	void keyPressed(int key);
	void keyReleased(int key);
	void mouseMoved(int x, int y);
	void mouseDragged(int x, int y, int button);
	void mousePressed(int x, int y, int button);
	void mouseReleased(int x, int y, int button);
	void mouseEntered(int x, int y);
	void mouseExited(int x, int y);
	void windowResized(int w, int h);
	void dragEvent(ofDragInfo dragInfo);
	void gotMessage(ofMessage msg);
	

	ofPolyline smoothPoly(const ofPolyline& polyline);
	void analyze(const ofPolyline& curFrame);
	void updateKinect(bool isDepthCam);
	void updateContours(bool isDepthCam);
	void drawKinect(bool isDepthCam);

	//void oscSendRoll
	
	float clip(float n, float lower, float upper);

	//ofPolyline m_curFrame;
	ofPolyline m_prevFrame;
	ofPolyline m_resampleSmoothed;


	ofxPanel m_panel; 
	ofParameter<int> m_threshold;
	ofParameter<int> m_minArea;
	ofParameter<int> m_maxArea; 
	ofParameter<float> m_nearClip;
	ofParameter<float> m_farClip;
	ofParameter<float> m_screenScaler;
	ofParameter<int> m_polylinePointCount;
	ofParameter<int> m_smoothingSize;
	ofParameter<float> m_smoothingShape;
    ofParameter<float> smoothingFactor = {"Smoothing Factor", 0.25, 0.0, 1.0};
    ofParameter<bool> bDrawSmoothed = {"Draw Smoothed", true};
	//int m_polylinePointCount = 104;
    ofVideoPlayer m_kinect;
	//ofVideoGrabber kinect;

	ofImage m_grayImage;
	ofxCv::ContourFinder m_contour; 
	int angle;
	int m_camWidth;
	int m_camHeight;

	std::deque<ofPolyline> m_polys;

	ofxBox2d m_box2d;
	vector<shared_ptr<ofxBox2dCircle>> m_circles;

    ofPolyline m_previousPoly;

	int m_size = 2;
	//int width, height;

	float m_screenWidth = 20; //*should* set this via oF but setting it manually for now 


    int movLoaded = 0;
    void loadMov(int movNum);
};

#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup()
{
	

	m_panel.setup("","", 20, 50);
	m_panel.add(m_threshold.set("threshold", 15, 0, 100));
	m_panel.add(m_minArea.set("minArea", 13, 0, 200));
	m_panel.add(m_maxArea.set("maxArea", 199, 0, 200));
//	m_panel.add(m_nearClip.set("near clip", 885.0, 500.0, 4000.0));
//	m_panel.add(m_farClip.set("far clip", 1045.0, 1000.0, 10000.0));
	m_panel.add(m_screenScaler.set("screen scale", 3.02, 0.5, 4.0));
	m_panel.add(m_polylinePointCount.set("polyline point count", 180, 0, 650));
	m_panel.add(m_smoothingSize.set("smoothing size", 2, 0, 30));
	m_panel.add(m_smoothingShape.set("smoothing shape", 0.0, 0.0, 1.0));
    m_panel.add(bDrawSmoothed);
    m_panel.add(smoothingFactor);
    
    loadMov(1);
    
	m_box2d.init();
	m_box2d.setGravity(0, 30);
	m_box2d.createGround(0.0f, m_kinect.getHeight(), m_kinect.getWidth(), m_kinect.getHeight());
	m_box2d.createBounds(ofRectangle(0, 0, m_kinect.getWidth(), m_kinect.getHeight()));
	m_box2d.enableEvents();

//	ofAddListener(m_box2d.contactStartEvents, this, &ofApp::contactStart);
//	ofAddListener(m_box2d.contactEndEvents, this, &ofApp::contactEnd);

	m_polys.resize(m_polylinePointCount);
	/*for (int i = 0; i < m_polys.size(); i++)
	{
		for (int j = 0; j < m_polylinePointCount; j++)
		{
			m_polys[i].addVertex(0, 0, 0);
		}
		//cout << m_polys[i].size() << endl;
	}*/

}

void ofApp::loadMov(int movNum){
    if(movNum >=0 && movNum <7){
        string mov = "movs/" +ofToString(movNum+1) +".mov";
        if(m_kinect.load(mov)){
            cout << "loaded : " << mov << endl;
            cout << "movNum : " << movNum << endl;
            movLoaded = movNum;
            m_kinect.setLoopState(OF_LOOP_NORMAL);
            m_kinect.play();
        }
    }
}

//--------------------------------------------------------------
void ofApp::update() {
	m_box2d.update(); 
	updateKinect(true); 
	updateContours(true); 
	
}

//--------------------------------------------------------------
void ofApp::draw() {
	ofPushMatrix();
		ofScale(m_size, m_size);
		ofSetRectMode(OF_RECTMODE_CORNER);
		ofSetColor(255);

		drawKinect(true);

		for (ofPolyline const& p : m_contour.getPolylines()) //THE PROBLEM...
		{
			ofSetLineWidth(5);
			ofSetColor(ofColor(255,0,0));
			ofNoFill();
			

//			ofSetLineWidth(1);
//			ofSetColor(0, 0, 0);
//			ofFill();
			ofPolyline s = p.getSmoothed(m_smoothingSize, m_smoothingShape);
//            if(bUseAnalyze){
//                ofPolyline smoothedPoly = analyze(s);
                
                
                
            if(bDrawSmoothed){
                analyze(s);
                m_resampleSmoothed.draw();
                if(m_resampleSmoothed.size()){
                    ofSetColor(ofColor:: yellow);
                    ofDrawCircle(m_resampleSmoothed.getVertices()[0], 5);
                }
            
            }else{
                p.draw();
                if(p.size()){
                    ofSetColor(ofColor:: yellow);
                    ofDrawCircle(p.getVertices()[0], 5);
                }
            }
                
//                createContourEdge(smoothedPoly);
//            }else{
//                ofPolyline smoothedPoly = smoothPoly(s);
//                createContourEdge(smoothedPoly);
//            }
			 			
			
			
		}
		//m_contourEdge->resize(m_size);
		//m_contourEdge->draw(); //...IS IN HERE.
		
		ofSetColor(255);
		for (auto circle : m_circles)
		{
			circle->draw();
		}
		//panel.setSize(panel.getWidth() / size, panel.getHeight() / size);
		//panel.setPosition(panel.getPosition().x / size, panel.getPosition().y / size);
	ofPopMatrix();
	m_panel.draw();
    
    ofDrawBitmapStringHighlight("FPS: " + ofToString(ofGetFrameRate()), 20, 20);
}

void ofApp::exit() {
//	m_kinect.setCameraTiltAngle(15.0);
	m_kinect.close();
}

float ofApp::clip(float n, float lower, float upper) {
	return std::max(lower, std::min(n, upper));
}


void ofApp::drawKinect(bool isDepthCam)
{
	
	m_kinect.draw(0, 0);
}

void ofApp::updateKinect(bool isDepthCam)
{
	
	m_kinect.update();
}

void ofApp::updateContours(bool isDepthCam)
{
	if (!m_kinect.isFrameNew())
	{
		return;
	}
	m_contour.setMinAreaRadius(m_minArea);
	m_contour.setMaxAreaRadius(m_maxArea);
//
//		m_grayImage.setFromPixels(m_kinect.getDepthPixels());
//		m_contour.findContours(m_grayImage);
//
	//m_contour.setTargetColor(m_trackingColor);
	m_contour.setThreshold(m_threshold);
	m_contour.findContours(m_kinect);
}

ofPolyline ofApp::smoothPoly(const ofPolyline& _polyline)
{
	auto polyline = _polyline.getResampledByCount(m_polylinePointCount /* + 1*/);
	//cout << polyline.size() << endl;
	glm::vec3 first_vert = { 0,0,0 };
	if (m_polys.size() > 0 && m_polys[0].size() > 0)
	{
		first_vert = m_polys[0].getVertices()[0];
	}
	size_t first_index = 0;
	float min_dist = std::numeric_limits<float>::max();
	for (size_t i = 0; i < polyline.size(); ++i)
	{
		auto d = glm::distance2(polyline[i], first_vert);
		if (d < min_dist)
		{
			min_dist = d;
			first_index = i;
		}
	}

	if (first_index > 0)
	{
		vector<glm::vec3> rearrangedVerts;
		rearrangedVerts.insert(rearrangedVerts.begin(), polyline.begin() + first_index, polyline.end());
		rearrangedVerts.insert(rearrangedVerts.end(), polyline.begin(), polyline.begin() + first_index);
		ofPolyline rearrangedPoly;
		rearrangedPoly.addVertices(rearrangedVerts);
		m_polys.push_front(rearrangedPoly);
	}
	else
	{
		m_polys.push_front(polyline);
	}
	m_polys.pop_back();

	ofPolyline averaged;
	//cout << m_polys.size() << endl;
	for (size_t i = 0; i < m_polylinePointCount; ++i)
	{
		glm::vec3 avg;
		for (size_t j = 0; j < m_polys.size(); ++j)
		{
			//cout << m_polys[j].size() << endl;
			//cout << m_polys[j][i] << endl;
			if (i < m_polys[j].size())
			{
				avg += m_polys[j][i];
			}
			//cout << "vertex[" << j << "][" << i << "] added" << endl;
		}		
		avg /= m_polys.size();
		averaged.addVertex(avg);
	}
	return averaged;
}


void ofApp::analyze(const ofPolyline& curFrame) {


//	ofPolyline line = curFrame;
    ofPolyline resampled = curFrame.getResampledByCount(m_polylinePointCount);

	// A++ if you fix getResampledByCount end points !
	//

//
//
	while (resampled.size() < m_polylinePointCount) {
		resampled.getVertices().push_back(resampled[resampled.size() - 1]);
	}
//
	if (m_prevFrame.size() > 0) {
//        cout << ",,,,,\n";
		// if you want to make this faster
		// you will get an A +
		// this is pretty slow :(
		// but it works.

		int smallestStart = -1;
		float smallestAvgLen = 10000000;

        float fCount = m_polylinePointCount.get();
        auto p_count = m_polylinePointCount.get();
		for (int i = 0; i < p_count; i++) {

			float avgLen = 0;
			for (int j = 0; j < p_count; j++) {
				avgLen += glm::distance2(resampled[(j + i) % p_count], m_prevFrame[j]) / fCount;
							//
			}
			if (avgLen < smallestAvgLen) {
				smallestAvgLen = avgLen;
				smallestStart = i;
			}

		}

		ofPolyline temp;
		for (int i = 0; i < p_count; i++) {
			temp.addVertex(resampled[(i + smallestStart) % p_count]);
		}
		resampled = temp;
	}
    m_prevFrame = resampled;
//	ofPolyline tempT = resampled.getResampledByCount(100);
//
//	while (tempT.size() < 100) {
//		tempT.getVertices().push_back(tempT[tempT.size() - 1]);
//	}

    ofPolyline & tempT = resampled;
	if (m_resampleSmoothed.size() == 0 || ofIsFloatEqual(smoothingFactor.get(), 1.0f)) {
		m_resampleSmoothed = tempT;
	}
	else {
		for (int i = 0; i < tempT.size(); i++) {
			m_resampleSmoothed[i] = (1.0 - smoothingFactor) * m_resampleSmoothed[i] + (smoothingFactor.get() * tempT[i]);
		}
	}

//if you want this function to return something it should be m_resampleSmoothed rather than resampled
    
//	return resampled;

}

//--------------------------------------------------------------
void ofApp::keyPressed(int key) {
	switch (key)
	{
	case 'c':
		m_circles.clear(); 
		break;
	case 'd':
		break;
	}
}

//--------------------------------------------------------------
void ofApp::keyReleased(int key) {
    if(key == OF_KEY_LEFT){
        loadMov(((movLoaded +1)%7));
    }else if(key == OF_KEY_RIGHT){
        loadMov(((movLoaded -1 + 7)%7));
    }
}

//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button) {

}

//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button) {
	/*auto circle = make_shared <ofxBox2dCircle>();
	circle->setPhysics(3.0, 0.5, 0.1);
	circle->setup(m_box2d.getWorld(), x / m_size, y / m_size, 32);
	m_circles.push_back(circle);*/
}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button) {

}

//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::windowResized(int w, int h) {

}

//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg) {

}

//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo) {

}

Using analize with the code above you get a quite stable first point.

Pull smoothingFactor all the way up to 1 if you want to have the polyline stuck to the silhouette.

I hope this helps.

lol. If it is of any comfort for you, I have narcolepsy (to a quite severe degree)

Ah, sorry to hear that. You do seem to be very productive in spite of it, as evidenced by your swiftness and thoroughness in helping others, so you’re doing something right.

OK, I’ve taken your code and set it to use the Kinect depth image instead of the video loader and here is the result.

Notes:

  1. It’s working! The circle no longer jumps when a hand goes above the head. There’s some jitteriness due to the kinect image being less stable than the test videos you were using, but the yellow circle stays in the same general area unless the shape of the detected contours changes drastically. Within that general area it still jumps around a bit… should I add another layer of averaging/lo-pass filter or is that what Smoothing Factor is supposed to do?

  2. Unless I make the Smoothing Factor 1, the size of the contour is continually shrinking and growing. (Making it 0 will freeze it in whatever size/shape it was up until it was 0- often a smaller shape)

  3. Hair is a problem for the kinect. Taking the beanie off throws it into chaos (particularly when Smoothing Factor is not 1). Guess I need to get a haircut before my performance.

Just in case, here’s the code of the video I linked.

#pragma once
#include "ofMain.h"
#include "ofxBox2d.h"
#include "ofxGui.h"
#include "ofxKinect.h"
#include "ofxCv.h"



class ofApp : public ofBaseApp {

public:
	void setup();
	void update();
	void draw();
	void exit();

	void keyPressed(int key);
	void keyReleased(int key);
	void mouseMoved(int x, int y);
	void mouseDragged(int x, int y, int button);
	void mousePressed(int x, int y, int button);
	void mouseReleased(int x, int y, int button);
	void mouseEntered(int x, int y);
	void mouseExited(int x, int y);
	void windowResized(int w, int h);
	void dragEvent(ofDragInfo dragInfo);
	void gotMessage(ofMessage msg);


	ofPolyline smoothPoly(const ofPolyline& polyline);
	void analyze(const ofPolyline& curFrame);
	void updateKinect(bool isDepthCam);
	void updateContours(bool isDepthCam);
	void drawKinect(bool isDepthCam);

	//void oscSendRoll

	float clip(float n, float lower, float upper);

	//ofPolyline m_curFrame;
	ofPolyline m_prevFrame;
	ofPolyline m_resampleSmoothed;


	ofxPanel m_panel;
	ofParameter<int> m_threshold;
	ofParameter<int> m_minArea;
	ofParameter<int> m_maxArea;
	ofParameter<float> m_nearClip;
	ofParameter<float> m_farClip;
	ofParameter<float> m_screenScaler;
	ofParameter<int> m_polylinePointCount;
	ofParameter<int> m_smoothingSize;
	ofParameter<float> m_smoothingShape;
	ofParameter<float> smoothingFactor = { "Smoothing Factor", 0.25, 0.0, 1.0 };
	ofParameter<bool> bDrawSmoothed = { "Draw Smoothed", true };
	//int m_polylinePointCount = 104;
	//ofVideoPlayer m_kinect;
	//ofVideoGrabber kinect;
	ofxKinect m_kinect;

	ofImage m_grayImage;
	ofxCv::ContourFinder m_contour;
	int angle;
	int m_camWidth;
	int m_camHeight;

	std::deque<ofPolyline> m_polys;

	ofxBox2d m_box2d;
	vector<shared_ptr<ofxBox2dCircle>> m_circles;

	ofPolyline m_previousPoly;

	int m_size = 2;
	//int width, height;

	float m_screenWidth = 20; //*should* set this via oF but setting it manually for now 


	int movLoaded = 0;
	//void loadMov(int movNum);
};
#include "ofApp.h"

//--------------------------------------------------------------
void ofApp::setup()
{


	m_panel.setup("", "", 20, 50);
	m_panel.add(m_threshold.set("threshold", 15, 0, 100));
	m_panel.add(m_minArea.set("minArea", 13, 0, 200));
	m_panel.add(m_maxArea.set("maxArea", 199, 0, 200));
	m_panel.add(m_nearClip.set("near clip", 885.0, 500.0, 4000.0));
	m_panel.add(m_farClip.set("far clip", 1045.0, 1000.0, 10000.0));
	m_panel.add(m_screenScaler.set("screen scale", 3.02, 0.5, 4.0));
	m_panel.add(m_polylinePointCount.set("polyline point count", 250, 0, 650));
	m_panel.add(m_smoothingSize.set("smoothing size", 2, 0, 30));
	m_panel.add(m_smoothingShape.set("smoothing shape", 0.0, 0.0, 1.0));
	m_panel.add(bDrawSmoothed);
	m_panel.add(smoothingFactor);

	//loadMov(1);

	m_box2d.init();
	m_box2d.setGravity(0, 30);
	m_box2d.createGround(0.0f, m_kinect.getHeight(), m_kinect.getWidth(), m_kinect.getHeight());
	m_box2d.createBounds(ofRectangle(0, 0, m_kinect.getWidth(), m_kinect.getHeight()));
	m_box2d.enableEvents();

	//	ofAddListener(m_box2d.contactStartEvents, this, &ofApp::contactStart);
	//	ofAddListener(m_box2d.contactEndEvents, this, &ofApp::contactEnd);

	m_polys.resize(m_polylinePointCount);
	/*for (int i = 0; i < m_polys.size(); i++)
	{
		for (int j = 0; j < m_polylinePointCount; j++)
		{
			m_polys[i].addVertex(0, 0, 0);
		}
		//cout << m_polys[i].size() << endl;
	}*/
	m_kinect.setRegistration(true);
	m_kinect.init();
	m_kinect.open();//necessary?

	m_grayImage.allocate(m_kinect.getWidth(), m_kinect.getHeight(), OF_IMAGE_GRAYSCALE);
}

	
//void ofApp::loadMov(int movNum) {
//	if (movNum >= 0 && movNum < 7) {
//		string mov = "sil.mov";
//		if (m_kinect.load(mov)) {
//			cout << "loaded : " << mov << endl;
//			cout << "movNum : " << movNum << endl;
//			movLoaded = movNum;
//			m_kinect.setLoopState(OF_LOOP_NORMAL);
//			m_kinect.play();
//		}
//	}
//}

//--------------------------------------------------------------
void ofApp::update() {
	m_box2d.update();
	updateKinect(true);
	updateContours(true);
}

//--------------------------------------------------------------
void ofApp::draw() {
	ofPushMatrix();
	ofScale(m_size, m_size);
	ofSetRectMode(OF_RECTMODE_CORNER);
	ofSetColor(255);

	drawKinect(true);

	for (ofPolyline const& p : m_contour.getPolylines()) //THE PROBLEM...
	{
		ofSetLineWidth(5);
		ofSetColor(ofColor(255, 0, 0));
		ofNoFill();


		//			ofSetLineWidth(1);
		//			ofSetColor(0, 0, 0);
		//			ofFill();
		ofPolyline s = p.getSmoothed(m_smoothingSize, m_smoothingShape);
		//            if(bUseAnalyze){
		//                ofPolyline smoothedPoly = analyze(s);



		if (bDrawSmoothed) {
			analyze(s);
			m_resampleSmoothed.draw();
			if (m_resampleSmoothed.size()) {
				ofSetColor(ofColor::yellow);
				ofDrawCircle(m_resampleSmoothed.getVertices()[0], 5);
			}

		}
		else {
			p.draw();
			if (p.size()) {
				ofSetColor(ofColor::yellow);
				ofDrawCircle(p.getVertices()[0], 5);
			}
		}

		//                createContourEdge(smoothedPoly);
		//            }else{
		//                ofPolyline smoothedPoly = smoothPoly(s);
		//                createContourEdge(smoothedPoly);
		//            }



	}
	//m_contourEdge->resize(m_size);
	//m_contourEdge->draw(); //...IS IN HERE.

	ofSetColor(255);
	for (auto circle : m_circles)
	{
		circle->draw();
	}
	//panel.setSize(panel.getWidth() / size, panel.getHeight() / size);
	//panel.setPosition(panel.getPosition().x / size, panel.getPosition().y / size);
	ofPopMatrix();
	m_panel.draw();

	ofDrawBitmapStringHighlight("FPS: " + ofToString(ofGetFrameRate()), 20, 20);
}

void ofApp::exit() {
	//	m_kinect.setCameraTiltAngle(15.0);
	m_kinect.close();
}

float ofApp::clip(float n, float lower, float upper) {
	return std::max(lower, std::min(n, upper));
}


void ofApp::drawKinect(bool isDepthCam)
{
	if (isDepthCam)
	{
		m_kinect.drawDepth(0, 0);
		return;
	}

	m_kinect.draw(0, 0);
}

void ofApp::updateKinect(bool isDepthCam)
{
	if (isDepthCam)
	{
		m_kinect.setDepthClipping(m_nearClip, m_farClip);
	}
	m_kinect.update();
}

void ofApp::updateContours(bool isDepthCam)
{
	if (!m_kinect.isFrameNew())
	{
		return;
	}
	m_contour.setMinAreaRadius(m_minArea);
	m_contour.setMaxAreaRadius(m_maxArea);
	if (isDepthCam)
	{
		m_grayImage.setFromPixels(m_kinect.getDepthPixels());
		m_contour.findContours(m_grayImage);
		return;
	}
	//m_contour.setTargetColor(m_trackingColor);
	m_contour.setThreshold(m_threshold);
	m_contour.findContours(m_kinect);
}

ofPolyline ofApp::smoothPoly(const ofPolyline& _polyline)
{
	auto polyline = _polyline.getResampledByCount(m_polylinePointCount /* + 1*/);
	//cout << polyline.size() << endl;
	glm::vec3 first_vert = { 0,0,0 };
	if (m_polys.size() > 0 && m_polys[0].size() > 0)
	{
		first_vert = m_polys[0].getVertices()[0];
	}
	size_t first_index = 0;
	float min_dist = std::numeric_limits<float>::max();
	for (size_t i = 0; i < polyline.size(); ++i)
	{
		auto d = glm::distance2(polyline[i], first_vert);
		if (d < min_dist)
		{
			min_dist = d;
			first_index = i;
		}
	}

	if (first_index > 0)
	{
		vector<glm::vec3> rearrangedVerts;
		rearrangedVerts.insert(rearrangedVerts.begin(), polyline.begin() + first_index, polyline.end());
		rearrangedVerts.insert(rearrangedVerts.end(), polyline.begin(), polyline.begin() + first_index);
		ofPolyline rearrangedPoly;
		rearrangedPoly.addVertices(rearrangedVerts);
		m_polys.push_front(rearrangedPoly);
	}
	else
	{
		m_polys.push_front(polyline);
	}
	m_polys.pop_back();

	ofPolyline averaged;
	//cout << m_polys.size() << endl;
	for (size_t i = 0; i < m_polylinePointCount; ++i)
	{
		glm::vec3 avg;
		for (size_t j = 0; j < m_polys.size(); ++j)
		{
			//cout << m_polys[j].size() << endl;
			//cout << m_polys[j][i] << endl;
			if (i < m_polys[j].size())
			{
				avg += m_polys[j][i];
			}
			//cout << "vertex[" << j << "][" << i << "] added" << endl;
		}
		avg /= m_polys.size();
		averaged.addVertex(avg);
	}
	return averaged;
}


void ofApp::analyze(const ofPolyline& curFrame) {


	//	ofPolyline line = curFrame;
	ofPolyline resampled = curFrame.getResampledByCount(m_polylinePointCount);

	// A++ if you fix getResampledByCount end points !
	//

//
//
	while (resampled.size() < m_polylinePointCount) {
		resampled.getVertices().push_back(resampled[resampled.size() - 1]);
	}
	//
	if (m_prevFrame.size() > 0) {
		//        cout << ",,,,,\n";
				// if you want to make this faster
				// you will get an A +
				// this is pretty slow :(
				// but it works.

		int smallestStart = -1;
		float smallestAvgLen = 10000000;

		float fCount = m_polylinePointCount.get();
		auto p_count = m_polylinePointCount.get();
		for (int i = 0; i < p_count; i++) {

			float avgLen = 0;
			for (int j = 0; j < p_count; j++) {
				avgLen += glm::distance2(resampled[(j + i) % p_count], m_prevFrame[j]) / fCount;
				//
			}
			if (avgLen < smallestAvgLen) {
				smallestAvgLen = avgLen;
				smallestStart = i;
			}

		}

		ofPolyline temp;
		for (int i = 0; i < p_count; i++) {
			temp.addVertex(resampled[(i + smallestStart) % p_count]);
		}
		resampled = temp;
	}
	m_prevFrame = resampled;
	//	ofPolyline tempT = resampled.getResampledByCount(100);
	//
	//	while (tempT.size() < 100) {
	//		tempT.getVertices().push_back(tempT[tempT.size() - 1]);
	//	}

	ofPolyline& tempT = resampled;
	if (m_resampleSmoothed.size() == 0 || ofIsFloatEqual(smoothingFactor.get(), 1.0f)) {
		m_resampleSmoothed = tempT;
	}
	else {
		for (int i = 0; i < tempT.size(); i++) {
			m_resampleSmoothed[i] = (1.0 - smoothingFactor) * m_resampleSmoothed[i] + (smoothingFactor.get() * tempT[i]);
		}
	}

	//if you want this function to return something it should be m_resampleSmoothed rather than resampled

	//	return resampled;

}

//--------------------------------------------------------------
void ofApp::keyPressed(int key) {
	switch (key)
	{
	case 'c':
		m_circles.clear();
		break;
	case 'd':
		break;
	}
}

//--------------------------------------------------------------
void ofApp::keyReleased(int key) {
	//if (key == OF_KEY_LEFT) {
	//	loadMov(((movLoaded + 1) % 7));
	//}
	//else if (key == OF_KEY_RIGHT) {
	//	loadMov(((movLoaded - 1 + 7) % 7));
	//}
}

//--------------------------------------------------------------
void ofApp::mouseMoved(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::mouseDragged(int x, int y, int button) {

}

//--------------------------------------------------------------
void ofApp::mousePressed(int x, int y, int button) {
	/*auto circle = make_shared <ofxBox2dCircle>();
	circle->setPhysics(3.0, 0.5, 0.1);
	circle->setup(m_box2d.getWorld(), x / m_size, y / m_size, 32);
	m_circles.push_back(circle);*/
}

//--------------------------------------------------------------
void ofApp::mouseReleased(int x, int y, int button) {

}

//--------------------------------------------------------------
void ofApp::mouseEntered(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::mouseExited(int x, int y) {

}

//--------------------------------------------------------------
void ofApp::windowResized(int w, int h) {

}

//--------------------------------------------------------------
void ofApp::gotMessage(ofMessage msg) {

}

//--------------------------------------------------------------
void ofApp::dragEvent(ofDragInfo dragInfo) {

}

Thanks. It sucks, and I am only able to manage it by taking an absurd amount of pills.

It is very strange how the smoothing behaves. It shouldn’t be doing that. When I tested it it did not do that but it actually smoothed the shape, but it introduces a lag.

What can help with the hair issue is to downscale the grayimage by some factor before passing it to findContours. Then you upscale the results of the contour finder by the same factor so it matches again the original image.
This is a very common practice in computer vision, which helps to reduce noise, and also improves performance. It is funny, that in a lot of cases lower resolution images are better than higher res for CV stuff.

Shrinking and re-upping the scale of the image helps both to get a smoother shape and to fix the weird shrinking/growing glitch- but only if I get a little farther from the camera.

I wanted to keep the high-res image to draw and use the low-res image for the contour edge. Right now I’m using two contourFinders, so that I can do this:

m_shader.begin();
	setShader();
	for (const ofPolyline& p : m_drawContour.getPolylines()) //THE PROBLEM...
	{
		ofBeginShape();
		for (int i = 0; i < p.getVertices().size(); ++i)
		{
			ofVertex(p.getVertices().at(i).x, p.getVertices().at(i).y);
		}
		ofEndShape();
	}
	m_shader.end();

with the high-res one, but this seems to hurt performance. Is there a better way to shade the high-res image? Using the shader with the ofImage just colors the whole screen with the shader.

Also, what’s a good way of using the same code with different types of ofxBox2d shapes, eg. is there a way to have a vector that holds both circles and rectangles? I guess if I want to use the same code on objects of both kinds I can use a template with a variable type.

Hi, sorry for the belated reply.

Yes, that is the slowest way to render stuff into open gl. can you just draw the polylines ? I think that is far more performant.

for (auto& p : m_drawContour.getPolylines()) 
	{
p.draw();
}

if this does not work for you, putting the vertices into an ofMesh is better.

ofMesh mesh;
mesh.setMode(OF_PRIMITIVE_LINE_STRIP);// or which ever other mesh mode.
	for (const ofPolyline& p : m_drawContour.getPolylines()) //THE PROBLEM...
	{
               mesh.addVertices(p.getVertices());
	}

m_shader.begin();
	setShader();
mesh.draw();
	m_shader.end();

all shapes seem to inherit from ofxBox2dBaseShape so you can store all in a vector<ofxBox2dBaseShape*> or better vector<unique_ptr<ofxBox2dBaseShape>>.

vector<unique_ptr<ofxBox2dBaseShape>> shapes;

shapes.emplace_back(make_unique<ofxBox2dCircle>());
shapes.emplace_back(make_unique<ofxBox2dRect>());

In order to access these and the functions that belong to each different type you will need to cast based on the type.


    for(auto& s: shapes){
        auto t = typeid(*(s.get())).name();
        
        std::cout << t << "\n";
                
        if (typeid(ofxBox2dCircle).name() == t){
        ofxBox2dCircle* circle =  static_cast<ofxBox2dCircle*>(s.get());

        }else  if (typeid(ofxBox2dRect).name() == t){
                        ofxBox2dRect* rect =  static_cast<ofxBox2dRect*>(s.get());
        }