posing with ofxAssimpModelLoader

[kylemcdonald]

i have a collada file that contains a mesh and a rig for the mesh.

i've loaded it into OF with ofxAssimpModelLoader, and i want to define custom poses for the rig.

i can see inside ofxAssimpModelLoader::updateAnimation() that deforming the mesh involves something like recursively applying matrix transformations that have been defined in the animation.

so i suppose what i want to do is define those transforms myself instead of loading them from the animation?

has anyone done something similar to this, and could you give me pointers?

i feel like it would be nice to have a function in ofxAssimpModelLoader that just lets you specify the orientation for all the bones in the rig.

[arturo]

kyle, i was thinking about this some days ago :)  I need to check but i think the problem with assimp is that the returned objects are always const. I think it shouldn't be a problem, though, to get a copy and modify it as long as you don't want to update it later from values in the original model.

The code for that was done by Anton but as far as i understand the animation is defined in key frames so you need to interpolate to get the transformations in between and the bones have some weights that are the amount of that transformation that is applied to each vertex.

Haven't looked into it much further but also interested in doing it.

When doing 007 there was a discussion about how to implement the model loader, Keith started to implement everything with OF objects. In the end we did everything with assimp mainly because of memory management but perhaps it's the time to look into having ofBone. I think it's the only part missing to be able to do this.

[makuz]

i did something like this a while ago. what i ended up doing is inherited my own class from ofxAssimpModelLoader with a custom Bone class and rewrote the part of updateAnimation that was dealing with the rigging. the Bone class is similar to ofNode, but it has to deal with the inheritance of orientation, because when you do manual rigging you probably don't want the orientation to be affected by the parent.

[kylemcdonald]

@makuz, what you describe is exactly what i was about to write! would it be possible for you to post that code?

@arturo, i think if what makuz describes really works, then there shouldn't be any issues about const-ness. after all, the animation works, and that's definitely deforming the mesh.

[kylemcdonald]

here's my first shot. it doesn't do anything, and i'm sure i'm completely misunderstanding how things work.

i was thinking that bone->mOffsetMatrix would be the thing i could set to rig the model, but it looks like nothing happens when i do this. i'm initializing all the incoming aiMatrix4x4 with a random orientation so i should see something if this was vaguely right. i do know my mesh has 74 bones so that part is working.

Code:

class RiggedModel : public ofxAssimpModelLoader {
public:
	int getNumBones(int which = 0) {
		const aiMesh* mesh = modelMeshes[which].mesh;	
		return mesh->mNumBones;
	}
	void pose(vector<aiMatrix4x4>& pose, int which = 0) {
		const aiMesh* mesh = modelMeshes[which].mesh;
		vector<aiMatrix4x4> boneMatrices(mesh->mNumBones);
		for(int a = 0; a < mesh->mNumBones; a++) {
			const aiBone* bone = mesh->mBones[a];
			aiNode* node = scene->mRootNode->FindNode(bone->mName);
			boneMatrices[a] = pose[a];//bone->mOffsetMatrix;
			const aiNode* tempNode = node;
			while(tempNode) {
				boneMatrices[a] = tempNode->mTransformation * boneMatrices[a];
				tempNode = tempNode->mParent;
			}
			modelMeshes[which].hasChanged = true;
			modelMeshes[which].validCache = false;
		}
		
		modelMeshes[which].animatedPos.assign(modelMeshes[which].animatedPos.size(),0);
		if(mesh->HasNormals()){
			modelMeshes[which].animatedNorm.assign(modelMeshes[which].animatedNorm.size(),0);
		}
		
		// loop through all vertex weights of all bones
		for(int a = 0; a < mesh->mNumBones; a++) {
			const aiBone* bone = mesh->mBones[a];
			const aiMatrix4x4& posTrafo = boneMatrices[a];
			for( size_t b = 0; b < bone->mNumWeights; b++) {
				const aiVertexWeight& weight = bone->mWeights[b];
				size_t vertexId = weight.mVertexId;
				const aiVector3D& srcPos = mesh->mVertices[vertexId];
				modelMeshes[which].animatedPos[vertexId] += weight.mWeight * (posTrafo * srcPos);
			}
			if(mesh->HasNormals()) {
				// 3x3 matrix, contains the bone matrix without the translation, only with rotation and possibly scaling
				aiMatrix3x3 normTrafo = aiMatrix3x3( posTrafo);
				for( size_t b = 0; b < bone->mNumWeights; b++) {
					const aiVertexWeight& weight = bone->mWeights[b];
					size_t vertexId = weight.mVertexId;
					const aiVector3D& srcNorm = mesh->mNormals[vertexId];
					modelMeshes[which].animatedNorm[vertexId] += weight.mWeight * (normTrafo * srcNorm);
				}
			}
		}
	}
};

[kylemcdonald]

so i forgot to add updateGLResources(); to the end of that. then i got something. except was all stretched out like an alien.

so i changed the line where i set the pose to multiply by the mOffsetMatrix first:

Code:

boneMatrices[a] = pose[a] * bone->mOffsetMatrix;

and now i get something that is scaled correctly, but not accumulating.

[kylemcdonald]

ok, here's a technique that works for me.

it's admittedly the wrong approach. but i don't really understand how the animation is setting the bones without breaking constness.

http://pastebin.com/Lmra7zvA

any insight on how the animation works normally would be very much appreciated.

[arturo]

usually it doesn't break constness because it's just doing a copy of the original mesh transformed with the bones matrices, the used meshes are in the mesh helpers, in modelMeshes. That is the data that gets uploaded to the vbos not the originals

[arturo]

and yes, that looks pretty correct to me. What i meant with the constness is that you cannot add bones to the model but you can apply transformations like you are doing without problem

[arturo]

not completely sure but from what i understand each key frame defines a translation, rotation and scaling (actually several in a node hierarchy)

When you pass the time it interpolates between 2 key frames to calculate the transformations at that time and then with those it creates a transformation matrix.

The bones weights tells you how much of that transformation has to be applied to each vertex in the mesh

I guess you could pass nodes instead of matrices for the transformations then the rest of the code you have seems pretty much correct

[kylemcdonald]

[makuz]

kyle, i'm afraid i am not allowed to post the code, because it's client work, but i see you managed well without it. if you have any more questions or need any help, just let me know.

[arturo]

wow, is it just random poses?

[kylemcdonald]

yes, a +/- 45 degree angle on a random axis:

Code:

ofMatrix4x4 randomRotation(float amt = 180) {
	ofQuaternion quat;
	ofMatrix4x4 mat;
	ofVec3f axis(ofRandomf(), ofRandomf(), ofRandomf());
	axis.normalize();
	quat.makeRotate(ofRandom(-amt, amt), axis);
	quat.get(mat);
	return mat;
}

void testApp::randomPose() {
	Pose pose = model.getPose();
	for(Pose::iterator i = pose.begin(); i != pose.end(); i++) {
		aiMatrix4x4& cur = i->second;
		ofMatrix4x4 mat = randomRotation(45);
		cur.a1 = mat(0, 0);
		cur.a2 = mat(0, 1);
		cur.a3 = mat(0, 2);
		cur.b1 = mat(1, 0);
		cur.b2 = mat(1, 1);
		cur.b3 = mat(1, 2);
		cur.c1 = mat(2, 0);
		cur.c2 = mat(2, 1);
		cur.c3 = mat(2, 2);
	}
	model.pose(pose);
	curRotation = randomRotation();
}

this really destroys some things, like the jaw. but for most things it works fine.

i'm definitely interested in glitch vs the body... i want to explore this style a little more.

but i really got started with this because i'm thinking about implementing the algorithm behind the skeleton tracker on xbox/ms sdk.

[arturo]

i need to take into account the orientation, but i think this can do to find the skeleton in a model:

https://github.com/arturoc/ofxAssimpSkeleton


I'm finding bounding boxes for all the vertexes that have weights for each bone