Posing with ofxAssimpModelLoader

i have a collada file that contains a mesh and a rig for the mesh.

i’ve loaded it into OF with ofxAssimpModelLoader, and i want to define custom poses for the rig.

i can see inside ofxAssimpModelLoader::updateAnimation() that deforming the mesh involves something like recursively applying matrix transformations that have been defined in the animation.

so i suppose what i want to do is define those transforms myself instead of loading them from the animation?

has anyone done something similar to this, and could you give me pointers?

i feel like it would be nice to have a function in ofxAssimpModelLoader that just lets you specify the orientation for all the bones in the rig.

kyle, i was thinking about this some days ago :slight_smile: I need to check but i think the problem with assimp is that the returned objects are always const. I think it shouldn’t be a problem, though, to get a copy and modify it as long as you don’t want to update it later from values in the original model.

The code for that was done by Anton but as far as i understand the animation is defined in key frames so you need to interpolate to get the transformations in between and the bones have some weights that are the amount of that transformation that is applied to each vertex.

Haven’t looked into it much further but also interested in doing it.

When doing 007 there was a discussion about how to implement the model loader, Keith started to implement everything with OF objects. In the end we did everything with assimp mainly because of memory management but perhaps it’s the time to look into having ofBone. I think it’s the only part missing to be able to do this.

i did something like this a while ago. what i ended up doing is inherited my own class from ofxAssimpModelLoader with a custom Bone class and rewrote the part of updateAnimation that was dealing with the rigging. the Bone class is similar to ofNode, but it has to deal with the inheritance of orientation, because when you do manual rigging you probably don’t want the orientation to be affected by the parent.

@makuz, what you describe is exactly what i was about to write! would it be possible for you to post that code?

@arturo, i think if what makuz describes really works, then there shouldn’t be any issues about const-ness. after all, the animation works, and that’s definitely deforming the mesh.

here’s my first shot. it doesn’t do anything, and i’m sure i’m completely misunderstanding how things work.

i was thinking that bone->mOffsetMatrix would be the thing i could set to rig the model, but it looks like nothing happens when i do this. i’m initializing all the incoming aiMatrix4x4 with a random orientation so i should see something if this was vaguely right. i do know my mesh has 74 bones so that part is working.

  
  
class RiggedModel : public ofxAssimpModelLoader {  
public:  
	int getNumBones(int which = 0) {  
		const aiMesh* mesh = modelMeshes[which].mesh;	  
		return mesh->mNumBones;  
	}  
	void pose(vector<aiMatrix4x4>& pose, int which = 0) {  
		const aiMesh* mesh = modelMeshes[which].mesh;  
		vector<aiMatrix4x4> boneMatrices(mesh->mNumBones);  
		for(int a = 0; a < mesh->mNumBones; a++) {  
			const aiBone* bone = mesh->mBones[a];  
			aiNode* node = scene->mRootNode->FindNode(bone->mName);  
			boneMatrices[a] = pose[a];//bone->mOffsetMatrix;  
			const aiNode* tempNode = node;  
			while(tempNode) {  
				boneMatrices[a] = tempNode->mTransformation * boneMatrices[a];  
				tempNode = tempNode->mParent;  
			}  
			modelMeshes[which].hasChanged = true;  
			modelMeshes[which].validCache = false;  
		}  
		  
		modelMeshes[which].animatedPos.assign(modelMeshes[which].animatedPos.size(),0);  
		if(mesh->HasNormals()){  
			modelMeshes[which].animatedNorm.assign(modelMeshes[which].animatedNorm.size(),0);  
		}  
		  
		// loop through all vertex weights of all bones  
		for(int a = 0; a < mesh->mNumBones; a++) {  
			const aiBone* bone = mesh->mBones[a];  
			const aiMatrix4x4& posTrafo = boneMatrices[a];  
			for( size_t b = 0; b < bone->mNumWeights; b++) {  
				const aiVertexWeight& weight = bone->mWeights[b];  
				size_t vertexId = weight.mVertexId;  
				const aiVector3D& srcPos = mesh->mVertices[vertexId];  
				modelMeshes[which].animatedPos[vertexId] += weight.mWeight * (posTrafo * srcPos);  
			}  
			if(mesh->HasNormals()) {  
				// 3x3 matrix, contains the bone matrix without the translation, only with rotation and possibly scaling  
				aiMatrix3x3 normTrafo = aiMatrix3x3( posTrafo);  
				for( size_t b = 0; b < bone->mNumWeights; b++) {  
					const aiVertexWeight& weight = bone->mWeights[b];  
					size_t vertexId = weight.mVertexId;  
					const aiVector3D& srcNorm = mesh->mNormals[vertexId];  
					modelMeshes[which].animatedNorm[vertexId] += weight.mWeight * (normTrafo * srcNorm);  
				}  
			}  
		}  
	}  
};  
  

so i forgot to add updateGLResources(); to the end of that. then i got something. except was all stretched out like an alien.

so i changed the line where i set the pose to multiply by the mOffsetMatrix first:

  
  
boneMatrices[a] = pose[a] * bone->mOffsetMatrix;  
  

and now i get something that is scaled correctly, but not accumulating.

ok, here’s a technique that works for me.

it’s admittedly the wrong approach. but i don’t really understand how the animation is setting the bones without breaking constness.

http://pastebin.com/Lmra7zvA

any insight on how the animation works normally would be very much appreciated.

usually it doesn’t break constness because it’s just doing a copy of the original mesh transformed with the bones matrices, the used meshes are in the mesh helpers, in modelMeshes. That is the data that gets uploaded to the vbos not the originals

and yes, that looks pretty correct to me. What i meant with the constness is that you cannot add bones to the model but you can apply transformations like you are doing without problem

not completely sure but from what i understand each key frame defines a translation, rotation and scaling (actually several in a node hierarchy)

When you pass the time it interpolates between 2 key frames to calculate the transformations at that time and then with those it creates a transformation matrix.

The bones weights tells you how much of that transformation has to be applied to each vertex in the mesh

I guess you could pass nodes instead of matrices for the transformations then the rest of the code you have seems pretty much correct

http://vimeo.com/31237031

kyle, i’m afraid i am not allowed to post the code, because it’s client work, but i see you managed well without it. if you have any more questions or need any help, just let me know.

wow, is it just random poses?

yes, a +/- 45 degree angle on a random axis:

  
  
ofMatrix4x4 randomRotation(float amt = 180) {  
	ofQuaternion quat;  
	ofMatrix4x4 mat;  
	ofVec3f axis(ofRandomf(), ofRandomf(), ofRandomf());  
	axis.normalize();  
	quat.makeRotate(ofRandom(-amt, amt), axis);  
	quat.get(mat);  
	return mat;  
}  
  
void testApp::randomPose() {  
	Pose pose = model.getPose();  
	for(Pose::iterator i = pose.begin(); i != pose.end(); i++) {  
		aiMatrix4x4& cur = i->second;  
		ofMatrix4x4 mat = randomRotation(45);  
		cur.a1 = mat(0, 0);  
		cur.a2 = mat(0, 1);  
		cur.a3 = mat(0, 2);  
		cur.b1 = mat(1, 0);  
		cur.b2 = mat(1, 1);  
		cur.b3 = mat(1, 2);  
		cur.c1 = mat(2, 0);  
		cur.c2 = mat(2, 1);  
		cur.c3 = mat(2, 2);  
	}  
	model.pose(pose);  
	curRotation = randomRotation();  
}  
  

this really destroys some things, like the jaw. but for most things it works fine.

i’m definitely interested in glitch vs the body… i want to explore this style a little more.

but i really got started with this because i’m thinking about implementing the algorithm behind the skeleton tracker on xbox/ms sdk.

i need to take into account the orientation, but i think this can do to find the skeleton in a model:

https://github.com/arturoc/ofxAssimpSkeleton

I’m finding bounding boxes for all the vertexes that have weights for each bone

![](http://forum.openframeworks.cc/uploads/default/1906/Pantallazo del 2011-10-28 18:07:21.png)

this kind of works: i’m finding the oriented bounding boxes and then finding the centroid of two opposite faces of the bb to find the skeleton. The problem is the orientation is not always right, i guess i need to find the farthest faces in each bb

![](http://forum.openframeworks.cc/uploads/default/1908/Pantallazo del 2011-10-28 19:28:10.png)

![](http://forum.openframeworks.cc/uploads/default/1910/Pantallazo del 2011-10-28 19:59:26.png)

![](http://forum.openframeworks.cc/uploads/default/1912/Pantallazo del 2011-10-28 20:15:24.png)

Hey,
some weeks ago I was dealing with this same thing but I had to work on another stuff so I left it aside.
I managed to modify bone positions and rotations at will. In the attached project, moving the mouse rotates a bone, and using the arrow keys you can select the different bones.
There are some strange things happening though, Ill try to figure the out this weekend.
If anyone does any improvement please let me know.

[AssimpBones 2.zip](http://forum.openframeworks.cc/uploads/default/1914/AssimpBones 2.zip)

wow, those bounding boxes are really interesting! the MS paper doesn’t need bounding boxes, but it might come in useful at some point anyway.

@roy i tried your code but everything just looks crazy warped and distorted, and the bone changes don’t accumulate recursively. i’ll try adding controls to my version soon to make sure it’s really working like i think it is.

i think now that i can at least modify the mesh i can work backwards to get posing properly built in to ofxAssimpLoader.

i have a better understanding now of what’s going on. if you’re interested in understanding how updateAnimation() works, here’s a summary:

a scene has a number of animations, each stored in an aiAnimation.

an animation has a number of channels, each stored in an aiNodeAnim.

a channel has a target node stored in channel->mNodeName that you need to find in the scene:

aiNode* targetNode = scene->mRootNode->FindNode(channel->mNodeName);

a channel also has keys with timestamps.

  • channel->mPositionKeys
  • channel->mRotationKeys
  • channel->mScalingKeys

updateAnimation() finds the two keys closest to the current time and interpolates between them. the results are stored in:

  • aiVector3D presentPosition
  • aiQuaternion presentRotation
  • aiVector3D presentScaling

all these things are loaded into an aiMatrix4x4 and assigned to targetNode->mTransformation

now if you just set all the targetNode->mTransformation to the identity matrix, you will get a severely messed up looking model. this is because the mTransformation starts out with a value. all of these values together are called the “bind pose”.

with this understanding, i’ve written a new version of the RiggedModel class that simplifies things and makes it run faster: http://pastebin.com/ApmbULEi

updatePose() handles the second half of updateAnimation(). it takes the current pose and updates all the vertices.

getPose() is something you can call on the first frame to get the bind pose, which you need to multiply any of your transforms by.

setPose() will apply those transforms and call updatePose() for you. it will also call updateGLResources(). i’m not sure why ofxAssimpModelLoader calls updateGLResources() in the draw() for animations…

updateAnimation() is basically unchanged, it’s just the first half. the second half has turned into a loop that calls updatePose().

could we find a way to incorporate this into the core?

i think having to multiply against the bind pose might be a little confusing. it’s also weird that Pose is just a map<>. maybe we can make a Pose class that handles these things?

about calling updateGLResources in draw, i think that’s correct since it’s uploading things to the graphics card. as i understand it the whole update/draw model is for updating calculations (cpu) in update and drawing in draw (gpu) although in OF there’s some gpu things that happen in update like updating video textures.

about bone changes accumulating recursively do you mean IK animation? i was also thinking that in make human (or any other 3d package) there’s actually a skeleton which seems to have IK constrains, i guess that’s what we are missing here