Transformation/Projection Problem

Im trying to create a basic masking application. Its working when avoid rotations on X or Y axes and have all rotations on Z (only 2d).

Imagine i have two hierarchical objects and i need to have image masks for each. In my design each object needs to have their mask pre calculated all the time and they will be only updated if my transform changes. For this reasons, lets say i have an object in the hierarchy and to calculate it mask i need to account (intersect) parents mask.

The way im doing at the moment is like this :

Im enabling the mask fbo, multiplying with the inverse of current transformation and drawing parents mask into this. Its should be working but at the moment i got it only for 2d rotations and i wasn’t able to identify the problem. I would love to hear if someone can tell me what is going on and how can i fix it.

2d

3d

Also attached source to play with it.
src.zip (1.8 KB)

I haven’t got any solution yet but it seems problem is related to the projection or viewport settings. Any suggestion so far?