View projection onto 3d planes [Help!]

Hello, i have to finish a Uni project with Interactive Suface with a displaywall and room interior through a 3d tracked projector device. Here is a example picture.

To archive this, my thoughts were recreating 3d surface of the room with planes, using ofcamera as the projector device and calculating the position its pointing at with 4 cornor points.

Im currently trying to use the Fbos as texture to archive drawing of this red view rectangle on a single surface.

The problem arises when i have to draw the correct rectangular project onto multiple surfaces. since its not only connecting the 4 dot and draw a retangular in a single surface. The view rectangular are divided in to different part on different sufaces.

I want to ask if you guys have any suggestion on how i can find out the solution to draw this projection correctly. Or maybe the approch with FBO as drawing texture isnt the correct way to do this.

Thanks for your time.
Any help would be greatly appreciated!

Regards,
Chen