Pepper's Ghost camera mapping calibration

Hey guys,

we are a team of Media Design students and we are trying to get a Pepper’s Ghost working.
Our Setup

We want to project some “augmented Reality” footage, like connections between points of interests and information into a real world 3D printed Room with walls. As we tried to calibrate the perspective we discovered a big problem (We are using Unrel Engine 4, which is using only horizontal Field of view and no physically based camera). The horizontal field of view ist distorting the whole Pepper’s Ghost as seen in the image above and therefore the mapping wont work at all. A extra note, we are ware that this illusion will only work from a certain point of view and therefore we also wanted to add head tracking for a parallaxe effect.
Our professor suggested that we should use Processing or openFrameworks to get the camera matrix and the viewing frustrum by mapping it on the real chess pattern and if it works we could apply those values to the camera in UE4.

How would you guys approach this problem? Is it even possible to map a Pepper’s Ghost projection into a 3D space at all?

thank you for your answers

(UE Forum Post for reference)

i am not sure who paper is and why she is a ghost :wink:

but should help you with the warping that your video needs.

maybe you can use
to help you send the video from your unreal engine 4 in to openframeworks