Object Interactions with box2d and ofxcv

My target is to build a system where I can interact with my 3D objects away from screen virtually. I am a newbie for OF and Opencv. But I learnt OF as per my requirement. Now \I need The best way to implement my Target.

Any suggestions.

Thanks in advance!!