   How to track 3d object in a video and change its dimension accordingly
  Hi friends,I want to make a tool to place a 3D object anywhere in a video scene. The inserted object’s view in the video should change accordingly as the video plays (perspective viewing, appear smaller if it is far and bigger as we go closer and disappear as we go past i.e. it should appear realistic). The video will be of a corridor or a similar place and will be in portrait format.I have written a program in which I am able to split video into frames and add 3d object to each frame (thanks to Nehe tutorials).Now I want to extend this program so that the size and view of the inserted 3d object can be change accordingly with the movement of video like perspective viewing, appear smaller if it is far and bigger as we go closer and disappear as we go past.Can any member give me any idea on how to add this feature and what framework I should use to implement this. Can this feature be implemented by using OpenGL or there is any other framework which should be used. I also came to know about OpenCV. Is it helpful?I have to develop this tool to submit it to my university. Hence kindly reply to this mail.Thanks in advance.Samir   
  You might be interested in this link:Camera tracking (Match moving)http://en.wikipedia.org/wiki/Camera_trackingSee the links at the bottom of the page for tracking software.Here's another:Voodoo Camera Trackerhttp://www.digilab.uni-hannover.de/docs/manual.html;  Hello,Thanks for your reply. I have read these documents but I could not find more information on what algorithm I should use to implement match moving feature beside this I am still confused on what framework provides this feature.Any help is highly appreciated.Thanks in advance.Samir   ;  Samir,I think you misunderstood about how close you are from your goal.  You are still at the starting point only. If you keep avoiding the implementation for pixel by pixel analysis, you probably cannot finish this task.   ;Quote:Original post by Samir MehtaThanks for your reply. I have read these documents but I could not find more information on what algorithm I should use to implement match moving feature beside this I am still confused on what framework provides this feature.What you are trying to implement is a very difficult task. There is no generic algorithm to achieve this. You will use a combination of many complex algorithms ranging for computer vision, feature detection, perspective matching, image analysis, etc. Some of the algorithms involved are good material for a PhD thesis, and nothing that you can just implement in a couple of weeks.OpenCV is a powerful realtime computer vision library, that contains features that can be used as building blocks to a camera tracking system. However, even with help from CV, implementing a robust camera tracker is daunting and requires a lot of experience.Your best bet is to use an existing full solution, such as the Voodoo tracker posted above.;  Thanks for your reply.My problem is that I need to develop this project as part of my engineering degree and since Voodoo is not an open source software hence I can't get much help from this software despite of being good software.Is there any open source software which has the feature of camera tracking/ match moving?Once again thanks a lot for your valuable replies.Samir   ;  Since your video is of a fixed scene or similar scenes you should be able to identify the perspective vanishing point and using various algorithms extrapolate the field of view from the that. Once you have that, recreating a 3d scene with the same persepctive setup and camera position is straightforward enough. If the video moves then you'll need to add in tracking algorithms to adjust the camera appropriately. What problem are you trying to solve? If its an algorithm for deconstructing a video into a 3d scene then there are plenty of papers on how to do that. If its a thesis about virtual reality and projecting objects into video stream, then you can cheat and embed trackers in the scene to help you deconstruct it. If it's just learning how to render 3D objects in video, then simplify it as much as possible, bake in scene, fixed camera, bake in lighting etc..Good Luck!-ddn;  In the application I need to adjust the size and perspective view of inserted 3d object with the movement of video.There is need to allow the insertion of 3d object into the video with the correct position, scale, orientation and motion in relation to the scene of the video.The intent is that when the virtual and real scenes are composited together to produce a new video they should look like as they come from the same perspective and appear seamless.ThanksSamir;  So you will need some tracking technology but it doesn't sound like your trying to solve the general problem of projecting 3d objects in any scene, just the given test scene (corridor scene)Since the scene is a corridor, you know that the edges (intersection formed by the ceiling and floor ) will converge to the horizon at the vanishing point. You'll need to identify those, either through AI vision algorithms for edge tracking or if you want, just embedded colored markers on the edges evenly spaced receding away from the camera. (I suggest u just embed markers ). You can easily extract the markers from the scene, this will give u the various pieces of information you need (relative location and height of camera, field of view, floor plane, etc )Once you have that you can recreate an analogous scene in openGL with the same camera position, floor plane and if you've got it, lighting setup. After that you can render objects in this 3D space like normal, and it should matchup with the realworld video. To track the camera, run the same algorithm to deconstruct the scene and see if the camera position has moved ( if so adjust the camera in the openGL scene appropriately ).If you have access to them, there are 3D cameras ( which not only can see color but also depth ) which you can use. They will give depth information of the scene which u can then add onto the openGL scene and create true occlusion with the 3D obejcts, which would be cool :DGood Luck!-ddn   