Camera calibration/pin hole camera model and working out 3d position I have a calibrated camera and have the intrinsic parameters. I also have the extrinsic parameters relative to a point (the world origin) on a planar surface in the real world. This point I have set as the origin in the real world coordinates [0,0,0] with a normal of [0,0,1].

From these extrinsic parameters I can work out the camera position and rotation in the world plane 3d coordinates using this here: http://en.wikipedia.org/wiki/Camera_resectioning

Now I have a second point which I have extracted the image coordinates for [x, y]. How do I now get the 3d position of this point in the world coordinate system?

I think the intuition here is that I have to trace a ray that goes from the optical center of the camera (which I now have the 3D position for as described above), through the image plane [x,y] of the camera and then through my real world plane which I defined at the top.

Now I can intersect a world coordinate 3d ray with a plane as I know normal and point on that plane. What I don't get is how I find out the 3d position and direction when it leaves the image plane through a pixel. It's the transformation through different coordinate systems that is confusing me.
If you have the extrinsics then it is very easy. Having extrinsics is the same as having "camera pose" and the same as having the homography. Check this post in stackoverflow.

You have extrinsics, also called camera pose, which is described as a translation and a rotation:

$\displaystyle Pose =\begin{bmatrix}R|t \end{bmatrix} = \begin{bmatrix}R_{11} &amp;R_{12}&amp;R_{13}&amp;t_x\\R_{21}&amp;R_{22}&amp;R_{23}&amp;t_y\\R_{31}&amp;R_{32}&amp;R_{33}&amp;t_z \end{bmatrix} $

You can get Homography from Pose this way:

$\displaystyle   H = \frac{1}{t_z}\begin{bmatrix}{R_{1x}}&amp;{R_{2x}}&amp;{t_x}\\{R_{1y}}&amp;{R_{2y}}&amp;{t_y}\\{R_{1z}}&amp;{R_{2z}}&amp;{t_z}\end{bmatrix}$

Then you can project your 2D points into the corresponding 3D points by multiplying the Homography by the points:

$p_{2D}=\begin{bmatrix}x &amp;y &amp;1\end{bmatrix}\quad$ add $\quad z=1\quad$ to make them homogeneous

$p_{3D}=H*p_{2D}  $ 

$p= p / p(z)\quad$   Normalize the points