title: Get 3D coordinated from 2D coordinates
body: <p>My object is to calculate the 3D (world) coordinates from the 2D (image) coordinates. For this, using a chessboard I've calculated the intrinsic (for camera) and extrinsic (for a particular image) matrix. Let's suppose that I've saved the camera matrix in <code>mtx</code> and distortion coefficients in <code>dist</code>. Also, the size of an individual square of the chessboard id <code>25</code> mm for a 9x6 board. I've used <code>solvePnP</code> to get the extrinsic matrix:</p>
  <pre><code>sq_size = 25
  rows = 9
  cols = 6
  
  objp = np.zeros((cols * rows,3), np.float32)
  objp[:,:2] = np.mgrid[0:rows,0:cols].T.reshape(-1,2)
  objp = objp*sq_size
  
  retval,myRvec,myTvec=cv2.solvePnP(objectPoints=objp, 
                                    imagePoints=corners.reshape(-1, 2), 
                                    cameraMatrix=mtx, 
                                    distCoeffs=dist)
  </code></pre>
  <p>Using the <code>myRvec</code> and <code>myTvec</code> I can convert the world coordinate to the image coordinate like this:</p>
  <pre><code>axis = np.float32([[1,0,0], [0,1,0], [0,0,-1]]).reshape(-1,3)*sq_size
  imgpts, jac = cv2.projectPoints(axis, myRvec, myTvec, mtx, dist)
  </code></pre>
  <p>Now I want to reverse the process; given the image coordinate I want to calculate the world coordinate, how to do that ?</p>
  <p><strong>Edit</strong>
  I've found Homography using following:</p>
  <pre><code>pts_dst = np.zeros((cols * rows,2), np.float32)
  pts_dst[:,:2] = np.mgrid[0:rows,0:cols].T.reshape(-1,2)
  pts_dst = pts_dst*sq_size
  
  pts_src, jac = cv2.projectPoints(objp, myRvec, myTvec, mtx, dist)
  pts_src = np.squeeze(pts_src)
  
  h, status = cv2.findHomography(pts_src, pts_dst)
  transformed = np.zeros((img.shape[0], img.shape[1]), dtype=np.uint8);
  dst = cv2.warpPerspective(img, h, transformed.shape)
  </code></pre>
  <p>And here is my output:
  <a href="https://i.stack.imgur.com/H2ACQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/H2ACQ.png" alt="enter image description here" /></a></p>
  <p>Why am I missing th right side of the chessboard and how to correct it ?</p>