title: 2D Image to 3D world Coordinates
body: <p>We crawled a set of images from the Google Street View (GSV) API. I want to estimate the 3D World coordinate from 2D Image given the following:</p>
  
  <p><strong>1. The GPS location (i.e., latitude and longitude) of the camera capturing the image</strong></p>
  
  <p>Conversion of GPS coordinates to translation matrix: Used 2 types of conversion
  methods to get the translation matrix -> UTM conversion and conversion to Cartesian coordinates.</p>
  
  <ul>
  <li>UTM conversion: Used Python's UTM library to convert GPS coordinates
  to UTM coordinates. Used the north and east values with a fixed height to create the translation matrix.</li>
  <li>Cartesian conversion - Used the following formula to generate translation matrix:</li>
  </ul>
  
  <blockquote>
    <p>x = Radius*math.cos(latitude)*math.cos(longitude)</p>
    
    <p>y = Radius*math.cos(latitude)*math.sin(longitude)</p>
    
    <p>z = Radius*math.sin(latitude)</p>
  </blockquote>
  
  <p><strong>2. The rotation matrix calculated using openSFM (i.e., the SFM
      algorithm).</strong></p>
  
  <p>The library provides <em>alpha</em>, <em>beta</em>, <em>gamma</em> angles (in Radian) which map to <em>yaw</em>, <em>pitch</em>, and <em>roll</em> angles, respectively. The rotation matrix is constructed using the formula (<a href="http://planning.cs.uiuc.edu/node102.html" rel="nofollow noreferrer">http://planning.cs.uiuc.edu/node102.html</a>)</p>
  
  <blockquote>
    <p>Rotation Matrix (R): R(alpha, beta, gamma)= R_z (alpha) * R_y (beta) *
    R_x (gamma)</p>
  </blockquote>
  
  <p><strong>3. Based on the angle of field of view and the dimensions of the image,
      we estimate the calibration matrix as the following (<a href="https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/" rel="nofollow noreferrer">https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/</a><a href="https://codeyarns.com/2015/09/08/how-to-compute-intrinsic-camera-matrix-for-a-camera/" rel="nofollow noreferrer">enter link description here</a>):</strong></p>
  
  <blockquote>
    <p>K =   [[f_x   s   X], [0  f_y Y], [0  0   1]]</p>
    
    <p>x and y are half of the image dimensions (i.e., x = width/2 and y = height/2)</p>
    
    <p>The GSV API provides field of view angle θ (e.g., 45 or 80) so the focal length can be calculated as</p>
    
    <p>f_x=  x/tan⁡(θ/2)</p>
    
    <p>f_y=  y/tan⁡(θ/2)</p>
  </blockquote>
  
  <p><strong>Using the matrices T, R, and K, how can we estimate the 3D World coordinates of each pixel in the 2D image?</strong></p>