title: How to calculate ray in real-world coordinate system from image using projection matrix?
body: <p>Given n images and a projection matrix for each image, how can i calculate the ray (line) emitted by each pixel of the images, which is intersecting one of the three planes of the real-world coordinate system? The object captured by the camera is at the same position, just the camera's position is different for each image. That's why there is a separate projection matrix for each image.</p>
  
  <p>As far as my research suggests, this is the inverse of the 3D to 2D projection. Since information is lost when projecting to 2D, it's only possible to calculate the ray (line) in the real-world coordinate system, which is fine.</p>
  
  <p>An example projection matrix P, that a calculated based on given K, R and t component, according to K*[R t]</p>
  
  <pre><code>   3310.400000 0.000000 316.730000 
  K= 0.000000 3325.500000 200.550000 
     0.000000 0.000000 1.000000 
  
  
     -0.14396457836077139000 0.96965263281337499000 0.19760617153779569000 
  R= -0.90366580603479685000 -0.04743335255026152200 -0.42560419233334673000 
     -0.40331536459778505000 -0.23984130575212276000 0.88306936201487163000 
  
     -0.010415508744 
  t= -0.0294278883669 
     0.673097816109
  
     -604.322  3133.973   933.850   178.711
  P= -3086.026  -205.840 -1238.247    37.127
     -0.403    -0.240     0.883     0.673
  </code></pre>
  
  <p>I am using the "DinoSparseRing" data set available at <a href="http://vision.middlebury.edu/mview/data" rel="nofollow noreferrer">http://vision.middlebury.edu/mview/data</a></p>
  
  <pre><code>for (int i = 0; i &lt; 16; i++) {
          RealMatrix rotationMatrix = MatrixUtils.createRealMatrix(rotationMatrices[i]);
          RealVector translationVector = MatrixUtils.createRealVector(translationMatrices[i]);
          // construct projection matrix according to K*[R t]
          RealMatrix projMatrix = getP(kalibrationMatrices[i], rotationMatrices[i], translationMatrices[i]);
          // getM returns the first 3x3 block of the 3x4 projection matrix
          RealMatrix projMInverse = MatrixUtils.inverse(getM(projMatrix));
          // compute camera center
          RealVector c = rotationMatrix.transpose().scalarMultiply(-1.f).operate(translationVector);
  
          // compute all unprojected points and direction vector per project point
          for (int m = 0; m &lt; image_m_num_pixel; m++) {
              for (int n = 0; n &lt; image_n_num_pixel; n++) {
                  double[] projectedPoint = new double[]{
                          n,
                          m,
                          1};
                  // undo perspective divide
                  projectedPoint[0] *= projectedPoint[2];
                  projectedPoint[1] *= projectedPoint[2];
                  // undo projection by multiplying by inverse:
                  RealVector projectedPointVector = MatrixUtils.createRealVector(projectedPoint);
                  RealVector unprojectedPointVector = projMInverse.operate(projectedPointVector);
  
                  // compute direction vector
                  RealVector directionVector = unprojectedPointVector.subtract(c);
                  // normalize direction vector
                  double dist = Math.sqrt((directionVector.getEntry(0) * directionVector.getEntry(0))
                          + (directionVector.getEntry(1) * directionVector.getEntry(1))
                          + (directionVector.getEntry(2) * directionVector.getEntry(2)));
                  directionVector.setEntry(0, directionVector.getEntry(0) * (1.0 / dist));
                  directionVector.setEntry(1, directionVector.getEntry(1) * (1.0 / dist));
                  directionVector.setEntry(2, directionVector.getEntry(2) * (1.0 / dist));
              }
          }
      }
  </code></pre>
  
  <p>The following 2 plots show the outer rays for each images (total of 16 images). The blue end is the camera point and the cyan is a bounding box containing the object captured by the camera. One can clearly see the rays projecting back to the object in world coordinate system.</p>
  
  <p><img src="https://i.stack.imgur.com/KmKW8.png" alt="enter image description here">
  <img src="https://i.stack.imgur.com/qi2rQ.png" alt="enter image description here"></p>