title: How do you map Kinect's depth data to its RGB color?
body: <p>I'm working with a given dataset using OpenCV, without any Kinect by my side. And I would like to map the given depth data to its RGB counterpart (so that I can get the actual color and the depth)</p>
  
  <p>Since I'm using OpenCV and C++, and don't own a Kinect, sadly I can't utilize MapDepthFrameToColorFrame method from the official Kinect API.</p>
  
  <p>From the given cameras' intrinsics and distortion coefficients, I could map the depth to world coordinates, and back to RGB based on the algorithm provided <a href="http://burrus.name/index.php/Research/KinectCalibration" rel="nofollow noreferrer">here</a></p>
  
  <pre><code>Vec3f depthToW( int x, int y, float depth ){
      Vec3f result;
      result[0] = (float) (x - depthCX) * depth / depthFX;
      result[1] = (float) (y - depthCY) * depth / depthFY;
      result[2] = (float) depth;
      return result;
  }
  
  Vec2i wToRGB( const Vec3f &amp; point ) {
      Mat p3d( point );
      p3d = extRotation * p3d + extTranslation;
  
      float x = p3d.at&lt;float&gt;(0, 0);
      float y = p3d.at&lt;float&gt;(1, 0);
      float z = p3d.at&lt;float&gt;(2, 0);
  
      Vec2i result;
      result[0] = (int) round( (x * rgbFX / z) + rgbCX );
      result[1] = (int) round( (y * rgbFY / z) + rgbCY );
      return result;
  }
  
  void map( Mat&amp; rgb, Mat&amp; depth ) {
      /* intrinsics are focal points and centers of camera */
      undistort( rgb, rgb, rgbIntrinsic, rgbDistortion );
      undistort( depth, depth, depthIntrinsic, depthDistortion );
  
      Mat color = Mat( depth.size(), CV_8UC3, Scalar(0) );
      ushort * raw_image_ptr;
  
      for( int y = 0; y &lt; depth.rows; y++ ) {
          raw_image_ptr = depth.ptr&lt;ushort&gt;( y );
  
          for( int x = 0; x &lt; depth.cols; x++ ) {
              if( raw_image_ptr[x] &gt;= 2047 || raw_image_ptr[x] &lt;= 0 )
                  continue;
  
              float depth_value = depthMeters[ raw_image_ptr[x] ];
              Vec3f depth_coord = depthToW( y, x, depth_value );
              Vec2i rgb_coord   = wToRGB( depth_coord );
              color.at&lt;Vec3b&gt;(y, x) = rgb.at&lt;Vec3b&gt;(rgb_coord[0], rgb_coord[1]);
          }
      }
  </code></pre>
  
  <p>But the result seems to be misaligned. I can't manually set the translations, since the dataset is obtained from 3 different Kinects, and each of them are misaligned in different direction. You could see one of it below (Left: undistorted RGB, Middle: undistorted Depth, Right: mapped RGB to Depth)</p>
  
  <p><img src="https://i.stack.imgur.com/Vc6Cr.png" alt="enter image description here"></p>
  
  <p>My question is, what should I do at this point? Did I miss a step while trying to project either depth to world or world back to RGB? Can anyone who has experienced with stereo camera point out my missteps?</p>