Why is the backprojection of the Middlesbury depth map "cone shaped"? I wrote some come to backproject depth maps. Testing it with the Middlebury cone set, I get:



First of all it has noise, which I shouldn't get from the ground truth, but more importantly it seems like the points are going further from each other with the depth.

Can anybody explain this phenomenon?

My code:

  float baseline = 0.3;
  float focalLength = 303.0;
  float numerator = baseline * focalLength;    

  for (int i=10; i&lt;groundTruthDisparity.rows-10; ++i) {
    for (int j=10; j&lt;groundTruthDisparity.cols-10; ++j) {
      float distance = groundTruthDisparity.at&lt;uchar&gt;(i,j) / 4.0;

      if (distance != 0 &amp;&amp; !isnan(distance)) {
        depthMap.at&lt;float&gt;(i,j) = numerator / distance;
      }
    }
  }

  for(int y=1; y&lt;height-1; y++) {
    for(int x=1; x&lt;width-1; x++) {
      float depth = depthMap.at&lt;float&gt;(y,x);

      // backproject to world
      cv::Matx21d homogeneous = cv::normalize((cv::Matx21d(x,y) - principalPoint) * (1.0 / focalDistance));
      cv::Matx31d transformedPoint = cv::Matx21d(homogeneous(0), homogeneous(1), 1) * depth;

      cloudPoints.push_back(transformedPoint);
    }
  }


The depth map looks like this (colored in HSV):
The reason that pixels get wider with depth is that the depth map works in projection space, so each pixel is basically shaped like a frustum, not like a box.  Not sure where the noise is coming from but the depth map doesn't look perfectly clean either.