title: How to rectify a stereo image pair using opencv?
body: <p>I have set up two cameras and calibrated them separately, after that I calibrated both using <code>stereoCalibrate</code> function. Calibration seemed to work fine, as all returned reprojection errors are in the range of 0.4 to 0.5. Now i want to calculate a depth map out of the stereo image pair. As far as i could follow the tutorials, i first need to rectify my images, grayscale them and pass them to StereoBM (or any other matcher). </p>
  
  <p>As far as i understand the code below should rectify and show two live images of the cameras.</p>
  
  <pre><code>... Load calibration matrices
  
  //compute rectification
  Mat R1, R2, P1, P2, Q;
  stereoRectify(cameraMatrix_left, distortionCoefficients_left,
     cameraMatrix_right, distortionCoefficients_right,
     Size(left_camera.get(CAP_PROP_FRAME_WIDTH),
     left_camera.get(CAP_PROP_FRAME_HEIGHT)),
     R, T, R1, R2, P1, P2, Q, 0, 0.0,
     Size(left_camera.get(CAP_PROP_FRAME_WIDTH),
     left_camera.get(CAP_PROP_FRAME_HEIGHT)));
  
  //compute undistortion
  Mat rmap[2][2];
  initUndistortRectifyMap(cameraMatrix_left, distortionCoefficients_left, R1, P1, Size(left_camera.get(CAP_PROP_FRAME_WIDTH), left_camera.get(CAP_PROP_FRAME_HEIGHT)), CV_16SC2, rmap[0][0], rmap[0][1]);
  initUndistortRectifyMap(cameraMatrix_right, distortionCoefficients_right, R2, P2, Size(right_camera.get(CAP_PROP_FRAME_WIDTH), right_camera.get(CAP_PROP_FRAME_HEIGHT)), CV_16SC2, rmap[1][0], rmap[1][1]);
  
  while (true) {
      if (!right_camera.read(capturedFrame_right))
          break;
      if (!left_camera.read(capturedFrame_left))
          break;
  
      remap(capturedFrame_left, drawFrame_left, rmap[0][0], rmap[0][1], INTER_LINEAR, BORDER_DEFAULT, Scalar());
      remap(capturedFrame_right, drawFrame_right, rmap[1][0], rmap[1][1], INTER_LINEAR, BORDER_DEFAULT, Scalar());
  
      cvtColor(drawFrame_left, grayScale_left, COLOR_RGB2GRAY);
      cvtColor(drawFrame_right, grayScale_right, COLOR_RGB2GRAY);
  
      imshow(RIGHT_CAMERA, grayScale_right);
      imshow(LEFT_CAMERA, grayScale_left);
  }
  </code></pre>
  
  <p>I would expect that both images are rectified as shown on <a href="https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#void%20stereoRectify(InputArray%20cameraMatrix1,%20InputArray%20distCoeffs1,%20InputArray%20cameraMatrix2,%20InputArray%20distCoeffs2,%20Size%20imageSize,%20InputArray%20R,%20InputArray%20T,%20OutputArray%20R1,%20OutputArray%20R2,%20OutputArray%20P1,%20OutputArray%20P2,%20OutputArray%20Q,%20int%20flags,%20double%20alpha,%20Size%20newImageSize,%20Rect*%20validPixROI1,%20Rect*%20validPixROI2)" rel="nofollow noreferrer">the documentation of stereoRectify</a>.</p>
  
  <p>However, they are simply not. There is a noticable vertical difference between both pictures.
  What did i miss?</p>