title: Straightforward solution on how to stereo calibration and rectifications OpenCV?
body: <p>I have been digging on this topic for almost a week and couldn't find any solid solution yet.
  It is interesting that no one ever posted a <strong>straightforward</strong> solution on how to calibrate and rectify a stereo camera with OpenCV in order to compute the depth, from here and there(<a href="https://stackoverflow.com/questions/28222763/stereo-calibration-opencv-python-and-disparity-map">this</a> for calibration and <a href="http://vgg.fiit.stuba.sk/2015-02/2783/" rel="nofollow noreferrer">this</a> for rectification, the codes posted are not quite integrated though) I have come up with the following code snap, BUT it does not rectify the image OK!!</p>
  
  <pre><code>import numpy as np
  import cv2
  import glob
  
  # termination criteria
  criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
  
  # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
  objp = np.zeros((6*9,3), np.float32)
  objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)
  
  # Arrays to store object points and image points from all the images.
  objpoints = {} # 3d point in real world space
  imgpoints = {} # 2d points in image plane.
  
  # calibrate stereo
  for side in ['left', 'right']:
      counter = 0
      images = glob.glob('images/%s*.jpg' %side)
      objpoints[side] = [];
      imgpoints[side] = [];
      for fname in images:
          img = cv2.imread(fname)
          gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
  
          # Find the chess board corners
          ret, corners = cv2.findChessboardCorners(gray, (9,6),None)
          # If found, add object points, image points (after refining them)
          if ret == True:
              objpoints[side].append(objp)
  
              cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
              imgpoints[side].append(corners)
              counter += 1
  
      assert counter == len(images), "missed chessboard!!"
  
  
  stereocalib_criteria = (cv2.TERM_CRITERIA_MAX_ITER + cv2.TERM_CRITERIA_EPS, 100, 1e-5)
  stereocalib_flags = cv2.CALIB_FIX_ASPECT_RATIO | cv2.CALIB_ZERO_TANGENT_DIST | cv2.CALIB_SAME_FOCAL_LENGTH | cv2.CALIB_RATIONAL_MODEL | cv2.CALIB_FIX_K3 | cv2.CALIB_FIX_K4 | cv2.CALIB_FIX_K5
  retval,cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, R, T, E, F = cv2.stereoCalibrate(objpoints['left'], imgpoints['left'], imgpoints['right'], (640, 480), criteria = stereocalib_criteria, flags = stereocalib_flags)
  
  rectify_scale = 0.1 # 0=full crop, 1=no crop
  R1, R2, P1, P2, Q, roi1, roi2 = cv2.stereoRectify(cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, (640, 480), R, T, alpha = rectify_scale)
  
  left_maps = cv2.initUndistortRectifyMap(cameraMatrix1, distCoeffs1, R1, P1, (640, 480), cv2.CV_16SC2)
  right_maps = cv2.initUndistortRectifyMap(cameraMatrix2, distCoeffs2, R2, P2, (640, 480), cv2.CV_16SC2)
  
  # Assuming you have left01.jpg and right01.jpg that you want to rectify
  lFrame = cv2.imread('images/left01.jpg')
  rFrame = cv2.imread('images/right01.jpg')
  
  left_img_remap = cv2.remap(lFrame, left_maps[0], left_maps[1], cv2.INTER_LANCZOS4)
  right_img_remap = cv2.remap(rFrame, right_maps[0], right_maps[1], cv2.INTER_LANCZOS4)
  
  for line in range(0, int(right_img_remap.shape[0] / 20)):
      left_img_remap[line * 20, :] = (0, 0, 255)
      right_img_remap[line * 20, :] = (0, 0, 255)
  
  cv2.imshow('winname', np.hstack([left_img_remap, right_img_remap]))
  cv2.waitKey(0)
  exit(0)
  </code></pre>
  
  <p>The output of the above is the image below
  <a href="https://i.stack.imgur.com/H0oYt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/H0oYt.png" alt="enter image description here"></a></p>
  
  <p>As you can see the images are not rectified!!</p>
  
  <h3>Question:</h3>
  
  <ul>
  <li>What is wrong with the code?</li>
  </ul>