title: OpenCV - Python - Stereo camera - undistortPoints() different results from initUndistortRectifyMap() + remap()
body: <p>I have a calibrated stereo camera rig and want to rectify the coordinates of some tracked image points.</p>
  
  <p>From the stereo calibration the camera matrices (ML, MR), the distortion coefficients (DL, DR), the rotation matrix (R) between those cameras and the translation vector (T) are obtained.</p>
  
  <p>To get the parameters for the rectification the following function is called</p>
  
  <pre><code>RL, RR, PL, PR, Q, _, _ = cv2.stereoRectify(ML, DL, MR, DR, IMG_SIZE, R, T, alpha=1)
  </code></pre>
  
  <p><strong>Rectify whole image</strong></p>
  
  <p>Now i want to rectify the images from the left and right camera. Before doing so, red circles are drawn around the tracked coordinates. To rectify the images I call:</p>
  
  <pre><code>map_l = cv2.initUndistortRectifyMap(ML, DL, RL, PL, IMG_SIZE, cv2.CV_32FC1)
  map_r = cv2.initUndistortRectifyMap(MR, DR, RR, PR, IMG_SIZE, cv2.CV_32FC1)
  
  img_l = draw_circles(img_l, coordinates_left, "red")  # white image with red circles
  img_r = draw_circles(img_r, coordinates_right, "red") 
  
  img_rect_l = cv2.remap(img_l, map_l[0], map_l[1], cv2.INTER_LINEAR)
  img_rect_r = cv2.remap(img_r, map_r[0], map_r[1], cv2.INTER_LINEAR)
  </code></pre>
  
  <p><strong>Rectify tracked points</strong></p>
  
  <p>Instead of rectifying the whole image I just want to rectify the tracked point coordinates, therefore I use undistortPoints() and draw those over the rectified images in green.</p>
  
  <pre><code>coordinates_left_rectified = cv2.undistortPoints(coordinates_left, ML, DL, R=RL, P=PL)
  coordinates_right_rectified = cv2.undistortPoints(coordinates_right, MR, DR, R=RR, P=PR)
  
  img_rect_l = draw_circles(img_rect_l, coordinates_left_rectified, "green")
  img_rect_l = draw_circles(img_rect_l, coordinates_right_rectified, "green")
  </code></pre>
  
  <p>Unfortunately I don't get the same results as with remap (image below). So what am I doing wrong? Any ideas? </p>
  
  <p>Thank you!</p>
  
  <p><a href="https://i.stack.imgur.com/sQoD5.png" rel="nofollow noreferrer">image - different results</a></p>