title: Cropping to only valid pixels after remap with fisheye (equidistant) stereo rectification?
body: <p>I have two cameras offset horizontally and have acquired their calibration parameters (Camera Matrix and Distortion Coefficients, as well as the transform between them) using Kalibr under the pinhole-equidistant model (distortion coefficients <code>k1, k2, k3, k4</code>).</p>
  
  <p>I want to use openCVs <a href="https://docs.opencv.org/4.1.2/db/d58/group__calib3d__fisheye.html#gac1af58774006689056b0f2ef1db55ecc" rel="nofollow noreferrer"><code>cv.fisheye.stereoRectify</code></a> to create new projection matrices for each camera that i can feed into <a href="https://docs.opencv.org/4.1.2/db/d58/group__calib3d__fisheye.html#ga0d37b45f780b32f63ed19c21aa9fd333" rel="nofollow noreferrer"><code>cv.fisheye.initUndistortRectifyMap</code></a> and then into <a href="https://docs.opencv.org/4.1.2/da/d54/group__imgproc__transform.html#gab75ef31ce5cdfb5c44b6da5f3b908ea4" rel="nofollow noreferrer"><code>cv.remap</code></a> to rectify and undistort each image.</p>
  
  <p>Unfortunately even with the <code>balance</code> parameter in fisheye.stereoRectify set to 0 the <code>remap</code>ed images still have black pixels bowing into them. I want to crop each image such that no invalid pixels exist in either of the undistorted camera images.</p>
  
  <p>I see that the standard <a href="https://docs.opencv.org/4.1.2/d9/d0c/group__calib3d.html#ga617b1685d4059c6040827800e72ad2b6" rel="nofollow noreferrer"><code>cv.stereoRectify</code></a> function has an <code>alpha</code> parameter that does exactly this. But it seems like <code>cv.fisheye.stereoRectify</code> does not have this parameter. Thus I want to replicate its features.</p>
  
  <p><code>cv.stereoRectify</code> seems to use the radtan distortion model (distortion parameters <code>k1, k2, p1, p2</code>) so I dont think i can easily swap that function in since i dont have <code>p1</code> and <code>p2</code>.</p>
  
  <p>snippet from my pipeline to follow below:</p>
  
  <pre><code>R1, R2, P1, P2, Q = cv2.fisheye.stereoRectify(mtx_right, dist_right, 
                                                mtx_left, dist_left, 
                                                (960,1280), R, tvec, 
                                                flags=cv2.CALIB_ZERO_DISPARITY,
                                                balance= 0.0, fov_scale=1)
  
  map1_right, map2_right = cv2.fisheye.initUndistortRectifyMap(mtx_right, dist_right, 
                                                               R1, P1[0:3, 0:3], 
                                                               (1280, 960), cv2.CV_16SC2)  
  
  map1_left, map2_left = cv2.fisheye.initUndistortRectifyMap(mtx_left, dist_left,
                                                             R2, P2[0:3, 0:3],
                                                             (1280, 960), cv2.CV_16SC2)
  
  undistorted_right = cv2.remap(img_rgb_right, map1_right, map2_right, 
                               interpolation=cv2.INTER_LINEAR,
                               borderMode=cv2.BORDER_CONSTANT)
  undistorted_left = cv2.remap(img_rgb_left, map1_left, map2_left, 
                              interpolation=cv2.INTER_LINEAR,
                              borderMode=cv2.BORDER_CONSTANT)
  </code></pre>
  
  <p>Is there an easy way to get the same functionality that <code>alpha</code> produces in the traditional <code>cv.stereoRectify</code>? <code>balance=0</code> seems close but doesn't completely cut off the invalid pixels.</p>
  
  <p>CURRENT OUTPUT (<code>balance=0.5</code> to zoom out a little)
  <a href="https://i.stack.imgur.com/bOgbT.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bOgbT.jpg" alt="fisheye.stereoRectify"></a></p>
  
  <p>GOAL IS FOR BOTH IMAGES TO ONLY SHOW WHATS IN THE GREEN BOX (same dimensions on either if that isn't clear,whichever has a smaller valid pixel rectangle):</p>
  
  <p><a href="https://i.stack.imgur.com/VElQZ.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VElQZ.jpg" alt="goal"></a></p>