title: Image-Processing: Converting normal pictures into FishEye images with intrinsic matrix
body: <p>I need to synthesize many FishEye images with different intrinsic matrices based on normal pictures. I am following the method mentioned in this <a href="https://arxiv.org/pdf/1804.04784.pdf" rel="nofollow noreferrer">paper</a>.</p>
  
  <p>Ideally, if the algorithm is correct, the ideal fish eye effect should look like this:</p>
  
  <p><img src="https://i.stack.imgur.com/CPS1I.jpg" alt="ideal fish eye effect">. </p>
  
  <p>But when I used my algorithm to convert a picture</p>
  
  <p><img src="https://i.stack.imgur.com/fQLRL.jpg" alt="originial image"> </p>
  
  <p>it looks like this </p>
  
  <p><img src="https://i.stack.imgur.com/c3J2x.png" alt="effect"></p>
  
  <p>So below is my code's flow:
  1. First, I read the raw image with cv2</p>
  
  <pre><code>def read_img(image):
  
      img = ndimage.imread(image) #this would return a 4-d array: [R,G,B,255]
      img_shape = img.shape
      print(img_shape)
  
      #get the pixel coordinate
      w = img_shape[1] #the width
      # print(w)
      h= img_shape[0] #the height
      # print(h)
      uv_coord = []
      for u in range(w):
      for v in range(h):
          uv_coord.append([float(u),float(v)])  #this records the coord in the fashion of [x1,y1],[x1, y2], [x1, y3]....
      return np.array(uv_coord)
  </code></pre>
  
  <p>Then, based on the paper:</p>
  
  <p><strong><em>r(θ) = k1θ + k2θ^3 + k3θ^5 + k4θ^7,  (1)</em></strong>
  where Ks are the distorted coefficients             </p>
  
  <p>Given pixel coordinates (x,y) in the pinhole projection image, the corresponding image coordinates (x',y')in the fisheye can be computed as:</p>
  
  <p><strong><em>x'=r(θ) cos(ϕ), y' = r(θ) sin(ϕ), (2)</em></strong></p>
  
  <p>where <strong><em>ϕ = arctan((y − y0)/(x − x0))</em></strong>, and (x0, y0) are the coordinates of the principal point in the pinhole projection image. </p>
  
  <p>And then the image coordinates (x',y') is converted into pixel coordinates (xf,yf): (xf, yf): 
  <strong>*xf = mu * x' + u0, yf = mv * y' + v0,*</strong> (3)</p>
  
  <p>where (u0, v0) are the coordinates of the principle points in the fisheye, and mu, mv denote the number of pixels per unit distance in the horizontal and vertica directions. So I am guessing there are just from the <strong><em>intrinsic matrix</em></strong> [fx, fy] and u0 v0 are the [cx, cy]. </p>
  
  <pre><code>def add_distortion(sourceUV, dmatrix,Kmatrix):
      '''This function is programmed to remove the pixel of the given original image coords
      input arguments:
      dmatrix          -- the intrinsic matrix [k1,k2,k3,k4] for tweaking purposes
      Kmatrix          -- [fx, fy, cx, cy, s]'''
      u = sourceUV[:,0] #width in x
      v = sourceUV[:,1] #height in y
  
      rho = np.sqrt(u**2 + v**2) 
  
      #get theta
      theta = np.arctan(rho,np.full_like(u,1))
  
      # rho_mat = np.array([rho, rho**3, rho**5, rho**7])
      rho_mat = np.array([theta,theta**3, theta**5, theta**7])
  
      #get the: rho(theta) = k1*theta + k2*theta**3 + k3*theta**5 + k4*theta**7
      rho_d = dmatrix@rho_mat
  
      #get phi
      phi = np.arctan2((v - Kmatrix[3]), (u - Kmatrix[2]))
      xd = rho_d * np.cos(phi)
      yd = rho_d * np.sin(phi)
  
      #converting the coords from image plane back to pixel coords
      ud = Kmatrix[0] * (xd + Kmatrix[4] * yd) + Kmatrix[2]
      vd = Kmatrix[1] * yd + Kmatrix[3]
      return np.column_stack((ud,vd))
  </code></pre>
  
  <p>Then after gaining the distorded coordinates, I perform moving pixels in this way, where I think the problem might be:</p>
  
  <pre><code>def main():
      image_name = "original.png"
      img = cv2.imread(image_name)
      img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR) #the cv2 read the image as BGR
  
      w = img.shape[1]
      h = img.shape[0]
      uv_coord = read_img(image_name)
  
      #for adding distortion
      dmatrix = [-0.391942708316175,0.012746418822063 ,-0.001374061848026 ,0.005349692659231]
  
      #the Intrinsic matrix of the original picture's 
      Kmatrix = np.array([9.842439e+02,9.808141e+02 , 1392/2, 2.331966e+02, 0.000000e+00])
  
      # Kmatrix = np.array([2234.23470710156  ,2223.78349134123,  947.511596277837,   647.103139639432,-3.20443253476976]) #the distorted intrinsics
      uv = add_distortion(uv_coord,dmatrix,Kmatrix)
  
      i = 0
      dstimg = np.zeros_like(img)
  
      for x in range(w):   #tthe coo
          for y in range(h):
             if i &gt; (512 * 1392 -1):
                 break
  
              xu = uv[i][0] #x, y1, y2, y3
              yu = uv[i][1]
              i +=1
  
              # if new pixel is in bounds copy from source pixel to destination pixel
              if 0 &lt;= xu and xu &lt; img.shape[1] and 0 &lt;= yu and yu &lt; img.shape[0]:
                  dstimg[int(yu)][int(xu)] = img[int(y)][int(x)]
  
      img = Image.fromarray(dstimg, 'RGB')
      img.save('my.png')
      img.show()
  </code></pre>
  
  <p>However, this code does not perform in the way I want. Could you guys please help me with debugging it? I spent 3 days but I still could not see any problem with it. Thanks!!</p>