title: Anchoring Rendered 3d Model to CharUcoBoard with OpenCV to OpenGL Pose Estimation
body: <p>I am having difficulty spotting the issue in my code with regards to an OpenCV to OpenGL pseudo augmented reality program. I say &quot;pseudo&quot; because the constraint is taking images and mat objects from openCV and converting them to OpenGL.</p>
  <p>I have working models rendered, have used the cv2.rodrigues method and camera calibration to come up with some translation and rotation vectors, I have inverted the matrix and taken the transpose for proper display in the openGL context. The issue I have stems from the fact that the model gets warped when I rotate my CharUcoBoard and begins to drift in the opposite direction when I move my camera. I assume it has something to do with the fact I am using outdated tech, on an unsupported platform (MacOS Big Sur, 11.6). Unfortunately I do not have time to switch platforms or utilize more abstracted software. Ergo, I am stuck with what I got.</p>
  <p>I have images, but since this is a new account am unable to post them directly.</p>
  <p>I have tried several methods, including hard coding in my calibrated camera parameters, modifying the initial identity matrix to coincide with my projected matrix and adding additional calibration images from all manner of angles, rotations and camera distances.</p>
  <p>Ive tried using glOrtho and glMultMatrixf to no avail. Perhaps I missed something in my code.</p>
  <p>The math part:</p>
  <pre><code>    self.INVERSE_MATRIX = np.array([[1.0, 1.0, 1.0, 1.0],
                                 [-1.0, -1.0, -1.0, -1.0],
                                 [-1.0, -1.0, -1.0, -1.0],
                                 [1.0, 1.0, 1.0, 1.0]])
  
      def _boardHandle(self, image):
  
          gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
          res = cv2.aruco.detectMarkers(gray, dictionary, parameters=arucoParams)
          corners = res[0]
          ids = res[1]
  
          if ids is not None:
              res2 = cv2.aruco.interpolateCornersCharuco(corners, ids, gray,
                                                         board)
              charucoRetval = res2[0]
              charucoCorners = res2[1]
              charucoIds = res2[2]
  
              # with camera calibration
              retval, rvecs, tvecs = cv2.aruco.estimatePoseCharucoBoard(charucoCorners, charucoIds, board,
                                                                        self.mtx, self.dist, self.rvecs, self.tvecs)
  
              if retval:
                  rmtx = cv2.Rodrigues(rvecs)[0]
                  view_matrix = np.array([[rmtx[0][0], rmtx[0][1], rmtx[0][2], tvecs[0]],
                                      [rmtx[1][0], rmtx[1][1], rmtx[1][2], tvecs[1]],
                                      [rmtx[2][0], rmtx[2][1], rmtx[2][2], tvecs[2]],
                                      [0.0, 0.0, 0.0, 1.0]], dtype=object)
                  view_matrix = view_matrix * self.INVERSE_MATRIX
                  view_matrix = np.transpose(view_matrix)
                  return view_matrix
          return self.INVERSE_MATRIX
  </code></pre>
  <p>The rendering pipeline and frame grabbing:</p>
  <pre><code>    def _draw_scene(self):
          glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
          glLoadIdentity()
  
          # get image from webcam
          image = self.webcam.get_current_frame()
  
          # convert image to OpenGL texture format
          bg_image = cv2.flip(image, 0)
          bg_image = Image.fromarray(bg_image)
          ix = bg_image.size[0]
          iy = bg_image.size[1]
          bg_image = bg_image.tobytes(&quot;raw&quot;, &quot;BGRX&quot;, 0, -1)
  
          # create background texture
          glBindTexture(GL_TEXTURE_2D, self.bgTexture)
          glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
          glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
          glTexImage2D(GL_TEXTURE_2D, 0, 3, ix, iy, 0, GL_RGBA, GL_UNSIGNED_BYTE, bg_image)
  
          # draw background
          glBindTexture(GL_TEXTURE_2D, self.bgTexture)
          glPushMatrix()
          glTranslatef(0.0, 0.0, -10.0)
          self._draw_background()
          glPopMatrix()
  
          # handle boards/pieces
          image = self._boardHandle(image)
  
          #drawing the other pieces
  
          glutSwapBuffers()
  </code></pre>
  <p>The above blocks handle the camera calibration files, the initial creation of the modelview matrix, and the rendering of the chessboard. I had originally assumed that estimating the pose and then utilizing that in a modelview matrix would have given me an accurate rendering, which seems to be the case when the camera is just shy of centered above the board. Im relatively new at openGL however and may have missed something crucial.</p>
  <p>I know display lists are outdated as well, but at this point have been unable to make the above work with VAO's, VBO's or EBO's. That will have to be a future project for me.</p>
  <p>Any help/pointers or resources will be appreciated.</p>