title: Object picking with ray casting
body: <p>I've been having a problem with inaccuracies in my ray casting algorithm for detecting mouse hits within a box. I'm completely at a loss as to how to fix this properly and it's been bugging me for weeks.</p>
  
  <p>The problem is easiest described with a picture (box centered around [0, 0, -30]):</p>
  
  <p><img src="https://i.imgur.com/t8YVj.jpg" alt="screen shot of problem"></p>
  
  <p>The black lines represent the actual hitbox which is drawn and the green box represents what actually appears to get hit. Notice how it's offset (which seems to get larger if the box is further from the origin) and is slightly smaller than the drawn hitbox.</p>
  
  <p>Here's some relevant code,</p>
  
  <p><strong>ray-box cast:</strong></p>
  
  <pre><code>double BBox::checkFaceIntersection(Vector3 points[4], Vector3 normal, Ray3 ray) {
  
      double rayDotNorm = ray.direction.dot(normal);
      if(rayDotNorm == 0) return -1;
  
      Vector3 intersect = points[0] - ray.origin;
      double t = intersect.dot(normal) / rayDotNorm;
      if(t &lt; 0) return -1;
  
      // Check if first point is from under or below polygon
      bool positive = false;
      double firstPtDot = ray.direction.dot( (ray.origin - points[0]).cross(ray.origin - points[1]) );
      if(firstPtDot &gt; 0) positive = true;
      else if(firstPtDot &lt; 0) positive = false;
      else return -1;
  
      // Check all signs are the same
      for(int i = 1; i &lt; 4; i++) {
          int nextPoint = (i+1) % 4;
          double rayDotPt = ray.direction.dot( (ray.origin - points[i]).cross(ray.origin - points[nextPoint]) );
          if(positive &amp;&amp; rayDotPt &lt; 0) {
              return -1;
          }
          else if(!positive &amp;&amp; rayDotPt &gt; 0) {
              return -1;
          }
      }
  
      return t;
  }
  </code></pre>
  
  <p><strong>mouse to ray:</strong></p>
  
  <pre><code>GLint viewport[4];
  GLdouble modelMatrix[16];
  GLdouble projectionMatrix[16];
  
  glGetIntegerv(GL_VIEWPORT, viewport);
  glGetDoublev(GL_MODELVIEW_MATRIX, modelMatrix);
  glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);
  
  GLfloat winY = GLfloat(viewport[3] - mouse_y);
  
  Ray3 ray;
  double x, y, z;
  gluUnProject( (double) mouse_x, winY, 0.0f, // Near
                modelMatrix, projectionMatrix, viewport,
                &amp;x, &amp;y, &amp;z );
  ray.origin = Vector3(x, y, z);
  
  gluUnProject( (double) mouse_x, winY, 1.0f, // Far
                modelMatrix, projectionMatrix, viewport,
            &amp;x, &amp;y, &amp;z );
  ray.direction = Vector3(x, y, z);
  
  if(bbox.checkBoxIntersection(ray) != -1) {
      std::cout &lt;&lt; "Hit!" &lt;&lt; std::endl;
  }
  </code></pre>
  
  <p>I've tried drawing the actual ray as a line and it seems to intersect the drawn box correctly.</p>
  
  <p>I had the offset problem partially fixed by minusing all the points and the ray origin/direction by the boxes position, but I have no idea why that worked and the size of the hitbox still remained inaccurate.</p>
  
  <p>Any ideas/alternative approaches? I have other code to supply if it's needed.</p>