title: 3D Ray Picking use Mouse Coordinates when Mouse isn't locked
body: <p>So basically I've made a program using OpenGL which can perform 3D Ray Picking. If the <code>Camera View Direction Ray</code> touches/intersects anything (which isn't air) then a little purple box will get rendered at the intersection point/points.</p>
  
  <p>If the ray intersects with any of the "Red Boxes" the once that are intersecting with the ray will turn green. The ground and walls won't change color or texture at all.</p>
  
  <p>Examples:</p>
  
  <p><a href="https://i.stack.imgur.com/9oslW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9oslW.png" alt=""></a></p>
  
  <p>The current way I do the 3D Ray Picking, is getting the camera's view direction ray, and then just calculating for intersections. <em>My function for calculating intersections doesn't return as a boolean, it returns as a 3D Vector (the coordinates of the intersection itself)</em></p>
  
  <h2>The Question</h2>
  
  <p>So what I'm trying to achieve is to calculate the <em>Picking Ray</em>, but according to the mouse when it's <strong>not</strong> locked to the screen.</p>
  
  <p><strong>Example</strong> - So here you can see the purple box is at the crosshair, though if I were to unlocked the mouse and move it (on top of the screen, like normally) and move it over to the center of the green X mark I've draw, then I want to calculate the ray from the camera center to the mouse coordinate on top of the screen.</p>
  
  <p><a href="https://i.stack.imgur.com/xbhBD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xbhBD.png" alt=""></a></p>
  
  <h3>Current Tests and Ideas</h3>
  
  <p>This should just be a mathematical problem. Here is just a short list of the things I currently use to calculate the Ray (and trying to calculate the second ray)</p>
  
  <ul>
  <li>Cameara X, Y, Z</li>
  <li>Camera Pitch Yaw Roll (Roll not used right now)</li>
  <li>Camera Near Far (distances)</li>
  <li>Camera Fov</li>
  <li>Camera Aspect</li>
  <li>Mouse X, Y (On top of the screen itself)</li>
  <li>Sceen Width, Height</li>
  </ul>
  
  <p><em>The Mouse X &amp; Y origin (0x0) is in the bottom left corner of the window/frame.</em></p>
  
  <h3>Calculating the main <em>Picking Ray</em> itself</h3>
  
  <pre><code>Vector3D position = new Vector3D(
          camera.x,
          camera.y,
          camera.z);
  
  Vector3D direction = new Vector3D(
          Math.cos(Math.toRadians(camera.pitch)) * -Math.sin(Math.toRadians(-camera.yaw)) * camera.far,
          Math.cos(Math.toRadians(camera.pitch)) * cameara.far,
          Math.cos(Math.toRadians(camera.pitch)) * -Math.sin(Math.toRadians(-camera.yaw)) * camera.far);
  
  direction.normalize();
  
  Ray3D ray = new Ray(position, direction);
  </code></pre>
  
  <p>That is how I calculate the main picking ray itself (the picking ray for the locked mouse). I made the classes myself, though they should make sense (<code>Vector3D</code>, <code>Ray3D</code>, etc) and the <code>normalize()</code> methods does exactly what it says, normalizes the vector.</p>
  
  <h3>Idea</h3>
  
  <p>So when I tried to calculate using the mouse coordinates I inserted the following code right before I'm calling <code>direction.normalize();</code>, so right after creating the <code>Vector3D direction</code>.</p>
  
  <pre><code>if (!Mouse.isGrabbed())
  {
      float mx = Mouse.getX() / (float) scene.width - 0.5f;
      float my = Mouse.getY() / (float) scene.height - 0.5f;
  
      mx *= camera.far;
      my *= camera.far;
  
      line.b.x += mx;
      line.b.y += my;
      line.b.z += mz;
  }
  </code></pre>
  
  <p>That gives me a weird result, when the mouse isn't locked/grabbed. It makes sense since I was just messing around and trying some of the things that came first in my mind.</p>
  
  <p>I'm guessing that I need to translate the mouse coordinates according to the pitch, yaw and roll. Though I don't have any idea, how I would go about doing that.</p>
  
  <p>So I'm hoping that there is somebody that can help me achieving this and/or give me some sort of resource so I can understand how to do what I'm trying to do.</p>
  
  <h3>Extra</h3>
  
  <p>If you need more info about this, just write a comment and I will do my best.</p>
  
  <h3>Answer - Thanks to <a href="https://stackoverflow.com/users/486617/fen">fen</a></h3>
  
  <p>I ended up using <a href="https://stackoverflow.com/users/486617/fen">fen</a>'s way for now, since it was a lot simpler than having to calculate everything!</p>
  
  <pre><code>FloatBuffer projection = BufferTools.createFloatBuffer(16);
  FloatBuffer modelview = BufferTools.createFloatBuffer(16);
  IntBuffer viewport = BufferTools.createIntBuffer(16);
  
  glGetFloat(GL_PROJECTION_MATRIX, projection);
  glGetFloat(GL_MODELVIEW_MATRIX, modelview);
  glGetInteger(GL_VIEWPORT, viewport);
  
  float win_x = Mouse.getX();
  float win_y = Mouse.getY();
  
  FloatBuffer position_near = BufferTools.createFloatBuffer(3);
  FloatBuffer position_far = BufferTools.createFloatBuffer(3);
  
  gluUnProject(win_x, win_y, 0f, modelview, projection, viewport, position_near);
  gluUnProject(win_x, win_y, 1f, modelview, projection, viewport, position_far);
  
  Ray3D ray = new Ray3D(
      new Vector3D(
          position_near.get(0),
          position_near.get(1),
          position_near.get(2)),
      new Vector3D(
          position_far.get(0),
          position_far.get(1),
          position_far.get(2)));
  </code></pre>