title: GLSL highlight mesh on mouse over
body: <p>I am working on a raycaster for my opengl project and I want to be able to highlight the part of the mesh that I am hovering over.</p>
  
  <p>I get a direction vector from my mouse position using this:</p>
  
  <pre><code>glm::vec3 Engine::Physics::RayCast::ViewToWorldSpace(glm::vec2 screenPos, 
  float depth, glm::mat4 projection, glm::mat4 view, graphics::Window* window)
  {
  //Screen to Normalised Device Coordinates
  float x = (2.0f * screenPos.x) / window-&gt;getWidth() - 1.0f;
  float y = 1.0f - (2.0f * screenPos.y) / window-&gt;getHeight();
  float z = 1.0f;
  glm::vec3 ray_nds = glm::vec3(x, y, z);
  
  //Normalised Device Coordinates to 4d Homogeneous Clip Coordinates
  glm::vec4 ray_clip = glm::vec4(ray_nds.x, ray_nds.y, -1.0, 1.0);
  
  //4d Homogeneous Clip Coordinates to Eye (Camera) Coordinates
  glm::vec4 ray_eye = glm::inverse(projection) * ray_clip;
  ray_eye = glm::vec4(ray_eye.x, ray_eye.y, -1.0, 0.0);
  
  //Eye (Camera) Coordinates to 4d World Coordinates
  glm::vec3 ray_wor = glm::inverse(view) * ray_eye;
  ray_wor = glm::normalize(ray_wor);
  return ray_wor;
  }
  </code></pre>
  
  <p>I then want to check if I pointing at part of a mesh, so I pass the camera position and this direction vector to these shaders.</p>
  
  <p>Vertex</p>
  
  <pre><code>#version 410
  
  layout (location = 0) in vec3 vertex_position;
  layout (location = 2) in vec2 VertexUV;
  
  uniform mat4 P;
  uniform mat4 V = mat4(1.0);
  uniform mat4 M = mat4(1.0);
  out vec2 uv;
  out vec4 position;
  
  uniform vec3 cam_pos;
  out vec3 cameraPos;
  uniform vec3 ray_dir;
  out vec3 rayDir;
  
  void main () {
  
  rayDir = vec3(M * vec4(ray_dir,1.0));
  cameraPos = vec3(V * M * vec4(cam_pos,1.0)); 
  gl_Position = P * V * M * vec4(vertex_position, 1);
  position = V * M * vec4(vertex_position, 1.0);
  uv = VertexUV;
  }
  </code></pre>
  
  <p>Frag</p>
  
  <pre><code>#version 410
  
  out vec4 fragment_colour; // final colour of surface
  
  in vec4 position;
  in vec2 uv;
  
  uniform vec3 light_pos;
  uniform vec3 light_ambient;
  uniform sampler2D texture2D;
  
  in vec3 cameraPos;
  in vec3 rayDir;
  
  void main () {
  vec4 test = vec4(0, 0, 0, 0);
  
  vec3 vDir = normalize(position.xyz - cameraPos);
  float cosAngle = dot(vDir, rayDir);
  float angle = degrees(acos(cosAngle));
  
  if(angle &lt; 5)
  {
      test = vec4(1, 0, 0, 1);
  }
  
  
  float intensity = (1.0 / length(position.xyz - light_pos))+0.25;
  intensity = clamp(intensity, 0, 1);
  vec4 ambient = vec4(light_ambient, 1);
  fragment_colour = ((vec4(texture(texture2D, uv).rgb, 1.0) * intensity) * 
  ambient)+test;
  }
  </code></pre>
  
  <p>Currently I can see the highlight section sometimes at certain camera rotations (the camera orbits around the model) but it only ever really follows the mouse when the camera is facing directly down the -Z axis. Any idea what I am doing wrong?</p>
  
  <p>Here is a gif of it working with the camera lined up, but remember it breaks if the camera moves.</p>
  
  <p><a href="https://i.stack.imgur.com/DCDZW.gif" rel="nofollow noreferrer">Working at correct camera angle</a></p>