title: Setting up ray (origin, direction) and triangle intersection(without glm)
body: <p>Edit3 : My problems were in completely different functions than i expected. ill let the code stay, maybe this helps someone :) (and dont forget to debug!).</p>
  
  <p>Im trying to find the vector where a line intersects with a triangle.</p>
  
  <p>Current state: Random intersections even if mouse is not at the floor and camera view dependend (lookat matrix)</p>
  
  <p><strong>Steps</strong></p>
  
  <ol>
  <li>Unproject mouse coordinations</li>
  <li>Check line / triangle intersection</li>
  </ol>
  
  <p><strong>Unproject mouse coordinations</strong></p>
  
  <p>I checked the source of glm::unproject and gluUnproject and created this function.</p>
  
  <pre><code>   pixel::CVector3 pixel::CVector::unproject(
      CVector2 inPosition,
      pixel::CShape window,
      pixel::matrix4 projectionMatrix,
      pixel::matrix4 modelViewMatrix,
      float depth
      )
  {
      // transformation of normalized coordinates
      CVector4 inVector;
      inVector.x = (2.0f * inPosition.x) / window.width - 1.0f;
      inVector.y = (2.0f * inPosition.y) / window.height - 1.0f;
      inVector.z = 2.0f * depth - 1.0f;
      inVector.w = 1.0f;
  
      // multiply inverted matrix with vector
      CVector4 rayWorld = pixel::CVector::multMat4Vec4(pixel::CMatrix::invertMatrix(projectionMatrix * modelViewMatrix), inVector);
  
      CVector3 result;
      result.x = rayWorld.x / rayWorld.w;
      result.y = rayWorld.y / rayWorld.w;
      result.z = rayWorld.z / rayWorld.w;
  
  
  
      return result;
  }
  </code></pre>
  
  <p><strong>Checking intersection</strong></p>
  
  <pre><code>pixel::CVector3 pixel::Ray::intersection(
      Ray ray,
      pixel::CVector3 v0,
      pixel::CVector3 v1,
      pixel::CVector3 v2
      )
  {
      // compute normal
      CVector3 a, b, n;
      a = v1 - v0;
      b = v2 - v0;
  
      n = ray.direction.cross(b);
  
      // find determinant
      float det = a.dot(n);
  
      if (det &lt; 0.000001f)
      {
          std::cout &lt;&lt; "Ray intersecting with backface triangles \n";
          return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
      }
      det = 1.0f / det;
  
      // calculate distance from vertex0 to ray origin
      CVector3 s = ray.origin - v0;
      float u = det * s.dot(n);
  
      if (u &lt; -0.000001f || u &gt; 1.f + 0.000001f)
      {
          std::cout &lt;&lt; "U: Intersection outside of the triangle!\n";
          return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
      }
  
      CVector3 r = s.cross(a);
      float v = det * ray.direction.dot(r);
      if (v &lt; -0.000001f || u + v &gt; 1.f + 0.000001f)
      {
          std::cout &lt;&lt; "V/U: Intersection outside of triangle!\n";
          return pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
      }
  
      // distance from ray to triangle
      det = det *  b.dot(r);
  
      std::cout &lt;&lt; "T: " &lt;&lt; det &lt;&lt; "\n";
  
      CVector3 endPosition;
      endPosition.x = ray.origin.x + (ray.direction.x * det);
      endPosition.y = ray.origin.y + (ray.direction.y * det);
      endPosition.z = ray.origin.z + (ray.direction.z * det);
  
      return endPosition;
  }
  </code></pre>
  
  <p><strong>Usage</strong></p>
  
  <pre><code>    if (event.button.button == SDL_BUTTON_RIGHT)
              {
  
                  camera-&gt;setCameraActive();
                  float mx = event.motion.x;
                  float my = window-&gt;info.height - event.motion.y;
                  // ray casting
                  pixel::Ray ray;
  
                  std::cout &lt;&lt; "\n\n";
  
                      // near
                  pixel::CVector3 rayNear = pixel::CVector::unproject(
                      pixel::CVector::vector2(mx, my),
                      pixel::CVector::shape2(window-&gt;info.internalWidth, window-&gt;info.internalHeight),
                      camera-&gt;camInfo.currentProjection,
                      camera-&gt;camInfo.currentView,
                      1.0f
                      );
                  // far
                  pixel::CVector3 rayFar = pixel::CVector::unproject(
                      pixel::CVector::vector2(mx, my),
                      pixel::CVector::shape2(window-&gt;info.internalWidth, window-&gt;info.internalHeight),
                      camera-&gt;camInfo.currentProjection,
                      camera-&gt;camInfo.currentView,
                      0.0f
                      );
  
  
                  // normalized direction results in the same behavior
                  ray.origin = cameraPosition;
  
                  ray.direction = pixel::CVector::normalize(rayFar- rayNear);
  
                  std::cout &lt;&lt; "Raycast \n";
                  std::cout &lt;&lt; "Mouse Position: " &lt;&lt; mx &lt;&lt; " - " &lt;&lt; my &lt;&lt; "\n";
                  std::cout &lt;&lt; "Camera Position: " &lt;&lt; ray.origin.x &lt;&lt; " - " &lt;&lt; ray.origin.y &lt;&lt; " - " &lt;&lt; ray.origin.z &lt;&lt; "\n";
                  std::cout &lt;&lt; "Ray direction: " &lt;&lt; ray.direction.x &lt;&lt; " - " &lt;&lt; ray.direction.y &lt;&lt; " - " &lt;&lt; ray.direction.z &lt;&lt; "\n";
  
  
                  pixel::CVector3 vertOne = pixel::CVector::vector3(0.0f, 0.0f, -300.0f);
                  pixel::CVector3 vertTwo = pixel::CVector::vector3(0.0f, 0.0f, 0.0f);
                  pixel::CVector3 vertThree = pixel::CVector::vector3(300.0f, 0.0f, 0.0f);
                  pixel::CVector3 vertFour = pixel::CVector::vector3(300.0f, 0.0f, -300.0f);
  
  
                  pixel::CVector3 rayHit = pixel::Ray::intersection(ray, vertOne, vertTwo, vertThree);
                  pixel::CVector3 rayHit2 = pixel::Ray::intersection(ray, vertThree, vertFour, vertOne);
                  std::cout &lt;&lt; "Ray hit: " &lt;&lt; rayHit.x &lt;&lt; " - " &lt;&lt; rayHit.y &lt;&lt; " - " &lt;&lt; rayHit.z &lt;&lt; "\n";
                  std::cout &lt;&lt; "Ray hit: " &lt;&lt; rayHit2.x &lt;&lt; " - " &lt;&lt; rayHit2.y &lt;&lt; " - " &lt;&lt; rayHit2.z &lt;&lt; "\n";
                  std::cout &lt;&lt; "--------------------\n";
                  towerHouse-&gt;modelMatrix = pixel::CMatrix::translateMatrix(rayHit);
  </code></pre>
  
  <p><strong>Output</strong></p>
  
  <p>As ive never used glm::unproject or gluUnproject, i dont know how the normal output should look like, but im getting results like:</p>
  
  <p>Ray direction: 0.109035  -0.0380502  0.0114562 </p>
  
  <p>Doesnt look right to me, but checking my code against other sources (mentioned above), i dont see the mistake/s.</p>
  
  <p>Ray intersection works in some special cases (camera rotation) and even then i get intersections even if i dont click on the floor.
  Same goes with intersection output varying from backface hits to outside of the triangle.</p>
  
  <p>All those errors look like the main source of problem is the unprojection.</p>
  
  <p>Any hints in the right direction?</p>