Ray→plane and ray→quad intersection
I'll preface this by saying that I'm not a Haskell programmer, so take my comments advisedly.


I would appreciate a short comment specifying the behaviour of each type and function. For example:

-- The point of intersection between a ray and a plane.
hitPlane :: (Fractional a) =&gt; Ray a -&gt; Plane a -&gt; V3 a


This allows a reader to quickly understand the purpose of a function without having to read the code to find out. Also it provides a specification that can be checked against the implementation.
In geometry the term ray usually refers to the half-line with a starting point and direction. So a ray does not necessarily hit a plane, and the type of hitPlane ought to be:

hitPlane :: (RealFrac a) =&gt; Ray a -&gt; Plane a -&gt; Maybe (V3 a)


If you intend your Ray type to be a full line, then it ought to be called Line instead, to avoid confusion. But even a full line might not have a point of intersection with a plane, because it might be parallel to the plane. In this case there will be a division by zero error: it would be better to avoid this and return Nothing.
I don't understand all the details of the class constraints. hitPlane requires both Num a and Fractional a but if I understand the standard Haskell type documentation, the latter implies the former, so the Num a is redundant. Similarly for hitQuad, where Num a is redundant given that you have Floating a.
hitQuad requires Epsilon a and Floating a, but I don't see how either of these classes is necessary. I would have expected just Ord a, Fractional a, and this combination is the same as RealFrac a.
Some of the helper functions could be usefully made into top-level functions. For example, the implementation of hitQuad finds the plane containing a triangle of points. But this operation is generally useful:

-- The plane containing three points, with a normal chosen so that
-- the points are clockwise when looking in the normal direction.
planeContaining :: (Num a) =&gt; (V3 a, V3 a, V3 a) -&gt; Plane a
planeContaining (a, b, c) = Plane a (cross (b - a) (c - a))