Need a fast ray-box intersection that handles if a ray is parallel and in line with a plane of the box
Here is how I found this can be solved.

(The below also happens at the start, just fyi)

  float tmin = std::numeric_limits&lt;float&gt;::lowest();
  float tmax = std::numeric_limits&lt;float&gt;::max();


Add an additional check:

if(rayDir.x == 0.0f)
{
  if (rayOrigin.x() &lt; box.min.x() || rayOrigin.x() &gt; box.max.x())
  {     
    return false;
  }
}
else 
{
  tx1 = (box.min.x - rayOrigin.x) * invRay.x;
  tx2 = (box.max.x - rayOrigin.x) * invRay.x;

  tmin = std::max(tmin, std::min(tx1, tx2));
  tmax = std::min(tmax, std::max(tx1, tx2));
}


This way the tx1/tx2 values never get calculated if the ray's direction vector is zero in that direction, aka if it's parallel to that plane.

It's not particularly compact or pretty, but it does neatly solve the issue.