Epsilon is the separation at which GJK considers the objects to be in contact or penetrating and then runs EPA. If this is too small, then the renormalization of the separating vector can lead to arbitrarily wrong normals for almost-touching objects.