For points outside the triangle it will return negative barycentric coordinates that should still sum to 1. However, if the triangle is degenerate and the point is not on the line/vertex, it will return (1,0,0) with no way to detect this condition.