Why is the Craft without filter not working correctly and how to debug?
Try using |without based on arrays of user IDs, not entire UserModel objects. Behind the scenes, it's just using PHP's in_array method, which works much better on simple primitives like ints, strings, etc. instead of complex objects like Models (that has magic methods, other objects, primitives, etc.).  in_array probably doesn't know how deep of a comparison to make on those objects.