How does a Fenestron remove or reduce the tip vortex losses in a helicopter?
There are basically two effects here:


One is the ducted fan effect. Ducted fans are more efficient than open fans, specifically creating smaller vortices at the blade tips.
The other is the fact that the tail rotor of a helicopter is not situated in an area with almost linear airflow along the rotor axis. It is also submitted to:


A steady flow due to helicopter forward motion. This is coming from the side of the rotor, so affects opposite rotor blades differently.
A buffeting flow from the h/c main rotor. This is an unsteady flow, also coming laterally from the point of view of the tail rotor (actually, the top).



Fenestron tail-rotor ducts can be considered a fairing that reduces airflow across the rotor. The end result is that the rotor blades are in a much smoother flow that helps reduce unwanted turbulence. This is one of the reasons such rotors are quieter than regular open tail-rotors (the other being the fairing material absorbing some of the noise).