Is there any evidence that the downwash vortices impact the performance of a wing?
You've correctly called them downwash vortices. They are caused by the air behind the wing—over all of its span—moving downward, while the air outside the span does not, which creates a pair of vortices that move downward at half the speed of the downwash (and somewhat slower later as there is more air below them in their way).

The air moves down behind the wing because the lift is an upward force acting on the wing, so the corresponding reaction force is downward and acts on the air, which, being free to move, accelerates downward.

This acceleration of air, however, requires energy (because its kinetic energy increases), and because energy is conserved, work has to be done to provide it. This work is done by induced drag.

There are several other ways to show, from basic laws of physics, that induced drag is a necessary consequence of producing aerodynamic lift that is proportional to the amount of lift and inversely proportional to wing span and dynamic pressure  (that is density and square of speed). If the span-wise distribution of lift is suboptimal, it will increase the induced drag, but there is a known minimum for optimal, elliptic, distribution, and it is well known it can't be reduced below that.