How can I "ignore" a DC power source?
Of course, the idea can be implemented on the circuit to select the correct supply automatically. There's a manual (yet easier) way though.
There's a jumper (JP1) on the programmer:

This jumper basically selects the VCC (5V or 3V3) of the target. If you remove it then there'll be no VCC at the ISP connector.