Target Display iMac steals Thunderbolt when I unplug and won't give it back
A DisplayPort KVM would probably work. Have you tried using a DisplayPort cable instead of a thunderbolt one between the iMac and the Thunderbolt Display? It might be cheaper than a KVM which would guarantee you can switch inputs without the iMac taking control of the attached display.