Switching "primary" imac in Target Display Mode
The FAQ for target display mode is pretty clear on the requirements. 

You get to bring in external video if it matches exactly the requirements. Display Port isn't the same as Thunderbolt, so perhaps that's the issue. You also need the master iMac to be able to drive all connected displays.

If you can't use the hardware directly, consider software to merge the screens such as AirDisplay or Screen Recycler.