Use iMac 27" in Target Display Mode with Raspberry Pi
What I have found out is that TDM (Target Display Mode) on my mid 2010 27" iMac only works with the following resolutions:


720p (1280*720) or lower
Native (2560*1440)


Any device that tries to output something in between, gets a signal it has to change it's resolution to 1280*720. Many devices can automatically do this (Xbox 360, PS4), but some can not (PS3, Raspberry Pi). This means that, if you want to use a device that can't automatically change resolution, you have to set the right resolution BEFORE you connect it to the iMac with TDM.

For the Raspberry pi, this means you have to use the following two lines in your config.txt:

hdmi_group=1 // VGA 60Hz
hdmi_mode=4 // 720p


Read more about the config-file here.

Since the Raspberry Pi doesn't support 2560*1440 (it does support 2560*1600 though), you won't be able to use the iMacs beautiful 1440p-screen to its full potential, but at least it works!

If I ever find a solution to add a custom resolution, I'll update my answer.