headset plugged in via Y-adapter and extension cable stopped working
Based on this statement in your question:


  The (bad) solution was plugging in my headset directly to the build in
  3.5mm port.


This tells me you didn't have good results going directly from headset to iMac audio plug.


  Directly plugged in headset sound far worse then over the previous
  setup. According to my friends, which here me everyday via TeamSpeak.


Which is confirmed by this statement.


  Testing the long extension cable and the Y-adapter I came to the conclusion both work fine. It is very unlikely that on both cables the mic channel broke at the same time.


This is absolutely correct.  Cables will last a long time if you don't abuse them.  You would know if your cable was giving you trouble and you didn't indicate it here.

Unfortunately, your iMac audio port may be bad.

Your audio cable plugging into the back of your iMac looks something similar to the image below.  Each one of the contacts is a "spring loaded" contact that pushes against the plug.   In effect, it is a mechanical component and prone to failure at some point.   Alternatively, you may have a bad solder joint on the audio jack's connection to the logic board.  It's impossible to know unless you have it diagnosed by a technician.



After doing some searches around the Internet, it seems the the audio port for the iMac 27 (2013) is still separate from the logic board, but for whatever reason it's not availabe in the aftermarket yet.

Looking at the the back of the iMac, we see it's to the left of the SD card reader.  

However, looking at the logic board, it's not physically attached:



(images courtesy of ifixit.com)

This gives me hope that you can have it replaced by Apple or an Authorized Service Center.  Hopefully, soon, it will be available for order on the secondary market because this is a very easy DIY project.

The other options is that you could switch over to USB and/or Bluetooth headsets.  Going USB would probably be the easiest way to go.

You mentioned that you have it connected to a stereo.  I am personally using Behringer UCA202 Audio Interface to hook my iMac up to my Bose Wave Radio.