What video resolutions does late-2015 iMac 5K support on external monitors?
You haven't detailed how you have actually connected the monitor and the iMac, but it sounds like the problem is the cabling/adapters. 

The monitor is from a time where you couldn't achieve the 2560x1600 resolution without using dual-link DVI. Note that this is different from just "DVI" as a non-dual link DVI cable will not allow you to transfer such a resolution.

You'll to use an adapter especially for this purpose. For example Apple's mini-DisplayPort to dual-link DVI adapter:

https://www.apple.com/dk/shop/product/MB571Z/A/mini-displayport-til-dual-link-dvi-mellemstik