Second monitor - Dell and iMac 27 5 Retina
Your iMac can support up to two displays at 4096 by 2160 pixels, or one display at 5120 by 2880 pixels.

In other words, to use the second display in addition to the built in display they both need to run at 4K. You can't run them both at 5K.

Your iMac's Thunderbolt digital video output also supports up to two Thunderbolt displays, but these must also be run at 4K.

You can view the full specs here.