Retina monitors and old Mac Pro
I am not sure about the GT120. However, per this post, yes can get full 4K resolution (3840x2160) @ 30Hz with the Radeon 5770 and an ACTIVE Mini DisplayPort to HDMI connector that supports 1.1 and not just 1.2. However, this will get you the full resolution, and not the 300dpi screen doubled "Retina" (aka HiDPI mode) experience. If you download SwitchResX it will list possible options for you.

I can confirm this worked perfectly to get both 3840x2160 and 1920x1080 HiDPI (Retina) with a Mac Pro, Radeon 5770, an Acer B286HK, and a regular mini-DisplayPort to DisplayPort cable.