Which OS should I downgrade to?
Considering your usage, the answer is fairly straightforward:


Xcode 9.0 to 9.2 require as a minimum macOS Sierra 10.12.6
Xcode 9.3 to 9.4.1 require macOS High Sierra 10.13.2


So you're basically limited in your options to these two versions of macOS.

You didn't specify the exact version of Xcode you have installed, but you may need to change that to go down to macOS Sierra. Also, you may need to consider your long-term usage, as Xcode 10 (still in beta) requires macOS High Sierra 10.13.6 and it won't be long before a version will require macOS Mojave.

If you decide you want to keep the iMac, then your best bet would be to upgrade to an SSD (either replacing the optical drive, or the internal HD). I suppose another option would be to get an external SSD which supports a Thunderbolt 1 connection and then install macOS on that to use as your main boot disk. While a little clumsy, it's still going to result in a better experience all around.