/usr/bin/nice and Handbrake
Yes, you can run a GUI app with nice. For example, to run TextEdit with a niceness of 10:

nice /Applications/TextEdit.app/Contents/MacOS/TextEdit


It really depends on whether handbrake is using more CPU or more RAM. Xcode is a RAM hog, and nice won't help there, but if it's CPU, you're good.