OS X takes long to boot after starting rEFInd once
Turns out resetting PRAM and NVRAM via pressing ⌘+⌥+P+R at boot (before the gray screen appears; hold until computer restarts and the startup chime has been heard twice) fixes the problem.

I have since successfully booted both OSX and Ubuntu via rEFInd without any excessive waiting.