Update to 10.14.1 on Mac Pro 5.1 results in black screen
I just had the same black screen issue on my 5,1 Mac Pro (also running a Sapphire RX 580). I tried putting back the original 5770, and after booting with that failed, I tried resetting the PRAM (power on and immediately hold Cmd-Opt-P-R until the machine restarts again) and that reset things and got the display working again.

It might have worked without swapping back to the older card first, I unfortunately didn’t try that, and I admit I hope not to find out. Thankfully it still was better than your OS reinstall, I was afraid it was headed that way.