Should I expect restoring Yosemite from a Time Machine backup to take forever?
First things first, make sure you backup before all of this:

When you restore from a backup you are not restoring to a previous version of OS X, you'll need to backup, erase the disk, reinstall with Yosemite and apply your backup. If you need help with this please comment this answer, I'll give all the detail needed 

Tiferrei