Is it safe to restore the ~/Library folder?
I always did that -- in fact, if you did NOT upgrade OS version I'd just copy the folder wholesale.  Restoring from a Time Machine backup would do something similar, and equally dumb -- I had problems with Mail once because time machine had passed a preference file across between OS versions, and the pref file conflicted with the new Mail [1], so Time Machine is no guarantee that all will be smooth.

So, if you are NOT also upgrading OS, just copy all across.  If you are also upgrading OS move the data in ~/Library (such as your emails), and the prefs for stuff that is not bundled with the OS (such as Mail or Safari), but I'd reset the preferences for those applications only once more from within the apps themselves.

[1] What is com.apple.mail.plist and why was it slowing Mail down?