Can a logged in user restore its ~/Library folder from a Time Machine backup made with a previous version of the OS?
Have you done anything on the Sierra install that you wouldn't want overwritten? I've never tried restoring between different versions myself, but my guess is that it wouldn't be an issue—it'd just upgrade everything it needed to when you next logged in.

As for doing it while you are logged in—I wouldn't try it. I would either log in under a different administrator account, if you have one (but this might result in permissions problems), or log out, then borrow someone else's machine and log in over SSH. That way you won't have background processes writing stuff to your Library folder as you're trying to copy it.



This is for Time Machine. It may not restore everything—see below!


Borrow a friend's Mac (sorry, but Windows machines won't work) and open Terminal.


If your backup drive is locked:


Connect your backup drive and unlock it if necessary.
Open Terminal.
See below, starting with the ssh command, and replace ~/Library with /Volumes/disk/Backups.backupdb/machine-name/Latest/Macintosh\ HD/Users/user/Library where disk is the name of the backup disk, machine-name is the name of the old machine, and user is your username, as in the SSH command. Any spaces will have to be preceded with backslashes, as with "Macintosh\ HD". You can hit Tab while typing a path segment and it will autocomplete it for you, including escaping things as necessary (the backslash thing). Stop before the chmod command, as you didn't do anything to your friend's Library folder and don't need to undo it. (Plus, if for some reason they wanted something in their Library folder to be read-only, they might get mad at you!)


If your backup drive is not locked (hopefully it is, but just in case):


Do the same thing as above, except connect the backup drive to the new machine (without logging in!) and remove the user@machine.local: bit from the transfer. Also do the transfer while still logged in to the remote machine. You can then also use ~ in place of /Users/user (but don't replace this with /Volumes/...). The above method works just fine, but this one is faster.




An important note!

I'm not completely sure about this, but Time Machine may not back up everything in ~/Library. If you find something missing and still have the old machine around, I would use that to transfer the files:


Boot up both machines and connect them to the same network. Do not log in on the new machine, or log out if you are logged in.
Log into the old machine, and open Terminal.
Type chmod -R -w ~/Library. This will prevent the old system from modifying the folder as you're copying it. From here on out, until you undo this with chmod -R +w ~/Library, preferences will not be saved. Some other things might not be saved either; it's just a good idea not to do any work on either machine while this is going on.
Type ssh user@machine.local, where user is your username on the new machine (the name of your home folder, not your full name), machine is your new machine name, and local is the TLD of your local network. It may be local (mine is), but may also be something else. It might say something to the effect of "authenticity cannot be verified", but if you're on a home network you probably don't have to worry about this. You'll need to enter your login password (for the new machine) when it asks you.


You are now logged in to the remote machine. Type mv ~/Library ~/.Trash/Library to trash the folder you're about to replace.


If you don't have enough disk space to keep the old version around, you can use rm -rv ~/Library instead. Be very careful with rm, however! It deletes files immediately, without asking for confirmation. I wouldn't recommend doing this unless absolutely necessary, as then you won't be able to restore if there's a problem.

Type ⌃D to log out of the remote system. If that doesn't work (although it should), use exit.

Type rsync -rlptgDEHP --stats ~/Library user@machine.local:/Users/user/Library, substituting as above. This is the actual copy; it will most likely take a very long time. If you need to stop, just hit ⌃C to cancel the transfer. When you pick up again, it will have to go through the "building file list" process all over again but the actual sending process should pick up where you left off.


You can add z to the -rlptgDEHP bit if you have a slow local network (which is kind of unlikely, but...). This will make rsync compress everything before sending it across the network, although it'll take time to compress it before sending it and decompress it at the other end.

After the transfer is complete and you've tested it to make sure nothing broke, type chmod -R +w ~/Library (on the local machine). This will make it writable again.


Edit: You should do this on the new machine as well, so you'll be able to save preferences and things there.





A small note about security (this shouldn't really matter at home, but it might elsewhere):

Rsync uses a remote-shell program for the transfer; if the remote-shell program isn't secure, I doubt the transfer would be. It should use SSH, which is secure, but according to the manual, it can be configured to use something else:


  For remote transfers, a modern rsync uses ssh for its communications, but it may have been configured to use a different remote shell by default, such as rsh or remsh.
  
  You can also specify any remote shell you like, either by using the -e command line option, or by setting the RSYNC_RSH environment variable.


So if you want to force it to use SSH, use RSYNC_RSH='ssh' rsync ...

This only matters for network connections; it doesn't matter with the "TM drive not locked" method, as SSH is definitely used there.