Why do I not have sufficient permissions to restore my Library folder?
Library like many of the folders that exist in the root of your home folder have special permissions. I have better luck selecting all inside the folder, trashing them, then restoring all in place. 

Working inside the folder is less error prone than trying to change the folder itself.