Finder freezes when downloading files in Safari
You don't have to restart your Mac. You can simply use Command-Option-Escape and Relaunch the Finder app. I have this same problem, but it only happens when I have a Finder window open to the Downloads folder at the same time I download a file in Safari.
If the Downloads folder isn't open, the file downloads and all is well. This problem started with OS X El Capitan on a Mac with a clean install of the OS and a disk that checks out with no issues under the Disk Utility.