Finder freezes for several minutes when opening folder for first time after restart
After further inspection, I noticed that when I first started Finder a process in Activity Monitor called InterCheck was taking up around 190% of my CPU (not sure how that's possible, but anyways). 

A Google search revealed that that process is from Sophos Anti-Virus. It turns out that when I start Finder it is trying to live scan my entire disk, which is taking a really long since I had scan "inside archives and compressed files" checked. By unchecking this option, Finder loads much quicker and InterCheck spikes CPU usage briefly on start up and then goes back down to a reasonable level (which according to Sophos forums, is the normal behavior).