Processors: E5-4620 v2 vs E5-2620 v3
The way to gauge hardware performance is to be take a look at your current workload. So I would begin by quantifying that:


Take a look at Brent Ozar's perf tuning stuff
Identify your bottlenecks (are you seeing bad parallelization, CPU chugging away without finishing, memory, storage)
Determine what differences there will be between your new and old system.


You're right that SQL Server rarely bottlenecks at the CPU, but it's not impossible and you can tell whether you're seeing a CPU bottleneck by looking at some perf counters.

Then you'll know whether or not you'll see a performance improvement simply by updating the CPU to a faster one or one with more cores. If your SQL isn't written to take advantage of parallel processing, it won't matter if you throw more cores at it.

Additionally, there's more things I would want to consider in buying a system:


Upgradability - SQL Server will rapidly test the limits of your hardware if you let it, so try to anticipate that
Affordability - SQL Server licensing in 2014 is core-based, so this should be a consideration.


This is in addition to other IT concerns you'd want to keep in mind like maintainability, support contracts, and your space/power requirements.