Why use SIMD if we have GPGPU?
Nothing is free.  GPGPUs are SIMD.  The SIMD instructions on GPGPUs tend to be wider than the SIMD instructions on CPUs.  GPGPUs tend to be fine-grained multi-threaded (and have many more hardware contexts than CPUs).  GPGPUs are optimized for streaming.  They tend to devote a greater percentage of area to floating point units, a lower percentage of area to cache, and a lower percentage of area to integer performance.

Let's do a comparison.  Intel's core i7-5960x has 8 cores, each with 4-wide (double precision) SIMD, running at 3 GHz (3.5GHz turbo), a 20M L3 cache, consumes 356mm^2 and 140W and costs \$1000.  So 8*4*3*4 = 384 double precision GFlops.  (The extra 4x is because you can do two fused-multiply-adds per vector lane per cycle.)  It can do 768 single precision GFlops.  That's about 1.08 DP GFlops/mm^2 and  2.75 DP GFlops/Watt.  There's also about 57.5 KB/mm^2 of on-chip cache.

NVidia's GeForce GTX Titan Black has 15 SMXs, each with 32-wide double precision SIMD, running at 890MHz (980MHz turbo), 3.5M of L1+L2 cache, consumes 561mm^2, 250W and costs \$1000.  So 15*32*.89*4 = 1709 double precision GFlops.  (Same 4x from two fused-multiply-adds per vector lane per cycle.)  It can do 5126 single precision GFlops.  That's about 3.05 DP GFlops/mm^2 and 6.8 DP GFlops/Watt.  So 3x the DP floating point density per unit area and 3x the DP floating point power efficiency.  And the tradeoff? 6.4 KB/mm^2 of on-chip cache. About 9x less dense than the CPU.

So the main difference is that the GPU has chosen an area balance that strongly favors floating point (and especially single-precision floating point) over cache.  Even ignoring the fact that you need to copy stuff back and forth between the CPU and GPU to do I/O, how well the GPU is going to do compared to the CPU depends on the program you are running.

If you have a data parallel floating point program with very little control divergence (all the vector lanes are doing the same thing at the same time) and your program is streaming (can not benefit from caching), then the GPU is going to be about 3x more efficient per unit area or per Watt.  But if you have any significant amount of divergent control, non-data-parallel work to do, or could benefit from large read-many-times data structures, the CPU will probably do better.