Cas Latency vs Cpu to Memory Access Time
Between the CPU and memory in a CPU is 3 layers of cache: L1, L2 and L3. You've got to get all 3 of those caches to "miss" before you'll ever touch memory. Even after you touch memory, that data has to then feed back down the cache hierarchy before populating L1 and the registers to actually use the data.

It might seem wasteful, but the advantage of this is that if you try and keep your data in L1, you can operate on it much faster than any sort of memory. This is where a lot of software performance improvements can come from if you understand the hardware underneath.