What limits the lower bound of DRAM CAS latency
What is the biggest bottleneck in this sequence 


My guess would be parasitics. The copper read line and the read ampilfier would look like a R/C low pass filter when reading from a DRAM cell.

And since there are limits on how wide the copper line and how thick isolations can be, the R/C time constant stayed roughly the same even when the silicon structures got smaller in newer processes. 

Trying to improve this time constant woud probably lower the data density too much, which is also an important parameter in DRAM design.