Why are SDRAM CAS latencies so high?
The CAS latency delay is from the column selector logic, there are muxes and column selector logic in most DRAMS, Because of this even though the data is ready after the sense amplifier and latch, the data must still travel through a selector mux to get to the data buss, and this takes time. Below is a simplified diagram of a small DRAM.

Source: https://en.wikipedia.org/wiki/Dynamic_random-access_memory (with edits)
DDR3 looks something like the picture below, the column selector logic adds some delay and also the read latch and mux. CAS lets the memory controller know which clock cycle the data will be available after the correct row and mux has been selected which takes time.

Source: https://en.bmstu.wiki/File:DDR3_Sheme.png (with edits)

Source: https://www.eenewseurope.com/news/understanding-ddr-sdram-timing-parameters/page/0/2
I've never designed a Column selector but what I do know about digital design and logic, I'd imagine that the CAS timing is related to the size memory array because that would increase the size and timing of the column decoders. Once could also design faster column decoders with increase size and cost.
EDIT:
The reason why the SRAM's latency is so fast is because it doesn't need a data selector mux, the memory goes straight to the data bus. There is no logic that propagates delays to the data bus and so as soon as the data is addressed it can be available on the bus (in the diagram below, you can see that in a typical SRAM the bus is tied right to the ram for minimal delay). In addition the memory cells for SRAMS are active transistors, not passive capacitors. So the memory cells in SRAMS can provide current to the sense amplifiers which would make the latency much lower (faster rise times). SRAMS are much more expensive in size and cost and also power, so they are only used where latency needs to be lowest.

Source: https://computationstructures.org/lectures/caches/caches.html