How large is the address decoder to select an address in RAM? Is the address decoder just a bunch of multiplexers?
For most RAM, the mux turns into a column select for a large grid, and an entire line of data is read out or written at once. The grid structure gives you the efficiency you need.

http://docencia.ac.upc.edu/master/DTM/docs/03-Memory%20Structures.pdf is a good PDF on the subject. It also shows a "predecode" structure which makes the decoder less complex than it would be otherwise.