Building a byte-addressable memory
...  even in the uncomplete state, it seems a little too large.


Nope, looks about right to me. I used to build motherboards for engineering workstations (overgrown PCs) in the 1980s, and this is exactly the sort of thing we had to do.

The right side of your drawing is the actual memory (note that one of your modules has a different size from the others). The left and center sections are what we call "byte steering" logic. This is what you need if your CPU isn't doing this for you internally.

You should see how much fun it is to interface a Motorola 680x0 CPU bus to an Intel (PC/AT) peripheral bus. This ended up being one of the first places we used an FPGA, because of the complexity.