Memory addressing rapidity
Why can't we simply put all of the RAM memory close physically to the CPU

Because that is not how geometry works. You cannot have everything close to the CPU, because the stuff that is close to the CPU is blocking the space close to the CPU, and thus other stuff has to be farther away.
Geometry works the same on the CPU as it does everywhere else: why don't all your neighbors live directly next door to you? Well, because your neighbors are taking up that space, so you simply can only have two direct neighbors. Everybody else has to live further away.
There are actually already 3-dimensional designs where the memory is stacked on top of the CPU, in order to be able to bring more memory closer to the CPU. But that only pushed the problem out, it does not solve it.
There is, however, a whole slew of other reasons, not just geometrical ones, why we have a memory hierarchy and not just a single flat memory. The main problems are that, in general, faster memory is also

more expensive,
more power hungry,
more complex, and
physically bigger (less dense).

That latter problem actually leads to an interesting optimization problem: as we said above, we can't have all the fast memory close to the CPU, because the space close to the CPU is limited. However, since slower memory is more dense, and thus takes up less physical space for the same memory capacity, you can actually put more slower memory close to the CPU than you can faster memory!
And even if you were to use only the fastest memory technology for all your memory, you quickly run into problems with the fundamental speed limit of the universe: the speed of light. As a general rule, our systems are designed in such a way that a single clock pulse must be able to propagate through the entire clock domain within a single clock cycle.
This means that no matter how close and how dense we pack our memory to the CPU, there comes a point where we cannot clock the memory with a high clock rate anymore. The further away we get, the lower the clock rate. But then it doesn't make sense to use the super-expensive, super-fast memory anymore, if we have to make it slower anyway. It makes more sense to use the cheaper, more dense, more energy-efficient memory.
And that is why the memory hierarchy exists: because the difference is not just speed, it is also cost, space, and energy. Here is an example of one possible memory hierarchy:

Registers (discrete flip-flops)
Level 0 Cache / µOP Cache (typically SRAM; per thread)
Level 1 Cache (typically SRAM; per core)
Level 2 Cache (typically SRAM; per core)
Level 3 Cache (typically SRAM; per die)
Level 4 Cache (typically SRAM; per module)
Local Main Memory (typically DRAM; per module)
Remote Main Memory (typically DRAM; &quot;not this module&quot;)
SSDs (Flash; per node)
HDDs (magnetic; per node)
SAN (mix of SSDs and HDDs; per system)
Tape Library (per system)
Offsite Cloud Storage

Obviously, not all of these levels exist in all systems. This is an extreme example of a multi-core, multi-CPU, multi-node NUMA cluster with a separate storage network – my laptop doesn't have #6, #8, #10, #11, and #12, for example.
#8 only exists in so-called &quot;NUMA&quot; systems (Non-Uniform Memory Architecture), i.e. systems where not all RAM is &quot;created equal&quot;. Typically, these are systems with a large number of CPUs, where it is not feasible to give all CPUs the same access to all of the RAM. Instead, the RAM is split into parts and typically one block of CPUs has direct access to one block of memory, and if it wants to access the rest of the memory, it typically has to talk to the CPU(s) that &quot;own(s)&quot; that block.
As a closing remark, I want to say that while the answers here have shown why your idea doesn't work, that does not mean that your idea is stupid! In fact, your idea is pretty much the Holy Grail of storage technology, and every couple of decades, a new technology comes along that tries to be fast, cheap, dense, energy-efficient, and ideally even non-volatile, at the same time, so that it can replace a large portion of that memory hierarchy.
However, as of today, none of those technologies have been able to live up to the hype, so to say: MRAM, FeRAM, T-RAM, Z-RAM, and many others. The latest entry being Intel's / Micron's 3D XPoint (better known under Intel's marketing brand Optane).