Why did AMD's putting an SSD on a GPU board reduce latency so much? AMD have recently been in the news with an interesting Radeon Pro board with a couple of M2 SSDs on it.

As some of the more detailed stories (here or here, for example) note, the benefits may principally accrue not from high bandwidth (the M2's are just on 4 PCIe lanes each, so the board's own 16 lane connector ought to have more) but from low latency.  This story includes the claim "this results in 10x lower memory access latency".

My question is basically: why should PCIe-connected SSDs on the GPU board have significantly less latency than the GPU accessing main system RAM or storage devices on the system PCIe bus?  What is it about the main system that "gets in the way" and means the on-board SSD's can be so much faster to access?
Answering from a somewhat layperson's perspective (I'm not a hardware expert, but I have built my own computer), I'll analogize, using the situation of getting some ingredients for cooking food.

Using "main system" RAM/storage is like you looking for an ingredient, not finding it in your cupboards, and, say, walking over to your neighbor and asking if they have any.  It turns out they do; they bring you it, you thank them and return to your house and use it.

Using "on board" RAM/storage is like you looking for an ingredient, finding it in your cupboard, and using it.

With on board RAM/storage, the GPU just has to do less work to obtain the information it needs.  The information is also simply closer, both physically and when considering the amount of hardware and software the information needs to go through.  With the on board RAM/storage, the GPU needs only look up where the information is, and then retrieve that information from RAM/storage, which then travels through maybe a couple inches of wire to the GPU's processing units.

With main system RAM/storage, there is a larger distance of wire for the signal to travel through which, although it may be only a minuscule fraction of a second, can still add up over thousands of calls to retrieve information.  Also, there are more potential bottlenecks, with all the different pieces that are interconnected.

Lastly, with on board RAM/storage, AMD has the opportunity to optimize the system, since they're controlling precisely hardware is being used.  This is just like how Apple tailors Mac OS to its Mac computers.