Why did AMD's putting an SSD on a GPU board reduce latency so much?
Answering from a somewhat layperson's perspective (I'm not a hardware expert, but I have built my own computer), I'll analogize, using the situation of getting some ingredients for cooking food.

Using "main system" RAM/storage is like you looking for an ingredient, not finding it in your cupboards, and, say, walking over to your neighbor and asking if they have any.  It turns out they do; they bring you it, you thank them and return to your house and use it.

Using "on board" RAM/storage is like you looking for an ingredient, finding it in your cupboard, and using it.

With on board RAM/storage, the GPU just has to do less work to obtain the information it needs.  The information is also simply closer, both physically and when considering the amount of hardware and software the information needs to go through.  With the on board RAM/storage, the GPU needs only look up where the information is, and then retrieve that information from RAM/storage, which then travels through maybe a couple inches of wire to the GPU's processing units.

With main system RAM/storage, there is a larger distance of wire for the signal to travel through which, although it may be only a minuscule fraction of a second, can still add up over thousands of calls to retrieve information.  Also, there are more potential bottlenecks, with all the different pieces that are interconnected.

Lastly, with on board RAM/storage, AMD has the opportunity to optimize the system, since they're controlling precisely hardware is being used.  This is just like how Apple tailors Mac OS to its Mac computers.