What's the advantage of multi-gpu training in real? The decreasing speed of training loss is almost the same between one gpu and multi-gpu.

After averaging the gradients, the only benefit from multi-gpu is that the model seems to see more data in the same time.

But why average the gradients? 
Is it that the model is indeed feed with more data in the same time?
I see two main advantages of using multi-GPU instead of one as they distribute certain resources:


using large DNN models - some recent models occupy vast space in memory so they simply cannot fit regular GPU and using multiple GPU allow to distribute some parts of the model to different GPU instances.
speed-up DNN training is also a very positive effect of using multiple GPU but only if you have a high-speed connection among GPUs as NVIDIA came with their NVLink