Should I worry about ground loop in power supply circuit for 3G modem or WiFi?
Small difference in ground potential not really a big issue in 'digital' electronic. 

For audio, it the 'ground loop' produce 10mV noise, it will be treated as analogue signal and can be heard and reduce SN ratio. Same for precision/small signal instrumentation. 

Digital signal is treated as hi or lo, once passed a threshold value and hence tens of mV difference, as above, would have no effect.  

One specific about cellular, it use TDM and have highly pulsating current drawn profile. Jump from 2A peak then down to says, 50 to 100mA. Use thick positive and negative wire for the phone module. Check if your USB style power supply work well under this. If not, use bigger power supply to be safe. Always use thicker wire. Some USB cable inside wire is really thin and one cannot tell easily without cutting it. My portable hard disk has issue (as it draws 2.5 watts) with some USB cable from shop. Original 'good cable' come with the disk are fine. 

If I do this, I feel better with bigger size power supply and self wire cable (screw terminal, etc). More peace of mind as I see what cable I use. (although your diagram may work well, if the cable is good quality, etc.)

Hope it helps