Inter-PCB Ground Loops
What you describe does create a ground loop, but it's unlikely to cause problems because the loop is so small. The ground loops that cause issues typically involve loops measured in meters, not centimeters. That said, the definition of "small" depends on the frequencies used in your circuit, so if you're working with very high frequencies (maybe, over 100 MHz), or your mezzanine board is especially large, you may need to reconsider this.

Just like how when you're laying out a circuit board, you normally want to lay out separate ground regions in a star topology, with a single connection point; if you could arrange it, you'd also want to move your two mezzanine connectors to the same end of your board to create a single connection area for your ground connections. 

But, if you simply can't change the locations of the connectors, it's almost certainly better to have ground pins in both connectors. The reason is if you remove the ground pins from one of the connectors, then the return currents for any ground-referenced signals on the other pins have to travel around a larger loop back to their origins through the other connector (the one that still has ground pins), and this is likely to cause emissions or signal integrity problems.