Assume AC Phase has zero potential
Let's say you're in a commercial building in the USA fed by 480 and you have your own 120/240V transformer, which is not grounded.  Other than that, you are wired to modern code with all EMT conduit. The conduit, boxes and service panel enclosure are the ground, bonded to a ground rod driven into earth. 

All the neutrals go to the neutral bus, which is insulated from the panel chassis (ground).  However, there is a metal strap, bonding the neutral bus to ground.  This is the only place neutral is bonded to ground. There are no ground-neutral or ground-hot faults.  With me so far? 

Remove the bond.  Replace it with a 12V car battery.   What changes? 

You've introduced a 12VDC bias between neutral and ground.  It's still 0 VAC though, and the little neon 3-lamp tester will give 2 amber lights and no red. (neon doesn't work on DC).

OK, now replace it with a 3V buck-boost transformer.  Now you have a 3VAC bias between neutral and ground.  L1-neutral will be 240v, L1-ground will be 117V, L2-ground will be 123V.  They could just put a 3v tap on the supply transformer. 

It's all arbitrary.  We humans make it up. 

How far can we push it?  Bond the ground to L1.  Now L1-ground is 0v, neutral-ground is 120v and L2-ground is 240v. Illegal?  Perhaps.  But no less safe than Europe, which has 240V hot-ground.  Weird?  Not compared to 3-phase: They ground anywhere they please - in the middle (wye), on a corner, or halfway down a side (high leg/wild leg).  Or not at all (delta).  

So why do we "make it up" the way we make it up?  

A bunch of people in offices picking through thousands of fatality reports, trying to compromise with the clamoring of builders, manufacturers and electricians to make things cheaper for them, and come up with something easy.