How much tolerance is there for a frameless glass shower door?
The maker of the door will give you the max and minimum opening it will fit in.
Off the shelf shower doors, so they will fit varying sizes of showers will make up the difference by the lap/overhang of the glass on the hinge side of the door. This will only allow the door to open out, which is code. If the door is custom made, it would be possible to have it swing in and out.
Typically the biggest concern is the plumb of the walls and level of the curb. The caulk that seals the glass can only fill up so big a gap.