Is it safe to swap the neutral wire between the supply and load side?
You already said they come into the box in pairs/cables.   And you said you marked the supply hot.  

The supply neutral should be in the same pair/cable as the supply hot.  

So there you go.  

Generally switches are direction-agnostic, unless the labeling or instructions says otherwise, as would surely be the case for a smart switch.