When using NOW in ModelSim in VHDL simulation, what determines the time resolution or its unit?
Modelsim SE User Manual  10.4c Chapter 6 VHDL Simulation, Usage Characteristics and and Requirements, Simulator Resolution Limit for VHDL:


  Simulator Resolution Limit for VHDL  
  
  The simulator internally represents time as a 64-bit integer in units equivalent to the smallest unit of simulation time, also known as the simulator resolution limit.  
  
  The default resolution limit is set to the value specified by the Resolution variable in the modelsim.ini file. You can view the current resolution by invoking the report command with the simulator state argument.  


So the simulation time units are whatever the resolution limit is.

Also see IEEE Std 1076-2008 5.7 String representations:


  
  For a value of a physical type, when forming the string representation for a TO_STRING operation, the abstract literal is a decimal literal that is an integer literal, there is no exponent, and there is a single SPACE character between the abstract literal and the unit name. If the physical type is TIME, the unit name is the simple name of the resolution limit (see 5.2.4.2); otherwise, the unit name is the simple name of the primary unit of the physical type. When forming the string representation for the WRITE procedure for type TIME, the physical literal is as described in 16.4.
  


The function now defined in library standard returns a value of type TIME, and anything that converts that to text will report the resolution limit.  

(The elective bit on Modelsim's part is whether or not to 'scale' units to the resolution limit, which isn't dictated by the VHDL standard).