BUG: format small floats correctly (GH9764)
closes #9764 

In order to display a number from range (0,1) in decimal format with N significant digits (and not in scientific format), the number needs to be greater than or equal to 1e(-N+1), not 1e-N. 

Did not test for range of display.precision, as this does not seem to current practice.

Did not include tests as they would imply an output format which might change in the future.
