Use regulator when input voltage is above threshold other wise use direct input voltage
You can do even better if you get a buck-boost regulator like the one below: -



Basically it produces 3.3 volts from an input voltage of 2.5 volts to 4.2 volts.

Note - irrespective of the regulator technique you still must use a Li-ion battery protection circuit to prevent over discharge of the cell.