Is it safe to use ESP8266 with 3.7V?
Also note that that battery is 3.7 V nominal voltage. Fully charged it
  will probably be 4.2 V. Way too much for the ESP8266. Conclusion: you
  need a voltage regulator.


Look for a buck boost regulator like this: -



Or this: -



Or this: -



Or this: -



Or this: -



Or try googling "low power buck boost 3.3 volts 150mA" yourself

EDIT SECTION

A really small device is this (3 mm x 3 mm package): -