Adafruit Huzzah ESP8266 Battery Power issue
After tinkering more, I figured out that the Huzzah module is happy to take in up 4.5V via a 3 AAA battery cell so I guess it's ok up to 5V since my previous battery input was over 5V.

That's a little weird considering that in Adafruit's page, it says 


  ..to make it easier to power, we put a high-current-capable 3.3V
  voltage regulator on the board. It can take 3.4-6V in but you should
  stick to 4-6V since the ESP8288 has high current usage when WiFi is
  on.


Hope this may help someone else.