Can I convert my 3.7 volt lithium ion batteries into a usb charging power bank?
You could use a boost converter such as the TI TPS61090. I think Adafruit sells a board that uses this chip for your particular application. (Ok. here , which also includes a charger). 



If you were building many of them and wanted to shave the cost, there are other less expensive chips available on the market, but the TI one is a good one and easy to get, with all the required information easily available.