Connecting a Lithium battery directly to the GSM module
You need over-discharge protection, to prevent damage to the battery under low power or fault conditions.

Typically it is safe to discharge Li-ion cells to at least 3.3V, but you should check for your specific chemistry.

But no, you should not need any kind of buck or boost converter.