Keeping voltage above 3.7 Volt with a lipo
As has been mentioned in the comments, your issue is that the voltage is dropping due to load and the battery voltage dropping.

As your amplifier can take 3.7-5V, and you want it to operate off a battery, you're options are either to change the battery, or change the load.

You could do a new battery set up, using two cells in series to have a voltage of 5-8V ish (depending on cells chosen). then use a buck converter (LDO if you want to keep it as simple as possible) to keep voltage around 4V. This would be the easiest, and probably best method.

Or you could use current battery with a boost converter, to boost the voltage to a constant 4V or so. Assuming the battery can provide the power, and your boost converter is good enough, this will do the job cheaper than the first option. But this is more complicated and more to go wrong. Designing a good boost converter for a sensitive circuit (such as audio) is a bit tricky.

Or you could limit the load by reducing your volume limit, using lower power amplifier and speaker set up. Which you clearly don't want to do as you're asking this question.