Battery Considerations For RP2040 and NEO-M9N GPS Logger
All of the information I read from the Sparkfun documentation  suggests the following about current consumption:

A larger amount of current (100 mA) is needed for satellite acquisition
For tracking only 31 mA is needed (for continuous mode) or 20 mA (in power save mode)

So a lot will depend on how often and how long you'll be in acquisition mode.
Sleeping the RP2040 will help, but putting the UBLOX into power save mode will help more. Depending on how often you want to get a GPS reading it might be better to just shut the entire UBLOX module down and pay the acquisition cost again. You can use a P-MOSFET to control power to the UBLOX module from the RP2040 if you can't control it any other way.

I have come across some people talking about voltage converters and such to decrease the power waste, but I am struggling to find a good primer on how to think about power consumption and supply.

A switching power converter is unlikely to help in this case. They have their own inefficiencies and a current penalty just to keep them operating. Your battery voltage is already close to your needs. Things might be different if your design needed to use, for instance, a 12V battery but you needed a 3.3V rail for your MCU.