Li-ion power supply with a charger, a booster, and a pMOS high side switch
As mkeith pointed out, the problem was not in the schematics posted in the question but elsewhere: the battery supply VBAT is routed to a multiplexer and then directly to an analog input of the microcontroller where it went through the protection diode straight to 5V. This also explains why the +5V voltage was at 3.04V, which is the VBAT (4V) minus the voltage drop over the diode.