Low voltage drop current limiter for charging supercaps with a MOSFET
You could use a device like the TI TPS25940A eFuse to improve the circuit. It has current ramp rate control for cap charging and programmable current limit.

This particular part has a minimum current limit of 0.6A it looks like. But you can probably find a similar part with lower current specs. It will be cheaper too. The eFuse voltage drop should be very small.

You can read the application note for a similar application that you are using here:
http://www.ti.com/lit/an/slva920/slva920.pdf

The app note above uses a boost converter to maintain a 5V output as the cap discharges. If your load can handle the voltage drop without the need for regulation. You could replace the boost converter with a shottkey diode or an ideal diode ("diode current switch") IC. You can find them here on digikey
https://www.digikey.com/products/en/integrated-circuits-ics/pmic-or-controllers-ideal-diodes/758?k=ideal%20diode

A diode is needed to replace the boost converter so the cap only supplies the load when the main voltage goes below the cap voltage. Choose the diode based on how much voltage drop is acceptable. But the caps may discharge very quickly to below your desired threshold. You can easily calculate how long you have. So a boost converter may be needed in your application too to prolong runtime. Otherwise you are wasting a very large portion of the energy stored in the caps.

I don't know what super caps you are using, but putting super caps in series is similar to putting batteries in series. If they are not perfectly matched, they can charge at different rates and discharge into each other and damage them. This will require balancing circuitry.

You can read about this kind of super cap "UPS" system and how the ESR can change in this Analog Devices appnote.
https://www.analog.com/en/technical-articles/supercap-backup-circuit-provides-reliable-uninterrupted-power.html

There are lot's more options to achieve this goal and it all depend on your exact requirements and application. Hope this helps.