3.3V Power Failover Supply with USB Li-Ion Charger
Your circuit looks OK.

The 73831 comes in -2 -3 4 &amp; -5 versions. What version do you have? 

Original (not your circuit) is a mess - one should never have a diode AFTER a regulator - quite apart from the too low Vout and non swap-over.



The following addresses charge LED issue if there is a problem BUT I think that lack of battery is an illegal condition - see next paragraph. Does any of the following change the charge indication. What is voltage at BATT output in each case? 


Diode leakage SHOULD be very low at room temperature.   
Remove battery AND D2 and see what happens.  
Then remove C5


HOWEVER:

A LiIon battery stops charging when the voltage is held at some fixed threshold (usually ~= 4.2V) and charge current falls to below a preset % of initial current at this voltage. Battery chemistry causes this current to decrease with time. A capacitor does not behave like  a battery does. I'd expect it to show end of charge almost immediately, but as you are doing something that the design does not specify as legal, anything MIGHT happen.

Does it stop charging a battery when battery is installed?
 What is the end battery voltage?
 What is the battery mAh capacity?