Capacitor In parallel with Battery
If your objective is to provide a supply for fast transient loads the capacitors should be as close as possible to the loads. That's why pretty much every digital device has power supply bypass capacitors right at the power pins. 

The link you have there was looking at the leakage through the capacitors, it doesn't quite matter where there are. Leakage through the capacitors is a parasitic load regardless. Those leakage currents are tiny compared to the LEDs you're driving. If your goal is to extend battery life when the circuit is sitting, that problem can be solved with a switch.