2 loads not working together on the same battery
If the 100mA load can tolerate a slightly lower voltage, you can put a diode in series with the positive terminal, and then put a capacitor across the 100mA load. The capacitor will charge up, and it will help keep the voltage more constant when the other load kicks in.