How to use a cap in parallel with power supply to help start a motor
What you propose seems reasonable to me. Indeed, you can use the capacitor in parallel with your battery to limit the inrush current for the battery. This is exactly why every IC on a PCB board should have a separate capacitor assigned to it. 

If you connect the capacitor in parallel with the battery, the battery will charge the capacitor until the cap is charged (= same voltage as battery). There won't flow any current after that. When you close the switch, the motor 'sees' the battery and capacitor in parallel. Because the capacitor has a lower initial resistance than the battery, most inrush current will be drawn from the capacitor. 

If the battery is removed after charging the capacitor (switch open), nothing happens, as the cap just stays at the charged voltage. (Ok, leakage currents will discharge it very slowly). 

I hope that makes things clear.