Can a battery powered circuit experience input voltage overshoot on turn-on?
Yes, it's possible. The length of the leads is not so important as the area of the loop encircled by the leads. If they're a twisted pair it's a lot better than if you have two parallel wires, and much, much better than if one of the leads goes walkabout your chassis to a switch or something. 

I suggest hanging a good old aluminum electrolytic across the input.. lots of capacitance and ESR to damp any ringing. Using tantalums 'bare' with a very low impedance battery source is asking for pyrotechnics of sorts, as any reputable maker should warn you. If it's a CR2032 etc. then, no problem.