Automatic/manual door lock with servo
So it's actually the worm gear that is causing your problem, since that can't be reverse driven by the cogs.

The simplest option, I think, would be to have the middle cog normally separated from the worm gear and only re-engage it when you need to drive the mechanism with the servo.  You could maybe use a second servo to move it into the right place, or maybe a solenoid, or something along those lines. Have it slide along a short track or slot with a spring to ensure it disengages when the solenoid is released.