Adobe After Effects Frame-By-Frame Editing
Have you used the Roto Brush? It was designed for situations like these.

For those unfamiliar, there is a nice concise tutorial here on how to use the Roto Brush.