Speed-up animation render which uses a movie texture
Thanks to the comments from poor and m.ardito.  Using an image sequence instead of a video clip (in this case an mp4) is the best.  Loading each frame (a particular image) took less than 1 second vs 5 seconds with the video.

If you can't use an image sequence for some reason, reducing the video resolution was also helpful, but not near as much and I had to reduce it a lot more than I wanted.

Another performance tweek was to disable "Cache BVH" under the Performance section of the Render tab.  I'm not clear on how this works but it only slowed down my render (about 10 secs/frame).