How do I control a small dc motor with a mosfet?
After reading Daniel's comment, I realized that I might have the direction of the current to the gate backwards.  After changing it so the smaller battery pack positive was connected to the gate, and negative to the source, it all works great.