How can I track detected objects from frame to frame
Object detection is relatively a heavy task as you've notice. Detecting the object (in your case human face) in every and each frame would be cumbersome and computationally immense. Therefore, you need to employ an object tracking technique. There are various tracking algorithms, of which, KLT and mean-shift are the two popular ones.
KLT works based on object features (corner,..) but mean-shift (usually) is implemented based on intensity of image.
Personally, I recommend mean-shift for your application, because I think face color would be more unique in a image sequence than general features (corner,edge,..) of the face.
The only thing you might need to add, is verification of the tracker by your object detection algorithm, every like 50 frames to ensure sound functionality of the tracker.

For implementations see:

http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_meanshift/py_meanshift.html



http://www.pyimagesearch.com/2015/09/14/ball-tracking-with-opencv/