Is my algorithm about blob tracking for vehicle following correct?
You might want to consider tracking using MeanShift &amp; Cam Shift:

http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_video/py_meanshift/py_meanshift.html#meanshift