Recently, skin detection has been employed in multifarious applications of computer vision including face detection, gesture recognition, etc. This is mainly due to the appealing characteristics of skin color and its potency to segment objects. However, there are certain challenges involved in utilizing human complexion as a feature to detect faces, and they have led to the inefficiency of many methods. In order to counteract these factors, in this paper, a skin segmentation method which exploits a multi step diffusion algorithm to detect skin regions is presented. The method starts with conservative extraction of skin seeds in each frame which is accomplished by using fusion of ternary-based human motion detection, modified Bayesian classifier, and a feedback mechanism. Subsequently, these candidate skin pixels are utilized in a 2-stage diffusion scheme to detect other skin pixels. Both quantitative and qualitative results demonstrate the effectiveness of the proposed system in comparison with other works.