A good skin detector that is capable of capturing skin tones under different conditions is important for human-machine interaction applications. In a general situation, skin detectors, such as skin probability maps or Gaussian mixture models, achieve acceptable skin segmentation results. However, the false positive rate increases significantly when the skin tones are in shadow or when skin-like background objects are under similar illumination. In this paper, we propose a novel skin feature learning algorithm based on stacked autoencoders, which are deep neural networks. To overcome the problems encountered in skin segmentation that are caused by different ethnicities and varying illumination conditions, the stacked autoencoders are utilized to learn more discriminative representations of the skin area in both the RGB color space and the HSV color space. Unlike traditional machine learning methods, instead of predicting each pixel individually, our algorithm utilizes blocks to learn the representations and detect the skin areas. The algorithm exploits the learning ability of deep neural networks to learn high-level representations of skin tones. Experiments on test images show that the proposed algorithm achieves acceptable results on several publicly available data sets. To reduce the difficulty of detecting skin pixels in these data sets, the ground truths of these data sets are commonly focused on foreground skin area detection. Our skin detector is also able to detect background areas, as shown in our experiments.