Skin Detection Classifier
I suggest to give transfer learning a try for two reasons:


Your dataset is relatively small
Skin, faces and similar human body features have been successfully classified or segmented by existing neural networks


Specifically, you might want to have a look at Skin Lesion Analysis Toward Melanoma Detection. While the tasks were slightly different it did include a segmentation task of Melanomas. Here is an example of what they did:



And this is a snapshot of how some of the models performed:


Source: both images are from the paper linked above

It might be close enough to your task to re-use their networks or some part of it. Once you have found a transfer network, you can vary the degree to which you replace or retrain it`s layers to see what works. Given your relatively small dataset I would not retrain too much but start with the final layers only. 

However, depending on how exactly your dataset looks like, there might be other tasks (e.g. from security applications where one needs to detect, classify or mask an object (e.g. a person or a face)) which resemble yours even better. Therefore, before digging into a specific model and apply transfer learning I'd do some more research.