Skin Detection From a Face Image
It'd be good idea to extract face segment first. In OpenCV there're predefined HaarCascades for face detection. Though, you can run into some problems using them. Photos really need to be well-exposed. They're also prone to false positives as hell.

https://realpython.com/face-recognition-with-python/

Slightly better, but still vanilla, there's dlib.
Worst problem with them is you'll only get bounding box. Background may creep in and mess your color analysis.

To get seminal-like scores you gotta go DNN. Here's some example.

I would propose a bit different approach. Take your data set. Extract faces using existing tool and label skin colour manually. Then train CNN. Just be careful, when you augment data set. Some operations might be damaging, like swapping channels. Color-based analysis is tough. You need to think of covering different exposure values, contrast, white-balance, saturation etc.