A data set to distinguish man from woman? Is there a set and/or a model that is available freely in the Internet that can tell man from woman based on input image?
You can use deepdetect which provides image classification functionality over API.

There is a gender classification model available on the models page. I have used it without problems before.