A data set to distinguish man from woman?
You can use deepdetect which provides image classification functionality over API.

There is a gender classification model available on the models page. I have used it without problems before.