Creating a image classification model
1.) You should not change your test set, even if you downsample the training set.
2.) At least in the plots you show there is no clear indication of overfitting. Did you see what happens if you train for longer epochs, and tried different learning rates ?
3.) If you have some overfitting, normally you would simplify the model, and not reduce the size of the dataset. Also you can think about data augmentation to prevent overfitting.
4.) You can compare your solution to the solutions available at kaggle.
5.) With such a black-box solution (vanilla CNN) you will not get super high results.  Though I do not know the upper bound. You can compare to kaggle for that..
6.) To further improve you have to integrate some prior knowledge (e.g. facial key points).