Accuracy changes very little when modifying parameters in a Keras model
When you can't improve the model any more, improve the data.


are there any unisex names in your dataframe? Those will obviously hurt your accuracy. Either toss them out for now or do multi label binary classification.
Visualize your data: are the labels clearly separated? Which are the top losses?
I just saw that the data has the popularity of a name. I think it would make sense to try adding this popularity as a weight of each datapoint when training the model. Intuitively, you want your model to understand what does it mean for a name to be feminine. And "Anna" (score=2604) is much more representative of this meaning than "Jewel" (score=5), don't you think?


PS: I just saw that "Joe" is included in the list of feminine names.