Final layer of neural network responsible for overfitting
Are you balancing your training dataset 50/50? It seems to me that you are making a lot of changes to account for 80/20 balance when it sounds like you have a deep enough dataset to balance your training to be 50/50 and still have a reasonable size for your training set. How about keeping it to a smaller network - as you and some notes above suggest - and with a 50/50 set on a smaller training sample?