Untrained Network Giving 80% Accuracy
If your class imbalance is roughly 4:1, then a network that always predicts the more frequent class would have an accuracy of 80%. (That's why accuracy alone is an insufficient metric: for binary classification, you should use ROC and PRC curves. There are also more holistic scalar metrics such as MCC)
Let's say your positive samples are the frequent class. Without training, a random network might always output positive (80% accuracy). In this case, you'd only have true positives and false positives. After training, the network might start to also predict negatives; then you'll start getting true and false positives in addition to true and false negatives, which could yield a lower accuracy.