Is this overfitting avoidable?
an epoch takes 3.5 days  


First of all, use colab to iterate quickly, it offers unlimited 12 hours of free GPU.  


  to retrain it on Imagenet.   


That said, we could use complex models without being afraid of overfitting due to large training size.


  will continued training increases the accuracy of the training set at the expense of the validation set  


In many cases that's the case.


  The idea was to mimic the massive feedback projections in the brain.  


My suggestion is to read the third part of the deeplearning book which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.


  Part III is the most important for a researcher—someone who wants to understand
  the breadth of perspectives that have been brought to the field of deep
  learning, and push the field forward towards true artificial intelligence.