Determine Optimal CNN Complexity: Achieve ~100% Training Accuracy, Then Add Regularization? I was just wondering whether it is a decent idea to first try to make a CNN complex enough to achieve close to 100% accuracy on the training dataset, and then slowly add regularization, augmentation etc to make sure that the network doesn't overfit completely but instead is able to generalize better.
Yes, that is in my opinion the best methodology for approaching Deep Learning. This isn't supported by just me but by many researchers in the field. (I can't cite it but I'm pretty sure even Andrew Ng mentioned it in a lesson in the Deep Learning Specialization in coursera.)

The first step is to make sure you have a model with a sufficient capacity to be able to learn all the relationships (useful and not) in the training set.

Then you start applying regularization techniques (dropout, augmentation, early stopping, etc.), to prevent overfitting. During this step your training accuracy is expected to take a drop (or at least take longer to reach 100%), but the accuracy on the validation set should increase.

This methodology, however, is only applicable in the field of deep learning, because of the strong regularization techniques available.