How to make overfitting (powerful) model? According to my professor one of the first steps in modelling a NN is to use a powerful enough model.


  The first step is to create a model that is powerful enough to 
  achieve very high accuracies (very low loss) on the training data, at
  least when no regularisation is used.


What are some of the things (obviously apart of regulazing and adjusting learning rate), that I can do to make my model "powerful" enough, in other words to let it overfit on the training data?

Am I looking in the right direction with the following things?


Add extra layers
Make layers thicker (more neurons)
Here are some general hyperparameter adjustments to increase model capacity:


Architecture-related:


add additional layers
increase the number of hidden units per layer
remove dropout or decrease dropout rates

Optimizer related:


find the optimal learning rate
train for more epochs



Section 11.4.1 in the Deep Learning Book provides a good overview. The paper "Practical recommendations for gradient-based training of deep architectures" is another good source to understand the effect of different hyperparameters (though it is a bit older). 

Moreover, there are specific hyperparameters to increase model capacity for different network types, e.g. 


kernel size for CNNs, 
sequence length for LSTMs and
embedding dimension for embedding layers.