What are the common pitfalls that we could face when training neural networks?
I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.

Other issue which happens a lot is overfitting on the training data hence network behaves badly on test set which can be solved using regression. So you have to make sure that network is generalised enough.

If the network is trying to have a classification then if the outputs are mutually inclusive or exclusive, depending upon that you define your loss function which will be categorical cross entropy for mutually inclusive output and binary cross entropy for mutually exclusive.

Other thing which I can think is about all zero initialization. Initializing the network weights as zero can lead to same gradient computation and due to which network does not learn most of the times.