Singular Value in Machine Learning
If the singular values of each layer are smaller than 1, then you can have vanishing gradients.  If the singular values are larger than 1, you can have exploding gradients.  Both cause problems for training.

Why?  As a heuristic, imagine that all the singular values for each layer were 2, and you have $n$ layers.  Then the singular values for the entire network would be $2^n$.  So, if the singular values of the layers are larger than 1, the singular values for the entire network can grow exponentially in the number of layers.  Similarly if they're less than 1, they can vanish exponentially fast.