How large of a value should a weight have in a neural network?
One of the problems that can occur when training a neural network is known as the exploding gradient problem. A poorly initialised network could lead to a large increase in the norm of the gradient during training. These larger values will basically run the weights out of the number precision of the computer, resulting in NaN values.

This post gives more information on the exploding gradient problem and how to solve it. A related post discusses different initialization strategies.