When is a Neural networked considered to be overfitted?
A model is said to have overfitted when it performs awesomely on the training set (error's low), but when tested on test set, it fails badly. This might be due to bad sampling distribution over our training and tes divisions, so that model might not have seen this kind of data or due to using many irrelevant features in the samples than we were supposed to etc.

You cannot really say that the cat is dead until you open the box. So until you test, you would never know.


  I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting


This statement can be explained in general, not only for neural nets. Assume you know the relationship is linear for your data, but you got some error and you are not happy with this, then say you add an extra term say $ax^2$ to this equation and the model now has an extra parameter to tune and this makes our line into curve. For a neural net we basically start by random weights from a distribution and then we keep reducing them until we feel like our training error is cool.

Assuming that you are using a multi-layer perceptron (fully connected), pruning is one of the techniques to remove weights from the model which might not be so contributing still making sure your model still behaves well with lesser number of parameters. SVD on this weight matrix can be one of the useful techniques to do this. But how many weights to keep depends on how many singular values you keep for reconstructing the matrix. I heard that early stopping can reduce overfitting, but I had never really done this, so can't really give my opinions about this.

Hope this helps.