How can the performance of a neural network vary considerably without changing any parameters? I am training a neural network with 1 sigmoid hidden layer and a linear output layer. The network simply approximates a cosine function. The weights are initiliazed according to Nguyen-Widrow initialization and the biases are initialized to 1. I am using MATLAB as a platform.

Running the network a number of times without changing any parameters, I am getting results (mean squared error) which range from 0.5 to 0.5*10^-6. I cannot understand how the results can even vary that much, I'd imagine there would at least be a narrower and more consistent window of errors.

What could be causing such a large variance?
In general, there is no guarantee that ANNs such as a multi-layer Perceptron network will converge to the global minimum squared error (MSE) solution. The final state of the network can be heavily dependent on how the network weights are initialized. Since most initialization schemes (including Nguyen-Widrow) use random numbers to generate the initial weights, it is quite possible that some initial states will result in convergence on local minima, whereas others will converge on the MSE solution.