What is the purpose of multiple neurons in a hidden layer?
To explain using the sample neural network you have provided:


Purpose of the multiple inputs: Each input represents a feature of the input dataset.
Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).


If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.



Edited to answer additional questions:


The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.