title: Why use softmax only in the output layer and not in hidden layers?
body: <p>Most examples of neural networks for classification tasks I've seen use the a softmax layer as output activation function. Normally, the other hidden units use a sigmoid, tanh, or ReLu function as activation function. Using the softmax function here would - as far as I know - work out mathematically too.</p>
  
  <ul>
  <li>What are the theoretical justifications for not using the softmax function as hidden layer activation functions?</li>
  <li>Are there any publications about this, something to quote?</li>
  </ul>