In what way is recurrent neural network state "hidden"?
The &quot;hidden&quot; layers and state in a neural network are those that do not have direct input or output with respect to the function being learned.
I don't think you need to read anything deep into the analogy. The term &quot;hidden&quot; was, according to Geoffrey Hinton, coined by him partly because he liked the mysterious sound of it (from hidden markov models). He freely admits that it is partly an awkward name since in the case of deep neural networks, the data in the hidden layers is not inaccessible or unknown.
I heard him state this in one of the lectures in a now closed Coursera course.
Sadly I am not able to find a current reference to this in his own words. I have found a reference to the same story in the book Backpropagation: Theory, Architectures, and Applications.