What exactly is a hidden state in an LSTM and RNN?
This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know.

Lets take this simple sequence first,

   X = [a,b,c,d,.......,y,z]
   Y = [b,c,d,e,.......,z,a]


Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details doesn't matter.

We can write this relationship in maths as 


  $f(x)\rightarrow y$


where x is an element of X and y is an element of Y and f() is our neural network.

After training, if given input x = a our neural network will give an output b because f() learned the mapping between the sequence X and Y.

Now instead of the above sequence try teach this sequence to the same neural network.

 X = [a,a,b,b,c,c,.........y,z,z]
 Y = [a,b,c,,...z,a,b,c......y,z]


More than likely this neural netwok will  not be able to learn the relationship between X and Y. This is because a simple neural network can't learn and understand the relationship between the previous and current characters.

Now we train the same sequence in RNN, in RNN we take two inputs one for our input and a previous hidden values and two outputs one for output and next hidden values


  $f(x,h)\rightarrow (y,h+t)$


Important: here  h+t represents next hidden value and not the arithematic addition with time step t.

We will execute some sequence of this RNN model, at the start hidden values are considerd as zeros

   x = a and h = 0
   f(x,h) = (a,next_hidden)
   prev_hidden = next_hidden

   x = a and h = prev_hidden
   f(x,h) = (b,next_hidden)
   prev_hidden = next_hidden

   x = b and h = prev_hidden
   f(x,h) = (c,next_hidden)
   prev_hidden = next_hidden

   and so on 


If we look at the above process we can see that we are taking previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.