How to initialize the first h in an RNN? Take a Vanilla RNN represented by the function $h_t = f(h_{t-1}, x_t)$, how do you determine $h_0$? 

Edit: This answer over on the stats page has helped.
There are a few different ways. One way is to always give it a zero vector, or some other constant that means "new sequence". Another way is to train the initial state vector just like you train any other weight: initialize it randomly, then apply gradient descent during training. The former is easier and faster, but the latter can sometimes make the RNN more effective.