LSTM / GRU weights during test time
The initial weights of h for GRU and h,c for LSTM are are often set to zeros, setting random weights is also an option. Also people have tried to learn the initial hidden states.
Since the hidden states are updated with every cell, if your sequences are long enough, it would not make a big difference how you initialize the hidden states.