In language modeling, why do I have to init_hidden weights before every new epoch of training? (pytorch)
I have a question about the following code in pytorch language modeling:

print("Training and generating...")
    for epoch in range(1, config.num_epochs + 1): 
        total_loss = 0.0
        model.train()  
        hidden = model.init_hidden(config.batch_size)  

        for ibatch, i in enumerate(range(0, train_len - 1, seq_len)):
            data, targets = get_batch(train_data, i, seq_len)          
            hidden = repackage_hidden(hidden)
            model.zero_grad()

            output, hidden = model(data, hidden)
            loss = criterion(output.view(-1, config.vocab_size), targets)
            loss.backward()  


Please check line 5.

And the init_hidden function is as follows:

def init_hidden(self, bsz):
    weight = next(self.parameters()).data
    if self.rnn_type == 'LSTM':  # lstm：(h0, c0)
        return (Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()),
                Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()))
    else:  # gru &amp; rnn：h0
        return Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_())


My question is:

Why do we need to init_hidden every epoch? Shouldn't it be that the model inherit the hidden parameters from last epoch and continue training on them.

The hidden state stores the internal state of the RNN from predictions made on previous tokens in the current sequence, this allows RNNs to understand context. The hidden state is determined by the output of the previous token. 

When you predict for the first token of any sequence, if you were to retain the hidden state from the previous sequence your model would perform as if the new sequence was a continuation of the old sequence which would give worse results. Instead for the first token you initialise an empty hidden state, which will then be filled with the model state and used for the second token. 

Think about it this way: if someone asked you to classify a sentence and handed you the US constitution (irrelevant information) vs. if someone gave you some background context about the sentence and then asked you to classify the sentence. 
