Loss doesn't decrease in training the pytorch RNN
Here is the RNN network I designed for a sentiment.

class rnn(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size, hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)
        self.h2h = nn.Linear(hidden_size , hidden_size)
        self.relu = nn.Tanh()
        self.sigmoid = nn.LogSigmoid()

    def forward(self, input, hidden):
        hidden_new = self.relu(self.i2h(input)+self.h2h(hidden))
        output = self.h2o(hidden)
        output = self.sigmoid(output)
        return output, hidden_new

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))


Then, I create and train the network as :

RNN = rnn(50, 50, 1)
learning_rate = 0.0005
criteria = nn.MSELoss()
optimizer = optim.Adam(RNN.parameters(), lr=learning_rate)
hidden = RNN.init_hidden()
epochs = 2
for epoch in range(epochs):
    for i in range(len(train['Phrase'])):
        input = convert_to_vectors(train['Phrase'][i])
        for j in range(len(input)):
            temp_input = Variable(torch.FloatTensor(input[j]))
            output, hidden = RNN(temp_input, hidden)
        temp_output = torch.FloatTensor([np.float64(train['Sentiment'][i])/4])
        loss = criteria( output, Variable(temp_output))
        loss.backward(retain_graph = True)
        if (i%20 == 0):
            print('Current loss is ', loss)


The problem is that the loss of the network isn't decreasing. It increases, then decreases and so on. It isn't stable at all. I tried using a smaller learning rate but it doesn't seem to help.

Why is this happening and how can I rectify this?

You just need to call optimizer.step() after you do loss.backward().

Which, by the way, illustrates a common misconception: Backpropagation is not a learning algorithm, it's just a cool way of computing the gradient of the loss w.r.t. your parameters. You then use some variant of Gradient Descent (eg. plain SGD, AdaGrad, etc., in your case Adam) to update the weights given the gradients.
