Recreating char level RNN for generating text
I tied to follow a book on deep learning, where there is an chapter about generating text in the style of an example. They used an char level RNN with two LSTM layers in it to generate text in the style of shakespare. But the code in the book (also online: https://github.com/DOsinga/deep_learning_cookbook/blob/master/05.1%20Generating%20Text%20in%20the%20Style%20of%20an%20Example%20Text.ipynb) is written in keras and I only use pytorch. So i tied to recreate it exactly in pytorch, with same network structure and hyperparameters.

So after recreating it and making it work without errors it trained it and it only learned to write the most common character, a space. Then i tried to overfit it on one realy simple sentence, so I had to decrease the sequence lenght to 8. This also did not work, but when decreasing the hidden size of the LSTMs to only 32 it learned it nearly perfectly.
So then I continued working on the original text and started to play arround with the hidden size, learning rate, optimizer (also tried adam) and trained it even longer. The best I could achieve were some random letters, still with a lot of spaces and somtimes something like "her", but far from readable, with still an quite high loss. I used RMSprop with lr=0.01 and a hidden size of 128 over 20000 epochs. I also tried to initialize the hidden state and cell state to zero.

The problem is, that my results are far worse than those in the book, but I did exactly the same just in pytorch. Can someone please tell me, what I should try or what I have done wrong. Any help is appreciated!
PS: Sorry for my bad english.

Here is my code with the original hyperparameters:

#hyperparameters
batch_size = 256
seq_len = 160
hidden_size = 640
layers = 2

#network structure
class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(len(chars),hidden_size,layers)
        self.linear = nn.Linear(hidden_size,len(chars))
        self.softmax = nn.Softmax(dim=2)
    def forward(self,x,h,c):
        x,(h,c) = self.lstm(x,(h,c))
        x = self.softmax(self.linear(x))
        return x,h,c

#create network, optimizer and criterion
rnn = RNN().cuda()
optimizer = torch.optim.RMSprop(rnn.parameters(),lr=0.01)
criterion = nn.CrossEntropyLoss()

#training loop
plt.ion()
losses = []
loss_sum = 0
for epoch in range(10000):
    #generate input and target filled with zeros
    input = numpy.zeros((seq_len,batch_size,len(chars)))
    target = numpy.zeros((seq_len,batch_size))
    for batch in range(batch_size):
        #choose random starting index in text
        start = random.randrange(len(text)-seq_len-1)
        #generate sequences for that batch filled with zeros
        input_seq = numpy.zeros((seq_len+1,len(chars)))
        target_seq = numpy.zeros((seq_len+1))
        for i,char in enumerate(text[start:start+seq_len+1]):
            #convert character to index
            idx = char_to_idx[char]
            #set value of index to one (one-hot-encoding)
            input_seq[i,idx] = 1
            #set value to index (only label)
            target_seq[i] = idx
        #insert sequences into input and target
        input[:,batch,:] = input_seq[:-1]
        target[:,batch] = target_seq[1:]
    #convert input and target from numpy array to pytorch tensor on gpu
    input = torch.from_numpy(input).float().cuda()
    target = torch.from_numpy(target).long().cuda()

    #initialize hidden state and cell state to zero
    h0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    c0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    #run the network on the input
    output,h,c = rnn(input,h0,c0)
    #calculate loss and perform gradient descent
    optimizer.zero_grad()
    loss = criterion(output.view(-1,len(chars)),target.view(-1))
    loss.backward()
    optimizer.step()



Plot of the loss with original hyperparameters:


Example of target and output after training:

Target:  can bring this instrument of honour
    again into his native quarter, be magnanimous in the enterprise,
    and go on; I will grace the attempt for a worthy e
Output:                                                                                                                                                                 


Plot of the loss with hidden size of 128 over 20000 epochs (best results):


I later finally found a way to achive something close to real sentences, maybe it will help someone. Here is an example result:


  -I have not seen him and the prince was a signt of the streme of the sumpering of the property of th


In my case the important change was to not inizialize the hidden and cell state to zero every batch but only every epoch. For this to work I had to rewrite the batch generator, so that it produces batches following on each other.
