How to specify different layer sizes in Pytorch LSTM/GRU/RNN
so I know how to work with LSTMs in general with Pytorch. But it bugs me, that you can only specify ONE hidden_size for all your layers in the LSTM. Like this:
lstm = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)

So for all three layers, the size will be 128. But is there really no way to say, for example, that the first layer should be 128, the second 32 and the third 128?
If I missed something in the documentation or you know a work-around, please let me know, thank ya!

Actually, it depends on the shape of your input and you can see How to decide input and hidden layer dimension to torch.nn.RNN?. Also, you have to understand what is the input and the output because there are different ways to deal with the input and the output. In the  A Beginner’s Guide on Recurrent Neural Networks with PyTorch, you can see how the input data is taken in by the model.

Your model can be
lstm = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)
lstm2 = nn.LSTM(input_size=26, hidden_size=32, num_layers=3, dropout=dropout_chance, batch_first=True)
lstm3 = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)

For multi-layer see this model.
# sequence classification model
class M1(nn.Module):
    def __init__(self):
        super(M1, self).__init__()
        
        self.recurrent_layer  = nn.LSTM(hidden_size = 100, input_size = 75, num_layers = 5)
        self.recurrent_layer1  = nn.LSTM(hidden_size = 200, input_size = 100, num_layers = 5)
        self.recurrent_layer2  = nn.LSTM(hidden_size = 300, input_size = 200, num_layers = 5)
        self.project_layer     = nn.Linear(300, 200)
        self.project_layer1    = nn.Linear(200, 100)
        self.project_layer2    = nn.Linear(100, 10)
    
    # the size of input is [batch_size, seq_len(15), input_dim(75)]
    # the size of logits is [batch_size, num_class]
    def forward(self, input, h_t_1=None, c_t_1=None):
        # the size of rnn_outputs is [batch_size, seq_len, rnn_size]
        # self.recurrent_layer.flatten_parameters()
        rnn_outputs, (hn, cn) = self.recurrent_layer(input)
        rnn_outputs, (hn, cn) = self.recurrent_layer1(rnn_outputs)
        rnn_outputs, (hn, cn) = self.recurrent_layer2(rnn_outputs)
        # classify the last step of rnn_outpus
        # the size of logits is [batch_size, num_class]
        logits = self.project_layer(rnn_outputs[:,-1])
        logits = self.project_layer1(logits)
        logits = self.project_layer2(logits)
        return logits

