Output from LSTM not changing for different inputs
I have the an LSTM implemented in PyTorch as below. 

import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

class LSTM(nn.Module):
    """
    Defines an LSTM.
    """

    def __init__(self, input_dim, hidden_dim, output_dim, num_layers):

        super(LSTM, self).__init__()

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)

    def forward(self, input_data):

        lstm_out_pre, _ = self.lstm(input_data)
        return lstm_out_pre

model = LSTM(input_dim=2, hidden_dim=2, output_dim=1, num_layers=8)

random_data1 = torch.Tensor(np.random.standard_normal(size=(1, 5, 2)))
random_data2 = torch.Tensor(np.random.standard_normal(size=(1, 5, 2)))

out1 = model(random_data1).detach().numpy()
out2 = model(random_data2).detach().numpy()

print(out1)
print(out2)


I am simply creating an LSTM network and passing two random inputs into it. The outputs does not make sense because no matter what random_data1 and random_data2 is, out1 and out2 are always the same. This does not make any sense to me as random inputs multiplied with random weights should give different outputs. 

This does not seem to be the case if I use less number of hidden layers. With num_layers=2, this effect seems to be nil. And as you increase it, out1 and out2 keeps on getting closer. This does not make sense to me because with more layers of the LSTM stacked on top of each other, we are multiplying the input with more number of random weights which should magnify the differences in the input and give a very different output.

Can someone please explain this behavior? Is there something wrong with my implementation?

In one particular run, random_data1 is 

tensor([[[-2.1247, -0.1857],
         [ 0.0633, -0.1089],
         [-0.6460, -0.1079],
         [-0.2451,  0.9908],
         [ 0.4027,  0.3619]]])


random_data2 is

tensor([[[-0.9725,  1.2400],
         [-0.4309, -0.7264],
         [ 0.5053, -0.9404],
         [-0.6050,  0.9021],
         [ 1.4355,  0.5596]]])


out1 is 

[[[0.12221643 0.11449362]
  [0.18342148 0.1620608 ]
  [0.2154751  0.18075559]
  [0.23373817 0.18768947]
  [0.24482158 0.18987371]]]


out2 is

[[[0.12221643 0.11449362]
  [0.18342148 0.1620608 ]
  [0.2154751  0.18075559]
  [0.23373817 0.18768945]
  [0.24482158 0.18987371]]]


EDIT:
I am running on the following configurations -

PyTorch - 1.0.1.post2 
Python - 3.6.8 with GCC 7.3.0 
OS - Pop!_OS 18.04 (Ubuntu 18.04, more-or-less) 
CUDA - 9.1.85
Nvidia driver - 410.78


Initial weights for LSTM are small numbers close to 0, and by adding more layers the initial weighs and biases are getting smaller: all the weights and biases are initialized from -sqrt(k) to -sqrt(k), where k = 1/hidden_size (https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)

By adding more layers you effectively multiply the input by many small numbers, so effect of the input is basically 0 and only biases in the later layers matter.

If you try LSTM with bias=False, you will see that output getting closer and closer to 0 with adding more layers.
