torch.nn.LSTM runtime error
I'm trying to implement the structure from "Livelinet: A multimodal Deep Recurrent Neural Network to Predict Liveliness in Educational Videos".

For a brief explanation, I split 10-second audio clip into ten 1-second audio clip and get a spectrogram (a picture) from that 1-second audio clip. Then I use CNN to get a representation vector from the picture, finally getting 10 vectors of each 1-second video clips.

Next, I feed those 10 vectors to LSTM, and I got some errors there.
My code and the error traceback are as follows:

class AudioCNN(nn.Module):

def __init__(self):
    super(AudioCNN,self).__init__()
    self.features = alexnet.features
    self.features2 = nn.Sequential(*classifier)
    self.lstm = nn.LSTM(512, 256,2)
    self.classifier = nn.Linear(2*256,2)

def forward(self, x):
    x = self.features(x)
    print x.size()
    x = x.view(x.size(0),256*6*6)
    x = self.features2(x)
    x = x.view(10,1,512)
    h_0,c_0 = self.init_hidden()
    _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
    assert False
    x = x.view(1,1,2*256)
    x = self.classifier(x)

    return x

def init_hidden(self):
    h_0 = torch.randn(2,1,256) #layer * batch * input_dim
    c_0 = torch.randn(2,1,256)
    return h_0, c_0

audiocnn = AudioCNN()
input = torch.randn(10,3,223,223)
input = Variable(input)
audiocnn(input)


Error: 

RuntimeErrorTraceback (most recent call last)
&lt;ipython-input-64-2913316dbb34&gt; in &lt;module&gt;()
----&gt; 1 audiocnn(input)

/home//local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--&gt; 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

&lt;ipython-input-60-31881982cca9&gt; in forward(self, x)
     15         x = x.view(10,1,512)
     16         h_0,c_0 = self.init_hidden()
---&gt; 17         _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
     18         assert False
     19         x = x.view(1,1,2*256)

/home/local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--&gt; 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

/home//local/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc in forward(self, input, hx)
    160             flat_weight=flat_weight
    161         )
--&gt; 162         output, hidden = func(input, self.all_weights, hx)
    163         if is_packed:
    164             output = PackedSequence(output, batch_sizes)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, *fargs, **fkwargs)
    349         else:
    350             func = AutogradRNN(*args, **kwargs)
--&gt; 351         return func(input, *fargs, **fkwargs)
    352 
    353     return forward

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, weight, hidden)
    242             input = input.transpose(0, 1)
    243 
--&gt; 244         nexth, output = func(input, hidden, weight)
    245 
    246         if batch_first and batch_sizes is None:

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
     82                 l = i * num_directions + j
     83 
---&gt; 84                 hy, output = inner(input, hidden[l], weight[l])
     85                 next_hidden.append(hy)
     86                 all_output.append(output)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
    111         steps = range(input.size(0) - 1, -1, -1) if reverse else range(input.size(0))
    112         for i in steps:
--&gt; 113             hidden = inner(input[i], hidden, *weight)
    114             # hack to handle LSTM
    115             output.append(hidden[0] if isinstance(hidden, tuple) else hidden)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in LSTMCell(input, hidden, w_ih, w_hh, b_ih, b_hh)
     29 
     30     hx, cx = hidden
---&gt; 31     gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
     32 
     33     ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)

/home//local/lib/python2.7/site-packages/torch/nn/functional.pyc in linear(input, weight, bias)
    551     if input.dim() == 2 and bias is not None:
    552         # fused op is marginally faster
--&gt; 553         return torch.addmm(bias, input, weight.t())
    554 
    555     output = input.matmul(weight.t())

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in addmm(cls, *args)
    922         @classmethod
    923         def addmm(cls, *args):
--&gt; 924             return cls._blas(Addmm, args, False)
    925 
    926         @classmethod

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in _blas(cls, args, inplace)
    918             else:
    919                 tensors = args
--&gt; 920             return cls.apply(*(tensors + (alpha, beta, inplace)))
    921 
    922         @classmethod

RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition


The error message


  RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition


generally indicates that you are passing a tensor or something else that cannot store history as an input into a module. In your case, your problem is that you return tensors in init_hidden() instead of Variable instances. Therefore, when the LSTM runs it cannot compute the gradient for the hidden layer as its initial input is not a part of the backprop graph.

Solution:

def init_hidden(self):
    h_0 = torch.randn(2,1,256) #layer * batch * input_dim
    c_0 = torch.randn(2,1,256)
    return Variable(h_0), Variable(c_0)


It is also likely that a mean of 0 and a variance of 1 is not helpful as initial values for the LSTM hidden state. Ideally, you would make the initial state trainable as well, e.g.:

h_0 = torch.zeros(2,1,256) # layer * batch * input_dim
c_0 = torch.zeros(2,1,256)
h_0_param = torch.nn.Parameter(h_0)
c_0_param = torch.nn.Parameter(c_0)

def init_hidden(self):
    return h_0_param, c_0_param


In this case the network can learn what initial state suits best. Note that in this case there is no need to wrap h_0_param in a Variable as a Parameter is essentially a Variable with require_grad=True.
