Implementing a many-to-many regression task
Sorry if I present my problem not clearly, English is not my first language

Problem

Short description:

I want to train a model which map input x (with shape of [n_sample, timestamp, feature]) to an output y (with exact same shape). It's like mapping 2 space

Longer version:

I have 2 float ndarrays of shape [n_sample, timestamp, feature], representing MFCC feature of n_sample audio file. These 2 ndarray are 2 speakers' speech of the same corpus, which was aligned by DTW.  Lets name these 2 arrays x and y. I want to train a model, which predict y[k] given x[k]. It's like mapping from space x to space y, and the output must be exact same shape as the input

What I've tried

It's time-series problem so I decide to use RNN approach. Here is my code in PyTorch (I put comment along the code. I removed the calculation of average loss for simplicity). Note that I've tried many option for learning rate, the behavior still the same

Class define

class Net(nn.Module):
    def __init__(self, in_size, hidden_size, out_size, nb_lstm_layers):
        super().__init__()
        self.in_size = in_size
        self.hidden_size = hidden_size
        self.out_size = out_size
        self.nb_lstm_layers = nb_lstm_layers

        # self.fc1 = nn.Linear()
        self.lstm = nn.LSTM(input_size=self.in_size, hidden_size=self.hidden_size, num_layers=self.nb_lstm_layers, batch_first=True, bias=True)
        # self.fc = nn.Linear(self.hidden_size, self.out_size)
        self.fc1 = nn.Linear(self.hidden_size, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, self.out_size)

    def forward(self, x, h_state):
        out, h_state = self.lstm(x, h_state)
        output_fc = []

        for frame in out:
            output_fc.append(self.fc3(torch.tanh(self.fc1(frame)))) # I added fully connected layer to each frame, to make an output with same shape as input

        return torch.stack(output_fc), h_state

    def hidden_init(self):
        if use_cuda:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)]).cuda()
        else:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)])

        return h_state


Training step:

net = Net(20, 20, 20, nb_lstm_layers)
optimizer = optim.Adam(net.parameters(), lr=0.0001, weight_decay=0.0001)
criterion = nn.MSELoss()

for epoch in range(nb_epoch):
    count = 0
    loss_sum = 0

    batch_x = None
    for i in (range(len(data))): 
    # data is my entire data, which contain A and B i specify above.
        temp_x = torch.tensor(data[i][0])
        temp_y = torch.tensor(data[i][1])

        for ii in range(0, data[i][0].shape[0] - nb_frame_in_batch*2 + 1): # Create batches 
            batch_x, batch_y = get_batches(temp_x, temp_y, ii, batch_size, nb_frame_in_batch)  
            # this will return 2 tensor of shape (batch_size, nb_frame_in_batch, 20), 
            # with `batch_size` is the number of sample each time I feed to the net, 
            # nb_frame_in_batch is the number of frame in each sample
            optimizer.zero_grad()

            h_state = net.hidden_init()

            prediction, h_state = net(batch_x.float(), h_state)
            loss = criterion(prediction.float(), batch_y.float())

            h_state = (h_state[0].detach(), h_state[1].detach())

            loss.backward()
            optimizer.step()


Problem is, the loss seems not to decrease but fluctuate a lot, without a clear behaviour



Please help me. Any suggestion will be greatly appreciated. If somebody can inspect my code and provide some comment, that would be so kind.
Thanks in advance!

It seems the network learning nothing from your data, hence the loss fluctuation (since weights depends on random initialization only). There are something you can try:

Try to normalize the data (this suggestion is quite broad, but I can't give you more details since I don't have your data, but normalize it to a specific range like [0, 1], or to a mean and std value is worth trying)
One very typical problem of LSTM in pytorch is its input dimension is quite different to other type of neural network. You must feed into your network a tensor with shape (seq_len, batch, input_size). You should go here, LSTM section for better details
One more thing: try to tune your hyperparameters. LSTM is harder to train compare to FC or CNN (to my experience).

Tell me if you have improvement. Debugging a neural network is always hard and full of potential coding mistake
