Finding patterns in time series with PyTorch
I started PyTorch with image recognition. Now I want to test (very basically) with pure NumPy arrays. I struggle with getting the setup to work, so basically I have vectors with values between 0 and 1 (normalized curves). Those vectors are always of length 1500 and I want to find e.g. &quot;high values at the beginning&quot; or &quot;sine wave-like function&quot;, &quot;convex&quot;, &quot;concave&quot; etc. stuff like that, so just shapes of those curves.
My training set consists of many vectors with their classes; I have chosen 7 classes. The net should be trained to classify a vector into one or more of those 7 classes (not one hot).
I'm struggling with multiple issues, but first my very basic Net
class Net(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(Net, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim)
        self.fc = nn.Linear(self.hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(1), self.hidden_dim).requires_grad_()
        out, h0 = self.rnn(x, h0.detach())
        out = out[:, -1, :]
        out = self.fc(out)
        return out

network = Net(1500, 70, 20, 7)
optimizer = optim.SGD(network.parameters(), lr=learning_rate, momentum=momentum)

This is just a copy-paste from an RNN demo. Here is my first issue. Is an RNN the right choice? It is a time series, but then again it is an image recognition problem when plotting the curve.
Now, this here is an attempt to batch the data. The data object contains all training curves together with the correct classifiers.
def train(epoch):
    network.train()
    network.float()
    batching = True
    index = 0
    # monitor the cummulative loss for an epoch
    cummloss = []
    # start batching some curves
    while batching:
        optimizer.zero_grad()


        # here I start clustering come curves to a batch and normalize the curves
        _input = []
        batch_size = min(len(data)-1, index+batch_size_train) - index
        for d in data[index:min(len(data)-1, index+batch_size_train)]:
            y = np.array(d['data']['y'], dtype='d')
            y = np.multiply(y, y.max())
            y = y[0:1500]
            y = np.pad(y, (0, max(1500-len(y), 0)), 'edge')
            if len(_input) == 0:
                _input = y
            else:
                _input = np.vstack((_input, y))
        input = torch.from_numpy(_input).float()
        input = torch.reshape(input, (1, batch_size, len(y)))
        target = np.zeros((1,7))
        # the correct classes have indizes, to I create a vector with 1 at the correct locations
        for _index in np.array(d['classifier']):
            target[0,_index-1] = 1
        target = torch.from_numpy(target)


        # get the result form the network
        output = network(input)
        # is this a good loss function?
        loss = F.l1_loss(output, target)
        loss.backward()
        cummloss.append(loss.item())
        optimizer.step()
        index = index + batch_size_train
        if index &gt; len(data):
            print(np.mean(cummloss))
            batching = False

for e in range(1, n_epochs):
    print('Epoch: ' + str(e))
    train(0)

The problem I'm facing right now is, the loss doesn't change very little, even with hundreds of epochs.
Are there existing examples of this kind of problem? I didn't find any, just pure png/jpg image recognition. When I convert the curves to png then I have a little issue to train a net, I took densenet and it worked just fine but it seems to be super overkill for this simple task.


This is just a copy-paste from an RNN demo. Here is my first issue. Is an RNN the right choice?

In theory what model you choose does not matter as much as &quot;How&quot; you formulate your problem.
But in your case the most obvious limitation you're going to face is your sequence length: 1500. RNN store information across steps and typically runs into trouble over long sequence with vanishing or exploding gradient.
LSTM net have been developed to circumvent this limitations with memory cell, but even then in the case of long sequence it will still be limited by the amount of information stored in the cell.
You could try using a CNN network as well and think of it as an image.

Are there existing examples of this kind of problem?

I don't know but I might have some suggestions : If I understood your problem correctly, you're going from a (1500, 1) input to a (7,1) output, where 6 of the 7 positions are 0 except for the corresponding class where it's 1.
I don't see any activation function, usually when dealing with multi class you don't use the output of the dense layer to compute the loss you apply a normalizing function like softmax and then you can compute the loss.
