Deal with negative values resulting from my nn model
I have a simple nn model that looks like this

class TestRNN(nn.Module):
    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_outputs):
        super(TestRNN, self).__init__()
        ...
        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)
        self.FC = nn.Linear(self.n_neurons, self.n_outputs)


    def forward(self, X):
        ...
        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)


and I am using criterion = nn.CrossEntropyLoss() for calculating my error.  The operation order goes something like this:

# get the inputs
x, y = data

# forward + backward + optimize
outputs = model(x)
loss = criterion(outputs, y)


Where my training data x is normalized and looks like this:

tensor([[[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [2.6164e-02, 2.6164e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 1.3108e-05],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [9.5062e-01, 3.1036e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [0.0000e+00, 1.3717e-05, 3.2659e-07,  ..., 0.0000e+00,
          0.0000e+00, 3.2659e-07]],

        [[5.1934e-01, 5.4041e-01, 6.8083e-06,  ..., 0.0000e+00,
          0.0000e+00, 6.8083e-06],
         [5.2340e-01, 6.0007e-01, 2.7062e-06,  ..., 0.0000e+00,
          0.0000e+00, 2.7062e-06],
         [8.1923e-01, 5.7346e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0714e-01, 7.0708e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 7.0407e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        ...,

        [[7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.1852e-01, 2.3411e-02, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0775e-01, 7.0646e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 3.9888e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.9611e-01, 5.8796e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0710e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.7538e-01, 2.4842e-01, 1.7787e-06,  ..., 0.0000e+00,
          0.0000e+00, 1.7787e-06],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00]],

        [[5.2433e-01, 5.2433e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [1.3155e-01, 1.3155e-01, 0.0000e+00,  ..., 8.6691e-02,
          9.7871e-01, 0.0000e+00],
         [7.4412e-01, 6.6311e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 0.0000e+00],
         [7.0711e-01, 7.0711e-01, 0.0000e+00,  ..., 0.0000e+00,
          0.0000e+00, 9.6093e-07]]])


While a typical output and y passed to the criterion function look like this:

tensor([[-0.0513],
        [-0.0445],
        [-0.0514],
        [-0.0579],
        [-0.0539],
        [-0.0323],
        [-0.0521],
        [-0.0294],
        [-0.0372],
        [-0.0518],
        [-0.0516],
        [-0.0501],
        [-0.0312],
        [-0.0496],
        [-0.0436],
        [-0.0514],
        [-0.0518],
        [-0.0465],
        [-0.0530],
        [-0.0471],
        [-0.0344],
        [-0.0502],
        [-0.0536],
        [-0.0594],
        [-0.0356],
        [-0.0371],
        [-0.0513],
        [-0.0528],
        [-0.0621],
        [-0.0404],
        [-0.0403],
        [-0.0562],
        [-0.0510],
        [-0.0580],
        [-0.0516],
        [-0.0556],
        [-0.0063],
        [-0.0459],
        [-0.0494],
        [-0.0460],
        [-0.0631],
        [-0.0525],
        [-0.0454],
        [-0.0509],
        [-0.0522],
        [-0.0426],
        [-0.0527],
        [-0.0423],
        [-0.0572],
        [-0.0308],
        [-0.0452],
        [-0.0555],
        [-0.0479],
        [-0.0513],
        [-0.0514],
        [-0.0498],
        [-0.0514],
        [-0.0471],
        [-0.0505],
        [-0.0467],
        [-0.0485],
        [-0.0520],
        [-0.0517],
        [-0.0442]], device='cuda:0', grad_fn=&lt;ViewBackward&gt;)
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')


When the criterion is being applied I get the following error (running with CUDA_LAUNCH_BLOCKING=1):

/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [7,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [20,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered


The fact that my model outputs negative values is causing the error message above, how can I resolve this issue?

TL;DR

You have two options:


Make the second dimension of outputs be of size 2 instead of 1.
Use nn.BCEWithLogitsLoss instead of nn.CrossEntropyLoss




I think that the problem is not the negative numbers. It is the shape of outputs.

Looking at your array y, I see that you have 2 different classes (maybe even more, but let's suppose that it's 2). This means that the last dimension of outputs should be 2. The reason is that outputs needs to give a "score" to each one of the 2 different classes (see the documentation). The score can be negative, zero or positive. But the shape of your outputs is [64,1], and not [64,2] as required.

One of the steps of the nn.CrossEntropyLoss() object will be to convert these scores to a probability distribution over the two classes. This is done using a softmax operation. However, When doing binary classification (that is, classification with only 2 classes, as in our current case), there is another option: Give a score for only one class, convert it to a probability for that class using a sigmoid function, and then perform "1-p" on this to get the probability for the other class. This option means that outputs needs to give a score for only one of the two classes, as in you current case. To chose this options, you will need to change nn.CrossEntropyLoss with nn.BCEWithLogitsLoss. You can then pass to it outputs and y as you are currently doing (note however that the shape of outputs needs to be precisely the shape of y, so in your example you will need to pass outputs[:,0] instead of outputs. Also you will need to convert y to a float: y.float(). Thus the call is criterion(outputs[:,0], y.float()))
