Why does my Pytorch tensor size change and contain NaNs after some batches?
I am Training a Pytorch model. After some time, even if on shuffle, the model contains, besides a few finite tensorrows only NaN values:
tensor([[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         ...,
         [ 1.4641,  0.0360, -1.1528,  ..., -2.3592, -2.6310,  6.3893],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],
       device='cuda:0', grad_fn=&lt;AddBackward0&gt;)

The detect_anomaly functions return:
  File &quot;TestDownload.py&quot;, line 701, in &lt;module&gt;
    main(learning_rate, batch_size, epochs, experiment)
  File &quot;TestDownload.py&quot;, line 635, in main
    train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
  File &quot;TestDownload.py&quot;, line 486, in train
    output = F.log_softmax(output, dim=2)
  File &quot;\lib\site-packages\torch\nn\functional.py&quot;, line 1672, in log_softmax
    ret = input.log_softmax(dim)
 (function _print_stack) Traceback (most recent call last):
  File &quot;TestDownload.py&quot;, line 701, in &lt;module&gt;
    main(learning_rate, batch_size, epochs, experiment)
  File &quot;TestDownload.py&quot;, line 635, in main
    train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, experiment)
  File &quot;TestDownload.py&quot;, line 490, in train
    loss.backward()
  File &quot;\lib\site-packages\comet_ml\monkey_patching.py&quot;, line 317, in wrapper
    return_value = original(*args, **kwargs)
  File &quot;\lib\site-packages\torch\tensor.py&quot;, line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File &quot;\lib\site-packages\torch\autograd\__init__.py&quot;, line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.

in reference to the next line output = F.log_softmax(output, dim=2)
It shows another error if I just do it with try-except: (when the loss function is running on a tensor containing NaNs)
[W ..\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in CtcLossBackward. Traceback of forward call that caused the error:
  File &quot;TestDownload.py&quot;, line 734, in &lt;module&gt;
    # In[ ]:
  File &quot;TestDownload.py&quot;, line 667, in main
    test(model, device, test_loader, criterion, epoch, iter_meter, experiment)
  File &quot;TestDownload.py&quot;, line 517, in train
    loss.backward()
  File &quot;\lib\site-packages\torch\nn\modules\module.py&quot;, line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File &quot;\lib\site-packages\torch\nn\modules\loss.py&quot;, line 1590, in forward
    return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
  File &quot;\lib\site-packages\torch\nn\functional.py&quot;, line 2307, in ctc_loss
    return torch.ctc_loss(
 (function _print_stack)
Traceback (most recent call last):
  File &quot;TestDownload.py&quot;, line 518, in train
  File &quot;\lib\site-packages\comet_ml\monkey_patching.py&quot;, line 317, in wrapper
    return_value = original(*args, **kwargs)
  File &quot;\lib\site-packages\torch\tensor.py&quot;, line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File &quot;\lib\site-packages\torch\autograd\__init__.py&quot;, line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function 'CtcLossBackward' returned nan values in its 0th output.

A normal tensor should look like this:
tensor([[[-3.3904, -3.4340, -3.3703,  ..., -3.3613, -3.5098, -3.4344]],

        [[-3.3760, -3.2948, -3.2673,  ..., -3.4039, -3.3827, -3.3919]],

        [[-3.3857, -3.3358, -3.3901,  ..., -3.4686, -3.4749, -3.3826]],

        ...,

        [[-3.3568, -3.3502, -3.4416,  ..., -3.4463, -3.4921, -3.3769]],

        [[-3.4379, -3.3508, -3.3610,  ..., -3.3707, -3.4030, -3.4244]],

        [[-3.3919, -3.4513, -3.3565,  ..., -3.2714, -3.3984, -3.3643]]],
       device='cuda:0', grad_fn=&lt;TransposeBackward0&gt;)

Please notice the double brackets, if they are import.
Code:
for batch_idx, _data in enumerate(train_loader):
    spectrograms, labels, input_lengths, label_lengths = _data
    spectrograms, labels = spectrograms.to(device), labels.to(device)
    optimizer.zero_grad()

    output = model(spectrograms)
    output = F.log_softmax(output, dim=2)
    output = output.transpose(0, 1)  # (time, batch, n_class) # X, 1, 29
    loss = criterion(output, labels, input_lengths, label_lengths)
    loss.backward()
    optimizer.step()
    scheduler.step()
    iter_meter.step()

Additionally, I tried to run it with a bigger batch size (current batch size:1, bigger batch size: 6) and it run without errors until 40% of the first epoch in which I got this error.

Cuda run out of memory

Also, I tried to normalize the data torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128, normalized=True)
And reducing the learning rate from 5e-4 to 5e-5 did not help either.
Additional information: My dataset contains nearly 300000 .wav files and the error came at 3-10% runtime in the first epoch.
I appreciate any hints and I will gladly submit further information.

The source of error can be a corrupted input or label, which would contain a NaN of inf value.
You can check that there is no NaN value in a tensor with
torch.isnan(tensor).any()

Or that all values in a tensor are neither inf nor NaN with
torch.isfinite(tensor).all()

