Python3 Pytorch RuntimeError on GCP - no msg
My System
I am running a neural network training on using Python 3.6.9 with pytorch 1.6.0
I am using a google cloud platform N1 Server with a Tesla T4, 2 cores CPU, 12GB RAM.
This is on an Ubuntu 18.04 image.
Problem
When my code reaches the training line I get the following RuntimeError with no real explanation that I can see:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File &quot;/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/multiprocessing/spawn.py&quot;, line 20, in _wrap
    fn(i, *args)
  File &quot;/home/or/my_model/train.py&quot;, line 88, in train_and_eval
    train(rank, epoch, hps, generator, optimizer_g, train_loader, logger, writer)
  File &quot;/home/or/my_model/train.py&quot;, line 117, in train
    scaled_loss.backward()
  File &quot;/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/tensor.py&quot;, line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File &quot;/home/or/.local/share/virtualenvs/or-M3_AaJfY/lib/python3.6/site-packages/torch/autograd/__init__.py&quot;, line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError


This happens while the 2 CPU cores are being used at 100% for a long while.
The RAM and GPU, though going up (as expected while training) do not reach close to their
limit.
I checked journalctl to see if this was an operating system issue but there is nothing  there. I also did not find anything relevant in the /var/log/ directory or using dmesg.
I would be happy to provide more log data but I am not aware (after searching) any python logs I can look at, or any other system logs.

Please let me know of ways I can get more information if you have any ideas.
The exact same code works 100% fine on other physical machines I have tested, and a GPU only version of it runs fine on another cloud computing provider
What I am looking for

Ways to get more information about this problem and figure out why it is happening.
Ways to fix this problem

Thanks in advance for your time and any help you may be able to provide.

Anthony Leo thank you so much for your detailed answer!
Unfortunately this ended up being a problem with one of the modules I installed while setting up my server.
This did not end up being a problem of the server itself or of my code, I just installed a module incorrectly while setting up.
I am sorry for all the time other people spent on this issue.
