You may have some code that tries to recover from out of memory
errors.
try:
       run_model(batch_size)
   except RuntimeError: # Out of memory
       for _ in range(batch_size):
           run_model(1)
But find that when you do run out of memory, your recovery code can't
allocate either. That's because the python exception object holds a
reference to the stack frame where the error was raised. Which
prevents the original tensor objects from being freed. The solution is
to move you OOM recovery code outside of the "except" clause.
oom = False
   try:
       run_model(batch_size)
   except RuntimeError: # Out of memory
       oom = True
if oom:
       for _ in range(batch_size):
           run_model(1)
My data loader workers return identical random numbers
You are likely using other libraries to generate random numbers in the
dataset and worker subprocesses are started via "fork". See