total_loss = 0
   for i in range(10000):
       optimizer.zero_grad()
       output = model(input)
       loss = criterion(output)
       loss.backward()
       optimizer.step()
       total_loss += loss
Here, "total_loss" is accumulating history across your training loop,
since "loss" is a differentiable variable with autograd history. You
can fix this by writing total_loss += float(loss) instead.
Other instances of this problem: 1.
Don't hold onto tensors and variables you don't need. If you
assign a Tensor or Variable to a local, Python will not deallocate
until the local goes out of scope.  You can free this reference by
using "del x".  Similarly, if you assign a Tensor or Variable to a
member variable of an object, it will not deallocate until the object
goes out of scope.  You will get the best memory usage if you don't
hold onto temporaries you don't need.
The scopes of locals can be larger than you expect.  For example:
for i in range(5):
       intermediate = f(input[i])