title: How to see the size of a computation graph in memory in Pytorch?
body: <p>How can one trace the memory allocation for the autograd graph created by the forward pass on cpu? For instance trying to use tracemalloc on cpu:</p>
  
  <pre><code>rnn=nn.RNNCell(100,100).to('cuda')
  x=torch.ones((1000,100),device='cuda')
  tracemalloc.start(25)
  while True:
     print(tracemalloc.get_traced_memory())
     x=rnn(x)
  </code></pre>
  
  <p>The printed memory should continually increase as the graph is increasing in each loop step, but the printed memory from</p>
  
  <pre><code>tracemalloc.get_traced_memory()
  </code></pre>
  
  <p>remains constant after the 3rd loop. What is going on?</p>