<p dir="auto">I've tried to train intermediate layers independently but when I run my code I've got out of memory issue.</p>
<p dir="auto">I would like to know <strong>retain_graph = True</strong> is the problem? or not and what I have to do to solve this problem.</p>
<p dir="auto">I am using hugging face BERT model and this is my code</p>
<p dir="auto">Thank you.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
t_outputs = t_model(**intputs)
s_outputs = s_model(**inputs)
encoder_layers = args.encoder_layers

loss = torch.nn.KLDivLoss(reduction='batchmean')
batch, row, col = s_outputs[3][1].size()

for i, k in enumerate(encoder_layers):
    output = loss(F.log_softmax(s_outputs[3][i+1], dim=2).view(batch*row, 1, col), F.softmax(t_result[3][k+1],dim=2).view(batch*row, 1, col))
   # to freeze layers
    for name, p in s_model.named_parameters():
        if &quot;layer.&quot; + str(i-1) in name:
        p.requires_grad = False
        output.backward(retain_graph=True)
"><pre class="notranslate"><code class="notranslate">
t_outputs = t_model(**intputs)
s_outputs = s_model(**inputs)
encoder_layers = args.encoder_layers

loss = torch.nn.KLDivLoss(reduction='batchmean')
batch, row, col = s_outputs[3][1].size()

for i, k in enumerate(encoder_layers):
    output = loss(F.log_softmax(s_outputs[3][i+1], dim=2).view(batch*row, 1, col), F.softmax(t_result[3][k+1],dim=2).view(batch*row, 1, col))
   # to freeze layers
    for name, p in s_model.named_parameters():
        if "layer." + str(i-1) in name:
        p.requires_grad = False
        output.backward(retain_graph=True)
</code>
</pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mruberry/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mruberry">@mruberry</a></p>
<p dir="auto">Ok, sorry to bother you again, and really sorry if I am just doing it wrong again (I looked at the docs for things that would address this, but didn't see anything after looking at it quickly).</p>
<p dir="auto">It appears that the next segment of code leaks CUDA memory. I don't believe I had this problem on 0.2.0 (or some version close to that) on the code that I was running (that I prefer to not publicly disclose, which is why I came up with this example instead):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="net = torch.nn.Linear(30000,1).cuda()
data = torch.ones(100, 30000).cuda()
for rep in range(1000):
  for batchIndices in range(0,100,100):
    batch = torch.autograd.Variable(data[batchIndices:batchIndices+10], requires_grad=True)
    net.forward(batch).norm(2).backward(create_graph=True)
    batch.grad.norm(2).backward()"><pre class="notranslate"><code class="notranslate">net = torch.nn.Linear(30000,1).cuda()
data = torch.ones(100, 30000).cuda()
for rep in range(1000):
  for batchIndices in range(0,100,100):
    batch = torch.autograd.Variable(data[batchIndices:batchIndices+10], requires_grad=True)
    net.forward(batch).norm(2).backward(create_graph=True)
    batch.grad.norm(2).backward()
</code></pre></div>
<p dir="auto"><code class="notranslate">nvidia-smi</code> output:</p>
<p dir="auto">After first run of this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 33%   55C    P8    14W / 250W |   2417MiB / 12189MiB |      0%      Default |"><pre class="notranslate"><code class="notranslate">|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 33%   55C    P8    14W / 250W |   2417MiB / 12189MiB |      0%      Default |
</code></pre></div>
<p dir="auto">After second run of this (but without the lines defining <code class="notranslate">net</code> and <code class="notranslate">data</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 32%   51C    P2    58W / 250W |   4417MiB / 12189MiB |      0%      Default |"><pre class="notranslate"><code class="notranslate">|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 32%   51C    P2    58W / 250W |   4417MiB / 12189MiB |      0%      Default |
</code></pre></div>
<p dir="auto">This seems like a similar issue to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="275917710" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/3824" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/3824/hovercard" href="https://github.com/pytorch/pytorch/issues/3824">#3824</a>. The code in that issue's first post that reproduces it is similar to mine, so I suspect I am doing it correctly (just to defend myself!).</p>
<p dir="auto"><code class="notranslate">torch.__version__</code> is <code class="notranslate">'0.3.0.post4'</code></p>
<p dir="auto">I downgraded pytorch to 0.2.0 to make sure it didn't happen, and it appears not to; <code class="notranslate">torch.__version__</code> says <code class="notranslate">'0.2.0_0'</code>.</p>
<p dir="auto">Sorry if you are working on this already; I hope that some more feedback wouldn't be too annoying, at least.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ssnl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ssnl">@ssnl</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/albanD/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/albanD">@albanD</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ngimel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ngimel">@ngimel</a></p>