<p dir="auto">I tried to run an 8 GPU training job, and it OOM'd, so I investigated whether it could run on 1 GPU. It could!</p>
<p dir="auto">So here are two commands, the first one says that it is using 14814MiB on GPU 0.<br>
The second says it is using<br>
<code class="notranslate">15594MiB</code> on each.</p>
<p dir="auto">This doesn't happen in PL, which leads me to believe that <code class="notranslate">distributed_scalars</code> is to blame, but I am not sure. Has anyone run into this?<br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patil-suraj/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patil-suraj">@patil-suraj</a></p>
<h3 dir="auto">Delta between two commands</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="- CUDA_VISIBLE_DEVICES=0 python
+ CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2"><pre class="notranslate"><code class="notranslate">- CUDA_VISIBLE_DEVICES=0 python
+ CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2
</code></pre></div>
<h3 dir="auto">Command 1</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
    --model_name_or_path student_pegasus_cnn_12_2 \
    --data_dir cnn_dm \
    --output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --gradient_accumulation_steps=4 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
    --num_train_epochs=5 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --load_best_model_at_end"><pre class="notranslate">CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
    --model_name_or_path student_pegasus_cnn_12_2 \
    --data_dir cnn_dm \
    --output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --gradient_accumulation_steps=4 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
    --num_train_epochs=5 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --load_best_model_at_end</pre></div>
<h3 dir="auto">Command 2</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 fine
tune_trainer.py \
    --model_name_or_path student_pegasus_cnn_12_2 \
    --data_dir cnn_dm \
    --output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --gradient_accumulation_steps=4 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
    --num_train_epochs=5 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --load_best_model_at_end"><pre class="notranslate">CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 fine
tune_trainer.py \
    --model_name_or_path student_pegasus_cnn_12_2 \
    --data_dir cnn_dm \
    --output_dir dpx_cnn_12_2_pl_comb_noDO --overwrite_output_dir --freeze_embeds \
    --learning_rate=3e-5 \
    --warmup_steps 500 --sortish_sampler \
    --gradient_accumulation_steps=4 \
    --per_device_train_batch_size=4 --per_device_eval_batch_size=8 --eval_beams 2 \
    --num_train_epochs=5 \
    --save_steps 3000 --eval_steps 3000 \
    --logging_first_step \
    --max_target_length 56 --val_max_target_length 142 --test_max_target_length 142 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate --load_best_model_at_end</pre></div>
<h2 dir="auto">Environment info</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 3.0.2</li>
<li>Platform: Linux-5.3.0</li>
<li>Python version: 3.6.10</li>
<li>PyTorch version (GPU?): 1.6.0 (True)</li>
<li>Tensorflow version (GPU?): 2.3.0 (True)</li>
<li>Using GPU in script?: Yes</li>
<li>Using distributed or parallel set-up in script?: Distributed</li>
</ul>
<h3 dir="auto">Who can help</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/LysandreJik/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/LysandreJik">@LysandreJik</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a>, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a></p>
<h2 dir="auto">Information</h2>
<p dir="auto">I am using 8xV100s (32GB). The script (<code class="notranslate">run_training.py</code>) works when running on a single machine but I am running into the <code class="notranslate">CUDA out of memory</code> when trying to run distributed training. The behavior is consistent whether or not <code class="notranslate">fp16</code> is <code class="notranslate">True</code>. I am using the publicly available wikitext data.</p>
<p dir="auto">Model I am using (Bert, XLNet ...): Bert</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>My code is <code class="notranslate">run_training.py</code>:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset
    )
global_steps, training_loss = trainer.train()"><pre class="notranslate"><code class="notranslate">bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset
    )
global_steps, training_loss = trainer.train()
</code></pre></div>
<ol start="2" dir="auto">
<li>Run the script in distributed mode:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m torch.distributed.launch \
  --nnodes={num_nodes} \
  --node_rank={rank} \
  --nproc_per_node=8  \
  --run_training.py \
  --max_steps 50 &quot; \
  --train_data_file {train_data_file} &quot; \
  --eval_data_file {test_data_file} &quot; \
  --logging_dir {output_dir} \
  --fp16 False &quot; \
  --gradient_accumulation_steps 1 &quot; \
  --per_gpu_train_batch_size 1 \
  --per_gpu_eval_batch_size 1"><pre class="notranslate"><code class="notranslate">python -m torch.distributed.launch \
  --nnodes={num_nodes} \
  --node_rank={rank} \
  --nproc_per_node=8  \
  --run_training.py \
  --max_steps 50 " \
  --train_data_file {train_data_file} " \
  --eval_data_file {test_data_file} " \
  --logging_dir {output_dir} \
  --fp16 False " \
  --gradient_accumulation_steps 1 " \
  --per_gpu_train_batch_size 1 \
  --per_gpu_eval_batch_size 1
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Given that the script works fine (i.e., not run into the out of memory issue) on a single machine, I would expect multi-node to be the same. Any insight into what might be going on is appreciated!</p>