<p dir="auto">Hi<br>
I am using transformer 4.1.1 on multiple gpus with distributed launch script below, the n_gpu set in this case is 1 instead of actual n_gpus, could you tell me how I can access to the total number of ranks calling the launcher with? Is there any variable in huggingface library which could show the number of gpus the script is called with? Here is the command I run:</p>
<p dir="auto"><code class="notranslate">export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9915  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500 </code></p>
<p dir="auto">thanks</p>
<p dir="auto">Hi<br>
I am using transformer 4.1.1 on multiple gpus with distributed launch script below, the n_gpu set in this case is 1 instead of actual n_gpus, could you tell me how I can access to the total number of ranks calling the launcher with? Is there any variable in huggingface library which could show the number of gpus the script is called with? Here is the command I run:</p>
<p dir="auto"><code class="notranslate">export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9915  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500 </code></p>
<p dir="auto">thanks</p>