<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.6.1</li>
<li>Platform: Linux-4.19.0-16-cloud-amd64-x86_64-with-glibc2.10</li>
<li>Python version: 3.8.10</li>
<li>PyTorch version (GPU?): 1.8.1+cu111 (True)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<p dir="auto">Also in case it matters, my deepspeed version is 0.3.16.</p>
<h3 dir="auto">Who can help</h3>
<p dir="auto">Maybe <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/stas00/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stas00">@stas00</a> ? I'm not sure.</p>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using: custom Pytorch transformer model, although I don't think it matters here.</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (probably, I haven't tried)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: I'm running a simple custom MLM training script using the transformers trainer.</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Run a hyperparameter search using the transformers Trainer with the <a href="https://huggingface.co/transformers/main_classes/trainer.html#zero-2-example" rel="nofollow">default zero-2 config from the documentation</a></li>
<li>Observe that parameters taken from the Deepspeed config like gradient accumulation steps and optimizer/scheduler params are not updated to reflect the Optuna trial parameters.</li>
</ol>
<p dir="auto">Here is some output from the script I'm running, with the middle omitted for brevity. I'm printing trial params myself but I do it from inside the Trainer, so these are definitely the same trial params the Trainer is getting.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[I 2021-05-27 02:20:41,133] A new study created in memory with name: no-name-26248906-22c0-4666-a7d4-159173902bc5
current trial params {'learning_rate': 2.0670100636747183e-05, 'adam_beta2': 0.98, 'gradient_accumulation_steps': 6, 'dropout': 0.0, 'local_attention_window': 192, 'weight_decay': 0.01, 'warmup_ratio': 0.12, 'deep_transformer_stack_layers': 10}

.....

[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   initial_dynamic_scale ........ 65536
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   loss_scale ................... 0
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   memory_breakdown ............. False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_legacy_fusion ...... False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_name ............... adamw
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_params ............. {'lr': 5e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-05-27 02:21:01,788] [INFO] [config.py:751:print]   pld_enabled .................. False

...."><pre class="notranslate"><code class="notranslate">[I 2021-05-27 02:20:41,133] A new study created in memory with name: no-name-26248906-22c0-4666-a7d4-159173902bc5
current trial params {'learning_rate': 2.0670100636747183e-05, 'adam_beta2': 0.98, 'gradient_accumulation_steps': 6, 'dropout': 0.0, 'local_attention_window': 192, 'weight_decay': 0.01, 'warmup_ratio': 0.12, 'deep_transformer_stack_layers': 10}

.....

[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   initial_dynamic_scale ........ 65536
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   loss_scale ................... 0
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   memory_breakdown ............. False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_legacy_fusion ...... False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_name ............... adamw
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_params ............. {'lr': 5e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-05-27 02:21:01,788] [INFO] [config.py:751:print]   pld_enabled .................. False

....
</code></pre></div>
<p dir="auto">I looked around briefly and I <em>think</em> the issue comes from the fact that that the Deepspeed config <a href="https://github.com/huggingface/transformers/blob/996a315e76f6c972c854990e6114226a91bc0a90/src/transformers/training_args.py#L677">is built as part of TrainingArgs</a> and then presumably never updated after that, even if the training args change. Consequently when the <a href="https://github.com/huggingface/transformers/blob/996a315e76f6c972c854990e6114226a91bc0a90/src/transformers/trainer.py#L861">training args are updated</a> as part of setup for the hyperparameter search, it's not reflected in the Deepspeed config.</p>
<p dir="auto">Note that this might also be an issue with Ray, I just haven't tried it with Ray.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Ideally Deepspeed would run with config/parameters that respected the content of the Optuna trials, although I know that getting two external integrations to play well together is easier said than done. In the meantime I'm going to see if I can work around this by using an HF scheduler and HF optimizer in the hopes that those will take their parameters from the training arguments directly.</p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h2>
<p dir="auto">I am using GPT2Tokenizer. I observed that some tokens are duplicated in the vocabulary with a space appended to them in the beginning. For example, there exist two separate tokens - <code class="notranslate">can</code> and <code class="notranslate">&lt;space&gt;can</code>. Both of them are mapped to different token IDs - <code class="notranslate">5171</code> and <code class="notranslate">460</code> respectively.</p>
<p dir="auto">Although, both -</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="can_token_id = tokenizer.encode(' can', add_special_tokens=False)"><pre class="notranslate"><code class="notranslate">can_token_id = tokenizer.encode(' can', add_special_tokens=False)
</code></pre></div>
<p dir="auto">and</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="can_token_id = tokenizer.encode('can', add_special_tokens=False)"><pre class="notranslate"><code class="notranslate">can_token_id = tokenizer.encode('can', add_special_tokens=False)
</code></pre></div>
<p dir="auto">return can_token_id as <code class="notranslate">[5171]</code></p>
<p dir="auto">Is this an intended property of the tokenizer? Is there a mapping available between what token IDs are just a space appended version of other tokenIDs?</p>