<h3 dir="auto">System Info</h3>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.30.0.dev0</li>
<li>Platform: Linux-5.15.107+-x86_64-with-glibc2.31</li>
<li>Python version: 3.10.11</li>
<li>Huggingface_hub version: 0.14.1</li>
<li>Safetensors version: 0.3.1</li>
<li>PyTorch version (GPU?): 2.0.1+cu118 (True)</li>
<li>Tensorflow version (GPU?): 2.12.0 (True)</li>
<li>Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)</li>
<li>Jax version: 0.4.10</li>
<li>JaxLib version: 0.4.10</li>
<li>Using GPU in script?: </li>
<li>Using distributed or parallel set-up in script?: </li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import BitsAndBytesConfig

model_name = 'mosaicml/mpt-7b-instruct'

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type=&quot;nf4&quot;,
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, offload_folder=&quot;offload&quot;, 
                                                 offload_state_dict = True, quantization_config=nf4_config)"><pre class="notranslate"><code class="notranslate">import torch
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import BitsAndBytesConfig

model_name = 'mosaicml/mpt-7b-instruct'

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, offload_folder="offload", 
                                                 offload_state_dict = True, quantization_config=nf4_config)
</code></pre></div>
<p dir="auto">Error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError: MPTForCausalLM does not support `device_map='auto'` yet."><pre class="notranslate"><code class="notranslate">ValueError: MPTForCausalLM does not support `device_map='auto'` yet.
</code></pre></div>
<p dir="auto">I saw a similar issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1625966407" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/22188" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/22188/hovercard" href="https://github.com/huggingface/transformers/issues/22188">#22188</a> for <code class="notranslate">XGLMForCausalLM</code>. I couldn't find <code class="notranslate">MPTForCausalLM </code> in the repository if the MPT model is not supported currently, is there any hack that I can use to get <code class="notranslate">acclerate</code> support ? Than</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Model gets loaded without any errors</p>
<h3 dir="auto">System Info</h3>
<p dir="auto">Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.</p>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 4.24.0</li>
<li>Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.31</li>
<li>Python version: 3.10.9</li>
<li>Huggingface_hub version: 0.10.1</li>
<li>PyTorch version (GPU?): 1.12.1 (False)</li>
<li>Tensorflow version (GPU?): not installed (NA)</li>
<li>Flax version (CPU?/GPU?/TPU?): not installed (NA)</li>
<li>Jax version: not installed</li>
<li>JaxLib version: not installed</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: yes</li>
</ul>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ArthurZucker/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ArthurZucker">@ArthurZucker</a>  <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<ol dir="auto">
<li>Loaded the InstructCodeT5+ model as per <a href="https://huggingface.co/Salesforce/instructcodet5p-16b" rel="nofollow">https://huggingface.co/Salesforce/instructcodet5p-16b</a></li>
<li>Tried to use LoRA to fine-tune InstructCodeT5+ on NL to code translation task. Code and Traceback given below:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
token_id=&quot;Salesforce/instructcodet5p-16b&quot;
tokenizer = AutoTokenizer.from_pretrained(token_id)
model = AutoModelForSeq2SeqLM.from_pretrained(token_id,
                                              torch_dtype=torch.float16,
                                              low_cpu_mem_usage=True,
                                              trust_remote_code=True, decoder_start_token_id=1, pad_token_id=-100, load_in_8bit=True, device_map='auto').to(device)"><pre class="notranslate"><code class="notranslate">from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
token_id="Salesforce/instructcodet5p-16b"
tokenizer = AutoTokenizer.from_pretrained(token_id)
model = AutoModelForSeq2SeqLM.from_pretrained(token_id,
                                              torch_dtype=torch.float16,
                                              low_cpu_mem_usage=True,
                                              trust_remote_code=True, decoder_start_token_id=1, pad_token_id=-100, load_in_8bit=True, device_map='auto').to(device)
</code></pre></div>
<p dir="auto">Error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-12-1436d2a64ffb&gt; in &lt;module&gt;
     10 
     11 # load model from the hub
---&gt; 12 model = AutoModelForSeq2SeqLM.from_pretrained(model_id,
     13                                               torch_dtype=torch.float16,
     14                                               low_cpu_mem_usage=True,

~/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    460                 class_ref, pretrained_model_name_or_path, **hub_kwargs, **kwargs
    461             )
--&gt; 462             return model_class.from_pretrained(
    463                 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    464             )

~/.cache/huggingface/modules/transformers_modules/instructcodet5p-16b/modeling_codet5p.py in from_pretrained(cls, *args, **kwargs)
    855             )
    856         kwargs[&quot;_fast_init&quot;] = False
--&gt; 857         return super().from_pretrained(*args, **kwargs)
    858 
    859     def forward(

~/.local/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2683 
   2684             if model._no_split_modules is None:
-&gt; 2685                 raise ValueError(f&quot;{model.__class__.__name__} does not support `device_map='{device_map}'` yet.&quot;)
   2686             no_split_modules = model._no_split_modules
   2687             if device_map not in [&quot;auto&quot;, &quot;balanced&quot;, &quot;balanced_low_0&quot;, &quot;sequential&quot;]:

ValueError: CodeT5pEncoderDecoderModel does not support `device_map='auto'` yet."><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-12-1436d2a64ffb&gt; in &lt;module&gt;
     10 
     11 # load model from the hub
---&gt; 12 model = AutoModelForSeq2SeqLM.from_pretrained(model_id,
     13                                               torch_dtype=torch.float16,
     14                                               low_cpu_mem_usage=True,

~/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    460                 class_ref, pretrained_model_name_or_path, **hub_kwargs, **kwargs
    461             )
--&gt; 462             return model_class.from_pretrained(
    463                 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    464             )

~/.cache/huggingface/modules/transformers_modules/instructcodet5p-16b/modeling_codet5p.py in from_pretrained(cls, *args, **kwargs)
    855             )
    856         kwargs["_fast_init"] = False
--&gt; 857         return super().from_pretrained(*args, **kwargs)
    858 
    859     def forward(

~/.local/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2683 
   2684             if model._no_split_modules is None:
-&gt; 2685                 raise ValueError(f"{model.__class__.__name__} does not support `device_map='{device_map}'` yet.")
   2686             no_split_modules = model._no_split_modules
   2687             if device_map not in ["auto", "balanced", "balanced_low_0", "sequential"]:

ValueError: CodeT5pEncoderDecoderModel does not support `device_map='auto'` yet.
</code></pre></div>
<p dir="auto">Seems Like this functionality is not included yet. When to expect it to be added? Thanks in advance!</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">Expect the model to get loaded without any error</p>