<h3 dir="auto">System Info</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- `transformers` version: 4.19.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.7
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO"><pre class="notranslate">- <span class="pl-s"><span class="pl-pds">`</span>transformers<span class="pl-pds">`</span></span> version: 4.19.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.7
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU<span class="pl-k">?</span>): 1.9.0 (False)
- Tensorflow version (GPU<span class="pl-k">?</span>): not installed (NA)
- Flax version (CPU<span class="pl-k">?</span>/GPU<span class="pl-k">?</span>/TPU<span class="pl-k">?</span>): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU <span class="pl-k">in</span> script<span class="pl-k">?</span>: NO
- Using distributed or parallel set-up <span class="pl-k">in</span> script<span class="pl-k">?</span>: NO</pre></div>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/younesbelkada/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/younesbelkada">@younesbelkada</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/patrickvonplaten/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/patrickvonplaten">@patrickvonplaten</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lys/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lys">@lys</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<ol dir="auto">
<li>Load opt-350m model from the hub with <code class="notranslate">AutoModelForCausalLM.from_pretrained('facebook/opt-350m')</code></li>
<li>Save the model using <code class="notranslate">model.save_pretrained('save_dir/')</code></li>
<li>Try loading back the model with <code class="notranslate">AutoModelForCausalLM.from_pretrained('save_dir/')</code></li>
</ol>
<p dir="auto">Full code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForCausalLM

opt350m = AutoModelForCausalLM.from_pretrained('facebook/opt-350m')
opt350m.save_pretrained(&quot;local_save_dir/&quot;)
loaded_opt = AutoModelForCausalLM.from_pretrained('local_save_dir/')"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">AutoModelForCausalLM</span>

<span class="pl-s1">opt350m</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForCausalLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'facebook/opt-350m'</span>)
<span class="pl-s1">opt350m</span>.<span class="pl-en">save_pretrained</span>(<span class="pl-s">"local_save_dir/"</span>)
<span class="pl-s1">loaded_opt</span> <span class="pl-c1">=</span> <span class="pl-v">AutoModelForCausalLM</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'local_save_dir/'</span>)</pre></div>
<h3 dir="auto">Expected behavior</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="A RuntimeError will be raised when loading the model from save_dir/

Logs:

Traceback (most recent call last):
  File &quot;/Users/gregoireretourne/opt/miniconda3/envs/health/lib/python3.9/code.py&quot;, line 90, in runcode
    exec(code, self.locals)
  File &quot;&lt;input&gt;&quot;, line 5, in &lt;module&gt;
  File &quot;lib/python3.9/site-packages/transformers/models/auto/auto_factory.py&quot;, line 446, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File &quot;lib/python3.9/site-packages/transformers/modeling_utils.py&quot;, line 2059, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model(
  File &quot;lib/python3.9/site-packages/transformers/modeling_utils.py&quot;, line 2251, in _load_pretrained_model
    raise RuntimeError(f&quot;Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}&quot;)
RuntimeError: Error(s) in loading state_dict for OPTForCausalLM:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([50272, 512]) from checkpoint, the shape in current model is torch.Size([50272, 1024])."><pre class="notranslate">A RuntimeError will be raised when loading the model from save_dir/

Logs:

Traceback (most recent call last):
  File <span class="pl-s"><span class="pl-pds">"</span>/Users/gregoireretourne/opt/miniconda3/envs/health/lib/python3.9/code.py<span class="pl-pds">"</span></span>, line 90, <span class="pl-k">in</span> runcode
    exec(code, self.locals)
  File <span class="pl-s"><span class="pl-pds">"</span>&lt;input&gt;<span class="pl-pds">"</span></span>, line 5, <span class="pl-k">in</span> <span class="pl-k">&lt;</span>module<span class="pl-k">&gt;</span>
  File <span class="pl-s"><span class="pl-pds">"</span>lib/python3.9/site-packages/transformers/models/auto/auto_factory.py<span class="pl-pds">"</span></span>, line 446, <span class="pl-k">in</span> from_pretrained
    <span class="pl-k">return</span> model_class.from_pretrained(pretrained_model_name_or_path, <span class="pl-k">*</span>model_args, config=config, <span class="pl-k">**</span>kwargs)
  File <span class="pl-s"><span class="pl-pds">"</span>lib/python3.9/site-packages/transformers/modeling_utils.py<span class="pl-pds">"</span></span>, line 2059, <span class="pl-k">in</span> from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model(
  File <span class="pl-s"><span class="pl-pds">"</span>lib/python3.9/site-packages/transformers/modeling_utils.py<span class="pl-pds">"</span></span>, line 2251, <span class="pl-k">in</span> _load_pretrained_model
    raise RuntimeError(f<span class="pl-s"><span class="pl-pds">"</span>Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}<span class="pl-pds">"</span></span>)
RuntimeError: Error(s) <span class="pl-k">in</span> loading state_dict <span class="pl-k">for</span> OPTForCausalLM:
	size mismatch <span class="pl-k">for</span> <span class="pl-smi">lm_head.weight: copying a param with shape torch.Size([50272, 512]) from checkpoint, the shape</span> <span class="pl-k">in</span> current model is torch.Size([50272, 1024]).</pre></div>
<h3 dir="auto">System Info</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- `transformers` version: 4.19.0
- Platform: macOS-12.3.1-arm64-i386-64bit
- Python version: 3.8.13
- Huggingface_hub version: 0.2.1
- PyTorch version (GPU?): 1.10.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: &lt;fill in&gt;
- Using distributed or parallel set-up in script?: &lt;fill in&gt;"><pre class="notranslate">- <span class="pl-s"><span class="pl-pds">`</span>transformers<span class="pl-pds">`</span></span> version: 4.19.0
- Platform: macOS-12.3.1-arm64-i386-64bit
- Python version: 3.8.13
- Huggingface_hub version: 0.2.1
- PyTorch version (GPU<span class="pl-k">?</span>): 1.10.2 (False)
- Tensorflow version (GPU<span class="pl-k">?</span>): not installed (NA)
- Flax version (CPU<span class="pl-k">?</span>/GPU<span class="pl-k">?</span>/TPU<span class="pl-k">?</span>): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU <span class="pl-k">in</span> script<span class="pl-k">?</span>: <span class="pl-k">&lt;</span>fill in<span class="pl-k">&gt;</span>
- Using distributed or parallel set-up <span class="pl-k">in</span> script<span class="pl-k">?</span>: <span class="pl-k">&lt;</span>fill in<span class="pl-k">&gt;</span></pre></div>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">When the OPT-350M variant is fine-tuned via huggingface, the resulting model will give the following error when loaded</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model = OPTForCausalLM.from_pretrained(model path)

RuntimeError: Error(s) in loading state_dict for OPTForCausalLM:
        size mismatch for lm_head.weight: copying a param with shape torch.Size([50272, 512]) from checkpoint, the shape in current model is torch.Size([50272, 1024]).
"><pre class="notranslate"><code class="notranslate">model = OPTForCausalLM.from_pretrained(model path)

RuntimeError: Error(s) in loading state_dict for OPTForCausalLM:
        size mismatch for lm_head.weight: copying a param with shape torch.Size([50272, 512]) from checkpoint, the shape in current model is torch.Size([50272, 1024]).

</code></pre></div>
<p dir="auto">##Code to load model</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed, OPTForCausalLM
import torch

def generate_text(model, tokenizer, prompt):
    
    input_ids = tokenizer(prompt, return_tensors=&quot;pt&quot;).input_ids
    generated_ids = model.generate(input_ids, do_sample=True, num_return_sequences=5, max_length=10)
    texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
    
    return texts
    
path = &quot;facebook/opt-350m&quot;
path = &quot;opt/model_ckpts&quot;
model = OPTForCausalLM.from_pretrained(path)
tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)

prompt = &quot;The woman worked as a&quot;

print(generate_text(model, tokenizer, prompt))"><pre class="notranslate"><code class="notranslate">from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed, OPTForCausalLM
import torch

def generate_text(model, tokenizer, prompt):
    
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids
    generated_ids = model.generate(input_ids, do_sample=True, num_return_sequences=5, max_length=10)
    texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
    
    return texts
    
path = "facebook/opt-350m"
path = "opt/model_ckpts"
model = OPTForCausalLM.from_pretrained(path)
tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)

prompt = "The woman worked as a"

print(generate_text(model, tokenizer, prompt))
</code></pre></div>
<p dir="auto">##Training Code</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch as th
from dataset import get_examples, GSMDataset
from transformers import GPT2Tokenizer, GPT2LMHeadModel
from transformers import GPT2Config, AdamW
from transformers import get_scheduler
from tqdm.auto import tqdm
from torch.utils.data import DataLoader

from transformers import AutoModelForCausalLM, AutoTokenizer, OPTModel, OPTConfig, OPTForCausalLM
import torch

model = OPTForCausalLM.from_pretrained(&quot;facebook/opt-350m&quot;)
tokenizer = AutoTokenizer.from_pretrained(&quot;facebook/opt-350m&quot;, use_fast=False)

try:
    model = OPTForCausalLM.from_pretrained(&quot;model_ckpts&quot;)
    print(&quot;model loaded&quot;)
except Exception as e:
    print(e)
train_examples = get_examples(&quot;train&quot;)
train_dset = GSMDataset(tokenizer, train_examples)

device = th.device(&quot;cuda&quot;)

model.to(device)
model.train()

train_loader = DataLoader(train_dset, batch_size=4, shuffle=True)
optim = AdamW(model.parameters(), lr=1e-5)

num_epochs = 10
num_training_steps = num_epochs * len(train_loader)
lr_scheduler = get_scheduler(
    &quot;linear&quot;,
    optimizer=optim,
    num_warmup_steps=0,
    num_training_steps=num_training_steps,
)

pbar = tqdm(range(num_training_steps))
for epoch in range(num_epochs):
    for batch in train_loader:
        optim.zero_grad()
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch, labels=batch[&quot;input_ids&quot;])
        loss = outputs[0]
        loss.backward()
        optim.step()
        lr_scheduler.step()
        pbar.update(1)
        pbar.set_description(f&quot;train_loss: {loss.item():.5f}&quot;)

model.save_pretrained(&quot;model_ckpts/&quot;)"><pre class="notranslate"><code class="notranslate">import torch as th
from dataset import get_examples, GSMDataset
from transformers import GPT2Tokenizer, GPT2LMHeadModel
from transformers import GPT2Config, AdamW
from transformers import get_scheduler
from tqdm.auto import tqdm
from torch.utils.data import DataLoader

from transformers import AutoModelForCausalLM, AutoTokenizer, OPTModel, OPTConfig, OPTForCausalLM
import torch

model = OPTForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m", use_fast=False)

try:
    model = OPTForCausalLM.from_pretrained("model_ckpts")
    print("model loaded")
except Exception as e:
    print(e)
train_examples = get_examples("train")
train_dset = GSMDataset(tokenizer, train_examples)

device = th.device("cuda")

model.to(device)
model.train()

train_loader = DataLoader(train_dset, batch_size=4, shuffle=True)
optim = AdamW(model.parameters(), lr=1e-5)

num_epochs = 10
num_training_steps = num_epochs * len(train_loader)
lr_scheduler = get_scheduler(
    "linear",
    optimizer=optim,
    num_warmup_steps=0,
    num_training_steps=num_training_steps,
)

pbar = tqdm(range(num_training_steps))
for epoch in range(num_epochs):
    for batch in train_loader:
        optim.zero_grad()
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch, labels=batch["input_ids"])
        loss = outputs[0]
        loss.backward()
        optim.step()
        lr_scheduler.step()
        pbar.update(1)
        pbar.set_description(f"train_loss: {loss.item():.5f}")

model.save_pretrained("model_ckpts/")
</code></pre></div>
<p dir="auto">##Dataset module</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import os
import re
import torch as th


def read_jsonl(path: str):
    with open(path) as fh:
        return [json.loads(line) for line in fh.readlines() if line]


def get_examples(split):
    path = os.path.join(&quot;data/&quot;, f&quot;{split}.jsonl&quot;)
    examples = read_jsonl(path)
    
    #examples = examples[0:100]

    for ex in examples:
        ex.update(question=ex[&quot;question&quot;] + &quot;\n&quot;)
        ex.update(answer=ex[&quot;answer&quot;] + &quot;&lt;|endoftext|&gt;&quot;)

    print(f&quot;{len(examples)} {split} examples&quot;)
    return examples


ANS_RE = re.compile(r&quot;#### (\-?[0-9\.\,]+)&quot;)
INVALID_ANS = &quot;[invalid]&quot;


def extract_answer(completion):
    match = ANS_RE.search(completion)
    if match:
        match_str = match.group(1).strip()
        match_str = match_str.replace(&quot;,&quot;, &quot;&quot;)
        return match_str
    else:
        return INVALID_ANS


def is_correct(model_completion, gt_example):
    gt_answer = extract_answer(gt_example[&quot;answer&quot;])
    assert gt_answer != INVALID_ANS
    return extract_answer(model_completion) == gt_answer


class GSMDataset(th.utils.data.Dataset):
    def __init__(self, tokenizer, examples, loss_on_prefix=True):
        self.examples = examples
        self.qns = [ex[&quot;question&quot;] for ex in self.examples]
        self.ans = [ex[&quot;answer&quot;] for ex in self.examples]
        self.qns = tokenizer(self.qns, padding=False)
        self.ans = tokenizer(self.ans, padding=False)
        self.loss_on_prefix = loss_on_prefix
        self.max_len = max(
            [
                len(self.qns[&quot;input_ids&quot;][i]) + len(self.ans[&quot;input_ids&quot;][i])
                for i in range(len(self.examples))
            ]
        )
        print(f&quot;Max tokens: {self.max_len}&quot;)

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, idx):
        qn_tokens = self.qns[&quot;input_ids&quot;][idx]
        ans_tokens = self.ans[&quot;input_ids&quot;][idx]
        pad_tokens = [0] * (self.max_len - len(qn_tokens) - len(ans_tokens))
        tokens = qn_tokens + ans_tokens + pad_tokens
        mask = (
            ([int(self.loss_on_prefix)] * len(qn_tokens))
            + ([1] * len(ans_tokens))
            + ([0] * len(pad_tokens))
        )
        tokens = th.tensor(tokens)
        mask = th.tensor(mask)
        return dict(input_ids=tokens, attention_mask=mask)```

### Expected behavior

```shell
Expected model to load"><pre lang="import" class="notranslate"><code class="notranslate">import os
import re
import torch as th


def read_jsonl(path: str):
    with open(path) as fh:
        return [json.loads(line) for line in fh.readlines() if line]


def get_examples(split):
    path = os.path.join("data/", f"{split}.jsonl")
    examples = read_jsonl(path)
    
    #examples = examples[0:100]

    for ex in examples:
        ex.update(question=ex["question"] + "\n")
        ex.update(answer=ex["answer"] + "&lt;|endoftext|&gt;")

    print(f"{len(examples)} {split} examples")
    return examples


ANS_RE = re.compile(r"#### (\-?[0-9\.\,]+)")
INVALID_ANS = "[invalid]"


def extract_answer(completion):
    match = ANS_RE.search(completion)
    if match:
        match_str = match.group(1).strip()
        match_str = match_str.replace(",", "")
        return match_str
    else:
        return INVALID_ANS


def is_correct(model_completion, gt_example):
    gt_answer = extract_answer(gt_example["answer"])
    assert gt_answer != INVALID_ANS
    return extract_answer(model_completion) == gt_answer


class GSMDataset(th.utils.data.Dataset):
    def __init__(self, tokenizer, examples, loss_on_prefix=True):
        self.examples = examples
        self.qns = [ex["question"] for ex in self.examples]
        self.ans = [ex["answer"] for ex in self.examples]
        self.qns = tokenizer(self.qns, padding=False)
        self.ans = tokenizer(self.ans, padding=False)
        self.loss_on_prefix = loss_on_prefix
        self.max_len = max(
            [
                len(self.qns["input_ids"][i]) + len(self.ans["input_ids"][i])
                for i in range(len(self.examples))
            ]
        )
        print(f"Max tokens: {self.max_len}")

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, idx):
        qn_tokens = self.qns["input_ids"][idx]
        ans_tokens = self.ans["input_ids"][idx]
        pad_tokens = [0] * (self.max_len - len(qn_tokens) - len(ans_tokens))
        tokens = qn_tokens + ans_tokens + pad_tokens
        mask = (
            ([int(self.loss_on_prefix)] * len(qn_tokens))
            + ([1] * len(ans_tokens))
            + ([0] * len(pad_tokens))
        )
        tokens = th.tensor(tokens)
        mask = th.tensor(mask)
        return dict(input_ids=tokens, attention_mask=mask)```

### Expected behavior

```shell
Expected model to load
</code></pre></div>