<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sshleifer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sshleifer">@sshleifer</a></p>
<p dir="auto">I'm using BART model (bart-large), when I try to use the BartForMaskedLM i'm getting the above error. The reason is that in the _combine_masks (line 146 in modeling_bart) is creating a tensor without the device. so by default it is on CPU.  To reproduce - simply use BartForMaskedLM model with GPU.<br>
can you help? am I missing anything?</p>
<h2 dir="auto">Additional details:</h2>
<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.5.1</li>
<li>Python version: 3.7.4</li>
<li>PyTorch version (GPU?): 1.3.0 (with GPU)</li>
<li>Using GPU in script?: yes</li>
<li>Using distributed or parallel set-up in script?: no</li>
</ul>
<p dir="auto">stuck trace:<br>
File "/specific/netapp5_2/gamir/adi/git/BERTese/lama/training.py", line 151, in train_and_eval<br>
outputs = model(b_in_tensor, lm_labels=b_label_tensor)<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 925, in forward<br>
decoder_cached_states=decoder_cached_states,<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 844, in forward<br>
decoder_cached_states=decoder_cached_states,<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 499, in forward<br>
need_attn_weights=self.output_attentions,<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 372, in forward<br>
attn_mask=attention_mask,<br>
File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 629, in forward<br>
attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask<br>
RuntimeError: expected device cuda:0 but got device cpu</p>
<p dir="auto">Thanks,<br>
Adi.</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">Model I am using (Bert, XLNet ...): BART - bart-large</p>
<p dir="auto">Language I am using the model on (English, Chinese ...): English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> my own modified scripts: (give details below)</li>
</ul>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> an official GLUE/SQUaD task: (give the name)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Load model</li>
<li>Tokenize text</li>
<li>Send model and tensor to cuda</li>
<li>Forward the model</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from transformers import BartConfig, BartTokenizer, BartForMaskedLM, BartModel

configuration = BartConfig()
tokenizer_class = BartTokenizer
model_class = BartForMaskedLM(configuration)

tokenizer = tokenizer_class.from_pretrained('bart-large')
model = model_class.from_pretrained('bart-large')

model.eval()
model.to('cuda')

tokens = tokenizer.encode(&quot;Text example to test natural language generation with bart.&quot;)
input_ids = torch.tensor([tokens]) 
input_ids = input_ids.to('cuda')

with torch.no_grad():
    last_hidden_states = model(input_ids)[0]
    
print(&quot;Len tokens:&quot;, len(tokens))
print(&quot;Shape last hidden states:&quot;, last_hidden_states.shape)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">transformers</span> <span class="pl-k">import</span> <span class="pl-v">BartConfig</span>, <span class="pl-v">BartTokenizer</span>, <span class="pl-v">BartForMaskedLM</span>, <span class="pl-v">BartModel</span>

<span class="pl-s1">configuration</span> <span class="pl-c1">=</span> <span class="pl-v">BartConfig</span>()
<span class="pl-s1">tokenizer_class</span> <span class="pl-c1">=</span> <span class="pl-v">BartTokenizer</span>
<span class="pl-s1">model_class</span> <span class="pl-c1">=</span> <span class="pl-v">BartForMaskedLM</span>(<span class="pl-s1">configuration</span>)

<span class="pl-s1">tokenizer</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer_class</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'bart-large'</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-s1">model_class</span>.<span class="pl-en">from_pretrained</span>(<span class="pl-s">'bart-large'</span>)

<span class="pl-s1">model</span>.<span class="pl-en">eval</span>()
<span class="pl-s1">model</span>.<span class="pl-en">to</span>(<span class="pl-s">'cuda'</span>)

<span class="pl-s1">tokens</span> <span class="pl-c1">=</span> <span class="pl-s1">tokenizer</span>.<span class="pl-en">encode</span>(<span class="pl-s">"Text example to test natural language generation with bart."</span>)
<span class="pl-s1">input_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">tensor</span>([<span class="pl-s1">tokens</span>]) 
<span class="pl-s1">input_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">input_ids</span>.<span class="pl-en">to</span>(<span class="pl-s">'cuda'</span>)

<span class="pl-k">with</span> <span class="pl-s1">torch</span>.<span class="pl-en">no_grad</span>():
    <span class="pl-s1">last_hidden_states</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input_ids</span>)[<span class="pl-c1">0</span>]
    
<span class="pl-en">print</span>(<span class="pl-s">"Len tokens:"</span>, <span class="pl-en">len</span>(<span class="pl-s1">tokens</span>))
<span class="pl-en">print</span>(<span class="pl-s">"Shape last hidden states:"</span>, <span class="pl-s1">last_hidden_states</span>.<span class="pl-s1">shape</span>)</pre></div>
<p dir="auto">This code raises the following error:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-24-3bba66ed6aeb&gt; in &lt;module&gt;
      3 
      4 with torch.no_grad():
----&gt; 5     last_hidden_states = model(input_ids)[0]
      6 
      7 print(&quot;Len tokens:&quot;, len(tokens))

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, decoder_cached_states, lm_labels, **unused)
    923             encoder_outputs=encoder_outputs,
    924             decoder_attention_mask=decoder_attention_mask,
--&gt; 925             decoder_cached_states=decoder_cached_states,
    926         )
    927         lm_logits = self.lm_head.forward(outputs[0])

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, encoder_outputs, decoder_attention_mask, decoder_cached_states)
    842             attention_mask,
    843             decoder_attn_mask,
--&gt; 844             decoder_cached_states=decoder_cached_states,
    845         )
    846         # Attention and hidden_states will be [] or None if they aren't needed

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, encoder_hidden_states, encoder_padding_mask, combined_mask, decoder_cached_states, **unused)
    497                 decoder_cached_states=layer_state,
    498                 attention_mask=combined_mask,
--&gt; 499                 need_attn_weights=self.output_attentions,
    500             )
    501             if self.output_past:

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, x, encoder_hidden_states, encoder_attn_mask, decoder_cached_states, attention_mask, need_attn_weights)
    370             decoder_cached_states=decoder_cached_states,
    371             need_weights=need_attn_weights,
--&gt; 372             attn_mask=attention_mask,
    373         )
    374         x = F.dropout(x, p=self.dropout, training=self.training)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, query, key, value, key_padding_mask, decoder_cached_states, need_weights, static_kv, attn_mask)
    627 
    628         if attn_mask is not None:
--&gt; 629             attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
    630             attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
    631 

RuntimeError: expected device cuda:0 but got device cpu"><pre class="notranslate"><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span>
<span class="pl-v">RuntimeError</span>                              <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">24</span><span class="pl-c1">-</span><span class="pl-c1">3</span><span class="pl-s1">bba66ed6aeb</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
      <span class="pl-c1">3</span> 
      <span class="pl-c1">4</span> <span class="pl-k">with</span> <span class="pl-s1">torch</span>.<span class="pl-en">no_grad</span>():
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">5</span>     <span class="pl-s1">last_hidden_states</span> <span class="pl-c1">=</span> <span class="pl-en">model</span>(<span class="pl-s1">input_ids</span>)[<span class="pl-c1">0</span>]
      <span class="pl-c1">6</span> 
      <span class="pl-c1">7</span> <span class="pl-s1">print</span>(<span class="pl-s">"Len tokens:"</span>, <span class="pl-en">len</span>(<span class="pl-s1">tokens</span>))

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">torch</span><span class="pl-c1">/</span><span class="pl-s1">nn</span><span class="pl-c1">/</span><span class="pl-s1">modules</span><span class="pl-c1">/</span><span class="pl-s1">module</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">530</span>             <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_slow_forward</span>(<span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">531</span>         <span class="pl-s1">else</span>:
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">532</span>             <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">forward</span>(<span class="pl-c1">*</span><span class="pl-s1">input</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">533</span>         <span class="pl-k">for</span> <span class="pl-s1">hook</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_forward_hooks</span>.<span class="pl-en">values</span>():
    <span class="pl-c1">534</span>             <span class="pl-s1">hook_result</span> <span class="pl-c1">=</span> <span class="pl-en">hook</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input</span>, <span class="pl-s1">result</span>)

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">transformers</span><span class="pl-c1">/</span><span class="pl-s1">modeling_bart</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input_ids</span>, <span class="pl-s1">attention_mask</span>, <span class="pl-s1">encoder_outputs</span>, <span class="pl-s1">decoder_input_ids</span>, <span class="pl-s1">decoder_attention_mask</span>, <span class="pl-s1">decoder_cached_states</span>, <span class="pl-s1">lm_labels</span>, <span class="pl-c1">**</span><span class="pl-s1">unused</span>)
    <span class="pl-c1">923</span>             <span class="pl-s1">encoder_outputs</span><span class="pl-c1">=</span><span class="pl-s1">encoder_outputs</span>,
    <span class="pl-c1">924</span>             <span class="pl-s1">decoder_attention_mask</span><span class="pl-c1">=</span><span class="pl-s1">decoder_attention_mask</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">925</span>             <span class="pl-s1">decoder_cached_states</span><span class="pl-c1">=</span><span class="pl-s1">decoder_cached_states</span>,
    <span class="pl-c1">926</span>         )
    <span class="pl-c1">927</span>         <span class="pl-s1">lm_logits</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">lm_head</span>.<span class="pl-en">forward</span>(<span class="pl-s1">outputs</span>[<span class="pl-c1">0</span>])

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">transformers</span><span class="pl-c1">/</span><span class="pl-s1">modeling_bart</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input_ids</span>, <span class="pl-s1">attention_mask</span>, <span class="pl-s1">decoder_input_ids</span>, <span class="pl-s1">encoder_outputs</span>, <span class="pl-s1">decoder_attention_mask</span>, <span class="pl-s1">decoder_cached_states</span>)
    <span class="pl-c1">842</span>             <span class="pl-s1">attention_mask</span>,
    <span class="pl-c1">843</span>             <span class="pl-s1">decoder_attn_mask</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">844</span>             <span class="pl-s1">decoder_cached_states</span><span class="pl-c1">=</span><span class="pl-s1">decoder_cached_states</span>,
    <span class="pl-c1">845</span>         )
    <span class="pl-c1">846</span>         <span class="pl-c"># Attention and hidden_states will be [] or None if they aren't needed</span>

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">transformers</span><span class="pl-c1">/</span><span class="pl-s1">modeling_bart</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input_ids</span>, <span class="pl-s1">encoder_hidden_states</span>, <span class="pl-s1">encoder_padding_mask</span>, <span class="pl-s1">combined_mask</span>, <span class="pl-s1">decoder_cached_states</span>, <span class="pl-c1">**</span><span class="pl-s1">unused</span>)
    <span class="pl-c1">497</span>                 <span class="pl-s1">decoder_cached_states</span><span class="pl-c1">=</span><span class="pl-s1">layer_state</span>,
    <span class="pl-c1">498</span>                 <span class="pl-s1">attention_mask</span><span class="pl-c1">=</span><span class="pl-s1">combined_mask</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">499</span>                 <span class="pl-s1">need_attn_weights</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">output_attentions</span>,
    <span class="pl-c1">500</span>             )
    <span class="pl-c1">501</span>             <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">output_past</span>:

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">transformers</span><span class="pl-c1">/</span><span class="pl-s1">modeling_bart</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>, <span class="pl-s1">encoder_hidden_states</span>, <span class="pl-s1">encoder_attn_mask</span>, <span class="pl-s1">decoder_cached_states</span>, <span class="pl-s1">attention_mask</span>, <span class="pl-s1">need_attn_weights</span>)
    <span class="pl-c1">370</span>             <span class="pl-s1">decoder_cached_states</span><span class="pl-c1">=</span><span class="pl-s1">decoder_cached_states</span>,
    <span class="pl-c1">371</span>             <span class="pl-s1">need_weights</span><span class="pl-c1">=</span><span class="pl-s1">need_attn_weights</span>,
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">372</span>             <span class="pl-s1">attn_mask</span><span class="pl-c1">=</span><span class="pl-s1">attention_mask</span>,
    <span class="pl-c1">373</span>         )
    <span class="pl-c1">374</span>         <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-v">F</span>.<span class="pl-en">dropout</span>(<span class="pl-s1">x</span>, <span class="pl-s1">p</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">dropout</span>, <span class="pl-s1">training</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">training</span>)

<span class="pl-c1">~</span><span class="pl-c1">/</span><span class="pl-s1">projects</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">/</span>.<span class="pl-s1">conda</span><span class="pl-c1">/</span><span class="pl-s1">envs</span><span class="pl-c1">/</span><span class="pl-s1">p130</span><span class="pl-c1">-</span><span class="pl-s1">nlg</span><span class="pl-c1">-</span><span class="pl-s1">env</span><span class="pl-c1">-</span><span class="pl-s1">kgpubart</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">site</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">transformers</span><span class="pl-c1">/</span><span class="pl-s1">modeling_bart</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">forward</span>(<span class="pl-s1">self</span>, <span class="pl-s1">query</span>, <span class="pl-s1">key</span>, <span class="pl-s1">value</span>, <span class="pl-s1">key_padding_mask</span>, <span class="pl-s1">decoder_cached_states</span>, <span class="pl-s1">need_weights</span>, <span class="pl-s1">static_kv</span>, <span class="pl-s1">attn_mask</span>)
    <span class="pl-c1">627</span> 
    <span class="pl-c1">628</span>         <span class="pl-k">if</span> <span class="pl-s1">attn_mask</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">629</span>             <span class="pl-s1">attn_weights</span> <span class="pl-c1">=</span> <span class="pl-s1">attn_weights</span>.<span class="pl-en">view</span>(<span class="pl-s1">bsz</span>, <span class="pl-s1">self</span>.<span class="pl-s1">num_heads</span>, <span class="pl-s1">tgt_len</span>, <span class="pl-s1">src_len</span>) <span class="pl-c1">+</span> <span class="pl-s1">attn_mask</span>
    <span class="pl-c1">630</span>             <span class="pl-s1">attn_weights</span> <span class="pl-c1">=</span> <span class="pl-s1">attn_weights</span>.<span class="pl-en">view</span>(<span class="pl-s1">bsz</span> <span class="pl-c1">*</span> <span class="pl-s1">self</span>.<span class="pl-s1">num_heads</span>, <span class="pl-s1">tgt_len</span>, <span class="pl-s1">src_len</span>)
    <span class="pl-c1">631</span> 

<span class="pl-v">RuntimeError</span>: <span class="pl-s1">expected</span> <span class="pl-s1">device</span> <span class="pl-s1">cuda</span>:<span class="pl-c1">0</span> <span class="pl-s1">but</span> <span class="pl-s1">got</span> <span class="pl-s1">device</span> <span class="pl-s1">cpu</span></pre></div>
<p dir="auto">But I have tested this code with another examples (like GPT-2) and it works.</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">I would expect to get the tensor size, as with another models I have tested this code.</p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version: 2.5.1</li>
<li>Platform: Linux</li>
<li>Python version: 3.6.7</li>
<li>PyTorch version (GPU?): 1.4.0. GPU: Yes</li>
<li>Tensorflow version (GPU?): No used</li>
<li>Using GPU in script?: Trying to</li>
<li>Using distributed or parallel set-up in script?: No</li>
</ul>