<h1 dir="auto"><g-emoji class="g-emoji" alias="question" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/2753.png">❓</g-emoji> Questions &amp; Help</h1>
<p dir="auto">I am trying out a finetuned BERT model for token classification (--&gt; <a href="https://huggingface.co/bert-base-cased-finetuned-conll03-english" rel="nofollow">https://huggingface.co/bert-base-cased-finetuned-conll03-english</a>), but when I observe the model output (i.e. the logits after applying the softmax) and compare it with the true label_ids, they are totally uncorrelated (see pictures).</p>
<p dir="auto"><a href="https://i.stack.imgur.com/gVyMn.png" rel="nofollow">https://i.stack.imgur.com/gVyMn.png</a><br>
<a href="https://i.stack.imgur.com/qS62L.png" rel="nofollow">https://i.stack.imgur.com/qS62L.png</a></p>
<h2 dir="auto">Details</h2>
<p dir="auto">I assume that the finetuned model (bert-base-cased-finetuned-conll03-english) is correctly pretrained, but I don't seem to understand why its predictions are off. I think one issue is that the pretrained model has another labelling scheme than I made myself during data prep (so that the tag2name dict is different), but I don't know how I can find out what label-index map the model uses for its predictions. Even then it is not the case that the model consistently makes the same mistakes, it is outputting things quite randomly.</p>
<p dir="auto">Any idea what the issue could be?</p>
<p dir="auto">``</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature request</h1>
<p dir="auto">I am using Bert as a decoder (by setting is_decoder=True). However, during sequential decoding, there is no way of reusing the hidden states, so for every word to be generated we need to rerun the model on the ENTIRE decoded sequence, which renders decoding inefficient. Can you add something similar to the keyword <code class="notranslate">past=</code> in GPT2 model to BERT's forward function (<a href="https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L938">https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L938</a>)?</p>
<h2 dir="auto">Generalization of my Issue</h2>
<p dir="auto">More generally, to the best of my knowledge, there's no model in this library that simultaneously supports 1) cross attention (by feeding <code class="notranslate">encoder_hidden_states=</code> or <code class="notranslate">memory=</code>), and 2) reusing decoder states during sequential decoding (by feeding <code class="notranslate">past=</code>). 1) rules out models like GPT-2 and XLNet which only supports language modeling (although in theory we can just use a decoder to do translation, I want to use a separate encoder and decoder); and 2) rules out models like BERT and T5 which supports 1) but not 2). For example, the point of T5 is to use it for text-to-text translation problems, but since we cannot reuse hidden states, sequential decoding (beam search) would be extremely inefficient.</p>
<h2 dir="auto">Example</h2>
<p dir="auto">In the provided summarization example, both 1) and 2) are supported. However, the decoder is defined in its own code (examples/summarization/modeling_bertabs.py) and cannot be used directly in the library. Besides, supporting incremental state update is a basic function that every decoder shall support.</p>
<h2 dir="auto">Relevant Issues</h2>
<p dir="auto">I checked the suggested similar issues and did not find the same issue. Please let me know if my issue duplicates others'.</p>