<h1 dir="auto">🌟 New model addition</h1>
<p dir="auto">Funnel-Transformer</p>
<h2 dir="auto">Model description</h2>
<p dir="auto">Funnel-Transformer is a new self-attention model that gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, Funnel-Transformer usually has a higher capacity given the same FLOPs. In addition, with a decoder, Funnel-Transformer is able to recover the token-level deep representation for each token from the reduced hidden sequence, which enables standard pretraining.</p>

<h2 dir="auto">Open source status</h2>
<p dir="auto">Released.</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the model implementation is available: (give details)<br>
<a href="https://github.com/laiguokun/Funnel-Transformer">https://github.com/laiguokun/Funnel-Transformer</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the model weights are available: (give details)<br>
<a href="https://github.com/laiguokun/Funnel-Transformer">https://github.com/laiguokun/Funnel-Transformer</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> who are the authors: (mention them, if possible by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gh-username/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gh-username">@gh-username</a>)<br>
Zihang Dai*, Guokun Lai*, Yiming Yang, Quoc V. Le</li>
</ul>
<h1 dir="auto">🌟 New model addition</h1>
<h2 dir="auto">Model description</h2>
<p dir="auto">The recently introduced Funnel-Transformer architecture and models would be a great feature for Transformers:</p>
<blockquote>
<p dir="auto">Funnel-Transformer is a new self-attention model that gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, Funnel-Transformer usually has a higher capacity given the same FLOPs. In addition, with a decoder, Funnel-Transformer is able to recover the token-level deep representation for each token from the reduced hidden sequence, which enables standard pretraining.</p>
</blockquote>
<p dir="auto">The paper can be found <a href="https://arxiv.org/abs/2006.03236" rel="nofollow">here</a>.</p>
<h2 dir="auto">Open source status</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the model implementation is available: <a href="https://github.com/laiguokun/Funnel-Transformer">official GitHub repo</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> the model weights are available: <a href="https://github.com/laiguokun/Funnel-Transformer/blob/master/download_all_ckpts.sh">Google Cloud Bucket</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> who are the authors: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zihangdai/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zihangdai">@zihangdai</a> and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/laiguokun/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/laiguokun">@laiguokun</a></li>
</ul>