<p dir="auto">Paper: <a href="https://arxiv.org/pdf/2005.14165" rel="nofollow">https://arxiv.org/pdf/2005.14165</a></p>
<p dir="auto">GitHub: <a href="https://github.com/openai/gpt-3">https://github.com/openai/gpt-3</a></p>
<p dir="auto">Author: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/8enmann/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/8enmann">@8enmann</a></p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="star2" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f31f.png">🌟</g-emoji> New model addition</h1>
<h2 dir="auto">Model description</h2>
<p dir="auto">The GPT-3 paper just landed on ArXiv: <a href="https://arxiv.org/abs/2005.14165" rel="nofollow">https://arxiv.org/abs/2005.14165</a>.</p>
<p dir="auto">Would be great to integrate it into Transformers, whenever models are available.</p>
<blockquote>
<p dir="auto">Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.</p>
</blockquote>
<h2 dir="auto">Open source status</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> GitHub repository is available: <a href="https://github.com/openai/gpt-3">here</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the model implementation is available: (give details)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the model weights are available: (give details)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> who are the authors: (mention them, if possible by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gh-username/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gh-username">@gh-username</a>)</li>
</ul>