# Collection google/gtr/1

Collection of Generalizable T5-based dense Retrievers (GTR) models.

<!-- task: text-embedding -->
<!-- network-architecture: transformer -->
<!-- language: en -->

## Overview

The GTR models are dual encoders that encode two pieces of text into two dense
vectors respectively. This is typically used to encode a query and a document to
compute their similarity for dense retrieval.

GTR models are built on top of [T5](https://arxiv.org/pdf/1910.10683.pdf) (i.e.
the Text-To-Text Transfer Transformer). The model is first initialized from a
pre-trained T5 checkpoint. It is then further pre-trained with a set of
community question-answer pairs we collected. Finally, the model is fine-tuned
on the [MS Marco](https://microsoft.github.io/msmarco/) dataset.

The two encoders are shared so the GTR model functions as a single text encoder.
The input is variable-length English text and the output is a 768-dimensional
vector.

#### Models

Different sizes of the GTR models are available:

| Model                                                 | Comments            |
| ----------------------------------------------------- | ------------------- |
| [GTR-Base](https://tfhub.dev/google/gtr/gtr-base/1)   | 12-layer, 12 heads, |
:                                                       : encoder-only        :
| [GTR-Large](https://tfhub.dev/google/gtr/gtr-large/1) | 24-layer, 16 heads, |
:                                                       : encoder-only        :
| [GTR-XL](https://tfhub.dev/google/gtr/gtr-xl/1)       | 24-layer, 32 heads, |
:                                                       : encoder-only        :
| [GTR-XXL](https://tfhub.dev/google/gtr/gtr-xxl/1)     | 24-layer, 128 heads, |
:                                                       : encoder-only        :

## References

[1] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma,
Vincent Zhao, Yi Luan, Keith B. Hall, Ming-wei Chang, Yinfei Yang.
[Large Dual Encoders Are Generalizable Retrievers.](https://arxiv.org/abs/)
December 2021.

Add the GTR collection to TF hub.
Add the GTR collection to TF hub.

PiperOrigin-RevId: 416161088
