
# doc2query/msmarco-russian-mt5-base-v1

This is a [doc2query](https://arxiv.org/abs/1904.08375) model based on mT5 (also known as [docT5query](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf)).

It can be used for:
- **Document expansion**: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our [BEIR](https://arxiv.org/abs/2104.08663) paper we showed that BM25+docT5query is a powerful search engine. In the [BEIR repository](https://github.com/beir-cellar/beir) we have an example how to use docT5query with Pyserini.
- **Domain Specific Training Data Generation**: It can be used to generate training data to learn an embedding model. In our [GPL-Paper](https://arxiv.org/abs/2112.07577) / [GPL Example on SBERT.net](https://www.sbert.net/examples/domain_adaptation/README.html#gpl-generative-pseudo-labeling) we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.

## Usage
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_name = 'doc2query/msmarco-russian-mt5-base-v1'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

text = "Python (МФА: [ˈpʌɪθ(ə)n]; в русском языке встречаются названия пито́н или па́йтон) — высокоуровневый язык программирования общего назначения с динамической строгой типизацией и автоматическим управлением памятью, ориентированный на повышение производительности разработчика, читаемости кода и его качества, а также на обеспечение переносимости написанных на нём программ."


def create_queries(para):
    input_ids = tokenizer.encode(para, return_tensors='pt')
    with torch.no_grad():
        # Here we use top_k / top_k random sampling. It generates more diverse queries, but of lower quality
        sampling_outputs = model.generate(
            input_ids=input_ids,
            max_length=64,
            do_sample=True,
            top_p=0.95,
            top_k=10, 
            num_return_sequences=5
            )
        
        # Here we use Beam-search. It generates better quality queries, but with less diversity
        beam_outputs = model.generate(
            input_ids=input_ids, 
            max_length=64, 
            num_beams=5, 
            no_repeat_ngram_size=2, 
            num_return_sequences=5, 
            early_stopping=True
        )


    print("Paragraph:")
    print(para)
    
    print("\nBeam Outputs:")
    for i in range(len(beam_outputs)):
        query = tokenizer.decode(beam_outputs[i], skip_special_tokens=True)
        print(f'{i + 1}: {query}')

    print("\nSampling Outputs:")
    for i in range(len(sampling_outputs)):
        query = tokenizer.decode(sampling_outputs[i], skip_special_tokens=True)
        print(f'{i + 1}: {query}')

create_queries(text)

```

**Note:** `model.generate()` is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.

## Training
This model fine-tuned [google/mt5-base](https://huggingface.co/google/mt5-base) for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the `train_script.py` in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the [mMARCO dataset](https://github.com/unicamp-dl/mMARCO).




{"language": "ru", "license": "apache-2.0", "datasets": ["unicamp-dl/mmarco"], "widget": [{"text": "Python (\u041c\u0424\u0410: [\u02c8p\u028c\u026a\u03b8(\u0259)n]; \u0432 \u0440\u0443\u0441\u0441\u043a\u043e\u043c \u044f\u0437\u044b\u043a\u0435 \u0432\u0441\u0442\u0440\u0435\u0447\u0430\u044e\u0442\u0441\u044f \u043d\u0430\u0437\u0432\u0430\u043d\u0438\u044f \u043f\u0438\u0442\u043e\u0301\u043d \u0438\u043b\u0438 \u043f\u0430\u0301\u0439\u0442\u043e\u043d) \u2014 \u0432\u044b\u0441\u043e\u043a\u043e\u0443\u0440\u043e\u0432\u043d\u0435\u0432\u044b\u0439 \u044f\u0437\u044b\u043a \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043e\u0431\u0449\u0435\u0433\u043e \u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044f \u0441 \u0434\u0438\u043d\u0430\u043c\u0438\u0447\u0435\u0441\u043a\u043e\u0439 \u0441\u0442\u0440\u043e\u0433\u043e\u0439 \u0442\u0438\u043f\u0438\u0437\u0430\u0446\u0438\u0435\u0439 \u0438 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438\u043c \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c \u043f\u0430\u043c\u044f\u0442\u044c\u044e, \u043e\u0440\u0438\u0435\u043d\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u043d\u0430 \u043f\u043e\u0432\u044b\u0448\u0435\u043d\u0438\u0435 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u0430, \u0447\u0438\u0442\u0430\u0435\u043c\u043e\u0441\u0442\u0438 \u043a\u043e\u0434\u0430 \u0438 \u0435\u0433\u043e \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u0430, \u0430 \u0442\u0430\u043a\u0436\u0435 \u043d\u0430 \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u0435 \u043f\u0435\u0440\u0435\u043d\u043e\u0441\u0438\u043c\u043e\u0441\u0442\u0438 \u043d\u0430\u043f\u0438\u0441\u0430\u043d\u043d\u044b\u0445 \u043d\u0430 \u043d\u0451\u043c \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c."}]}
transformers pytorch mt5 text2text-generation ru dataset:unicamp-dl/mmarco arxiv:1904.08375 arxiv:2104.08663 arxiv:2112.07577 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #mt5 #text2text-generation #ru #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 


# doc2query/msmarco-russian-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.

## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.

## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.




# doc2query/msmarco-russian-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models. ## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it. ## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.
TAGS
#transformers #pytorch #mt5 #text2text-generation #ru #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 # doc2query/msmarco-russian-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models. ## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it. ## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.
TAGS
#transformers #pytorch #mt5 #text2text-generation #ru #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
# doc2query/msmarco-russian-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.