
# doc2query/msmarco-vietnamese-mt5-base-v1

This is a [doc2query](https://arxiv.org/abs/1904.08375) model based on mT5 (also known as [docT5query](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf)).

It can be used for:
- **Document expansion**: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our [BEIR](https://arxiv.org/abs/2104.08663) paper we showed that BM25+docT5query is a powerful search engine. In the [BEIR repository](https://github.com/beir-cellar/beir) we have an example how to use docT5query with Pyserini.
- **Domain Specific Training Data Generation**: It can be used to generate training data to learn an embedding model. In our [GPL-Paper](https://arxiv.org/abs/2112.07577) / [GPL Example on SBERT.net](https://www.sbert.net/examples/domain_adaptation/README.html#gpl-generative-pseudo-labeling) we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.

## Usage
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_name = 'doc2query/msmarco-vietnamese-mt5-base-v1'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

text = "Python (phát âm tiếng Anh: /ˈpaɪθɑːn/) là một ngôn ngữ lập trình bậc cao cho các mục đích lập trình đa năng, do Guido van Rossum tạo ra và lần đầu ra mắt vào năm 1991. Python được thiết kế với ưu điểm mạnh là dễ đọc, dễ học và dễ nhớ. Python là ngôn ngữ có hình thức rất sáng sủa, cấu trúc rõ ràng, thuận tiện cho người mới học lập trình và là ngôn ngữ lập trình dễ học; được dùng rộng rãi trong phát triển trí tuệ nhân tạo. Cấu trúc của Python còn cho phép người sử dụng viết mã lệnh với số lần gõ phím tối thiểu. Vào tháng 7 năm 2018, van Rossum đã từ chức lãnh đạo trong cộng đồng ngôn ngữ Python sau 30 năm làm việc."


def create_queries(para):
    input_ids = tokenizer.encode(para, return_tensors='pt')
    with torch.no_grad():
        # Here we use top_k / top_k random sampling. It generates more diverse queries, but of lower quality
        sampling_outputs = model.generate(
            input_ids=input_ids,
            max_length=64,
            do_sample=True,
            top_p=0.95,
            top_k=10, 
            num_return_sequences=5
            )
        
        # Here we use Beam-search. It generates better quality queries, but with less diversity
        beam_outputs = model.generate(
            input_ids=input_ids, 
            max_length=64, 
            num_beams=5, 
            no_repeat_ngram_size=2, 
            num_return_sequences=5, 
            early_stopping=True
        )


    print("Paragraph:")
    print(para)
    
    print("\nBeam Outputs:")
    for i in range(len(beam_outputs)):
        query = tokenizer.decode(beam_outputs[i], skip_special_tokens=True)
        print(f'{i + 1}: {query}')

    print("\nSampling Outputs:")
    for i in range(len(sampling_outputs)):
        query = tokenizer.decode(sampling_outputs[i], skip_special_tokens=True)
        print(f'{i + 1}: {query}')

create_queries(text)

```

**Note:** `model.generate()` is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.

## Training
This model fine-tuned [google/mt5-base](https://huggingface.co/google/mt5-base) for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the `train_script.py` in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the [mMARCO dataset](https://github.com/unicamp-dl/mMARCO).




{"language": "vi", "license": "apache-2.0", "datasets": ["unicamp-dl/mmarco"], "widget": [{"text": "Python (ph\u00e1t \u00e2m ti\u1ebfng Anh: /\u02c8pa\u026a\u03b8\u0251\u02d0n/) l\u00e0 m\u1ed9t ng\u00f4n ng\u1eef l\u1eadp tr\u00ecnh b\u1eadc cao cho c\u00e1c m\u1ee5c \u0111\u00edch l\u1eadp tr\u00ecnh \u0111a n\u0103ng, do Guido van Rossum t\u1ea1o ra v\u00e0 l\u1ea7n \u0111\u1ea7u ra m\u1eaft v\u00e0o n\u0103m 1991. Python \u0111\u01b0\u1ee3c thi\u1ebft k\u1ebf v\u1edbi \u01b0u \u0111i\u1ec3m m\u1ea1nh l\u00e0 d\u1ec5 \u0111\u1ecdc, d\u1ec5 h\u1ecdc v\u00e0 d\u1ec5 nh\u1edb. Python l\u00e0 ng\u00f4n ng\u1eef c\u00f3 h\u00ecnh th\u1ee9c r\u1ea5t s\u00e1ng s\u1ee7a, c\u1ea5u tr\u00fac r\u00f5 r\u00e0ng, thu\u1eadn ti\u1ec7n cho ng\u01b0\u1eddi m\u1edbi h\u1ecdc l\u1eadp tr\u00ecnh v\u00e0 l\u00e0 ng\u00f4n ng\u1eef l\u1eadp tr\u00ecnh d\u1ec5 h\u1ecdc; \u0111\u01b0\u1ee3c d\u00f9ng r\u1ed9ng r\u00e3i trong ph\u00e1t tri\u1ec3n tr\u00ed tu\u1ec7 nh\u00e2n t\u1ea1o. C\u1ea5u tr\u00fac c\u1ee7a Python c\u00f2n cho ph\u00e9p ng\u01b0\u1eddi s\u1eed d\u1ee5ng vi\u1ebft m\u00e3 l\u1ec7nh v\u1edbi s\u1ed1 l\u1ea7n g\u00f5 ph\u00edm t\u1ed1i thi\u1ec3u. V\u00e0o th\u00e1ng 7 n\u0103m 2018, van Rossum \u0111\u00e3 t\u1eeb ch\u1ee9c l\u00e3nh \u0111\u1ea1o trong c\u1ed9ng \u0111\u1ed3ng ng\u00f4n ng\u1eef Python sau 30 n\u0103m l\u00e0m vi\u1ec7c."}]}
transformers pytorch mt5 text2text-generation vi dataset:unicamp-dl/mmarco arxiv:1904.08375 arxiv:2104.08663 arxiv:2112.07577 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #mt5 #text2text-generation #vi #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 


# doc2query/msmarco-vietnamese-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.

## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.

## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.




# doc2query/msmarco-vietnamese-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models. ## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it. ## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.
TAGS
#transformers #pytorch #mt5 #text2text-generation #vi #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 # doc2query/msmarco-vietnamese-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models. ## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it. ## Training
This model fine-tuned google/mt5-base for 66k training steps (4 epochs on the 500k training pairs from MS MARCO). For the  training script, see the 'train_script.py' in this repository.

The input-text was truncated to 320 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, passage) from the mMARCO dataset.
passage: TAGS
#transformers #pytorch #mt5 #text2text-generation #vi #dataset-unicamp-dl/mmarco #arxiv-1904.08375 #arxiv-2104.08663 #arxiv-2112.07577 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
# doc2query/msmarco-vietnamese-mt5-base-v1

This is a doc2query model based on mT5 (also known as docT5query).

It can be used for:
- Document expansion: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our BEIR paper we showed that BM25+docT5query is a powerful search engine. In the BEIR repository we have an example how to use docT5query with Pyserini.
- Domain Specific Training Data Generation: It can be used to generate training data to learn an embedding model. In our GPL-Paper / GPL Example on URL we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.## Usage


Note: 'model.generate()' is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.