class_docstr: Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents
  contents, and it formats them to be used with a RagModel.
  
  Args:
      config ([`RagConfig`]):
          The configuration of the RAG model this Retriever is used with. Contains parameters indicating which
          `Index` to build. You can load your own custom dataset with `config.index_name="custom"` or use a canonical
          one (default) from the datasets library with `config.index_name="wiki_dpr"` for example.
      question_encoder_tokenizer ([`PreTrainedTokenizer`]):
          The tokenizer that was used to tokenize the question. It is used to decode the question and then use the
          generator_tokenizer.
      generator_tokenizer ([`PreTrainedTokenizer`]):
          The tokenizer used for the generator part of the RagModel.
      index ([`~models.rag.retrieval_rag.Index`], optional, defaults to the one defined by the configuration):
          If specified, use this index instead of the one built using the configuration
  
  Examples:
  
  ```python
  >>> # To load the default "wiki_dpr" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact')
  >>> from transformers import RagRetriever
  
  >>> retriever = RagRetriever.from_pretrained(
  ...     "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed"
  ... )
  
  >>> # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py
  >>> from transformers import RagRetriever
  
  >>> dataset = (
  ...     ...
  ... )  # dataset must be a datasets.Datasets object with columns "title", "text" and "embeddings", and it must have a faiss index
  >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", indexed_dataset=dataset)
  
  >>> # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py
  >>> from transformers import RagRetriever
  
  >>> dataset_path = "path/to/my/dataset"  # dataset saved via *dataset.save_to_disk(...)*
  >>> index_path = "path/to/my/index.faiss"  # faiss index saved via *dataset.get_index("embeddings").save(...)*
  >>> retriever = RagRetriever.from_pretrained(
  ...     "facebook/dpr-ctx_encoder-single-nq-base",
  ...     index_name="custom",
  ...     passages_path=dataset_path,
  ...     index_path=index_path,
  ... )
  
  >>> # To load the legacy index built originally for Rag's paper
  >>> from transformers import RagRetriever
  
  >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", index_name="legacy")
  ```
class_docstr_tok: ["Retriever","used","to","get","documents","from","vector","queries",".","It","retrieves","the","documents","embeddings","as","well","as","the","documents","contents",",","and","it","formats","them","to","be","used","with","a","RagModel",".","Args",":","config","(","[","`","RagConfig","`","]",")",":","The","configuration","of","the","RAG","model","this","Retriever","is","used","with",".","Contains","parameters","indicating","which","`","Index","`","to","build",".","You","can","load","your","own","custom","dataset","with","`","config.index_name=","''","custom","''","`","or","use","a","canonical","one","(","default",")","from","the","datasets","library","with","`","config.index_name=","''","wiki_dpr","''","`","for","example",".","question_encoder_tokenizer","(","[","`","PreTrainedTokenizer","`","]",")",":","The","tokenizer","that","was","used","to","tokenize","the","question",".","It","is","used","to","decode","the","question","and","then","use","the","generator_tokenizer",".","generator_tokenizer","(","[","`","PreTrainedTokenizer","`","]",")",":","The","tokenizer","used","for","the","generator","part","of","the","RagModel",".","index","(","[","`","~models.rag.retrieval_rag.Index","`","]",",","optional",",","defaults","to","the","one","defined","by","the","configuration",")",":","If","specified",",","use","this","index","instead","of","the","one","built","using","the","configuration","Examples",":","``","`","python",">",">",">","#","To","load","the","default","``","wiki_dpr","''","dataset","with","21M","passages","from","wikipedia","(","index","name","is","'compressed","'","or","'exact","'",")",">",">",">","from","transformers","import","RagRetriever",">",">",">","retriever","=","RagRetriever.from_pretrained","(","...","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","dataset=","''","wiki_dpr","''",",","index_name=","''","compressed","''","...",")",">",">",">","#","To","load","your","own","indexed","dataset","built","with","the","datasets","library",".","More","info","on","how","to","build","the","indexed","dataset","in","examples\/rag\/use_own_knowledge_dataset.py",">",">",">","from","transformers","import","RagRetriever",">",">",">","dataset","=","(","...","...","...",")","#","dataset","must","be","a","datasets.Datasets","object","with","columns","``","title","''",",","``","text","''","and","``","embeddings","''",",","and","it","must","have","a","faiss","index",">",">",">","retriever","=","RagRetriever.from_pretrained","(","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","indexed_dataset=dataset",")",">",">",">","#","To","load","your","own","indexed","dataset","built","with","the","datasets","library","that","was","saved","on","disk",".","More","info","in","examples\/rag\/use_own_knowledge_dataset.py",">",">",">","from","transformers","import","RagRetriever",">",">",">","dataset_path","=","``","path\/to\/my\/dataset","''","#","dataset","saved","via","*","dataset.save_to_disk","(","...",")","*",">",">",">","index_path","=","``","path\/to\/my\/index.faiss","''","#","faiss","index","saved","via","*","dataset.get_index","(","``","embeddings","''",")",".save","(","...",")","*",">",">",">","retriever","=","RagRetriever.from_pretrained","(","...","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","...","index_name=","''","custom","''",",","...","passages_path=dataset_path",",","...","index_path=index_path",",","...",")",">",">",">","#","To","load","the","legacy","index","built","originally","for","Rag","'s","paper",">",">",">","from","transformers","import","RagRetriever",">",">",">","retriever","=","RagRetriever.from_pretrained","(","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","index_name=","''","legacy","''",")","``","`"]
docstr: Retriever initialization function. It loads the index into memory.
code: def init_retrieval(self):
      """
      Retriever initialization function. It loads the index into memory.
      """
  
      logger.info("initializing retrieval")
      self.index.init_index()
  
imports: import os
  import pickle
  import time
  from typing import Iterable, List, Optional, Tuple
  import numpy
  from ...tokenization_utils import PreTrainedTokenizer
  from ...tokenization_utils_base import BatchEncoding
  from ...utils import cached_file, is_datasets_available, is_faiss_available, logging, requires_backends, strtobool
  from .configuration_rag import RagConfig
  from .tokenization_rag import RagTokenizer
label_desc: Use image node_id 5 for calling the RagRetriever obj's underlying member method code with example usage: obj.init_retrieval() without return types