class_docstr: Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents
  contents, and it formats them to be used with a RagModel.
  
  Args:
      config ([`RagConfig`]):
          The configuration of the RAG model this Retriever is used with. Contains parameters indicating which
          `Index` to build. You can load your own custom dataset with `config.index_name="custom"` or use a canonical
          one (default) from the datasets library with `config.index_name="wiki_dpr"` for example.
      question_encoder_tokenizer ([`PreTrainedTokenizer`]):
          The tokenizer that was used to tokenize the question. It is used to decode the question and then use the
          generator_tokenizer.
      generator_tokenizer ([`PreTrainedTokenizer`]):
          The tokenizer used for the generator part of the RagModel.
      index ([`~models.rag.retrieval_rag.Index`], optional, defaults to the one defined by the configuration):
          If specified, use this index instead of the one built using the configuration
  
  Examples:
  
  ```python
  >>> # To load the default "wiki_dpr" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact')
  >>> from transformers import RagRetriever
  
  >>> retriever = RagRetriever.from_pretrained(
  ...     "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed"
  ... )
  
  >>> # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py
  >>> from transformers import RagRetriever
  
  >>> dataset = (
  ...     ...
  ... )  # dataset must be a datasets.Datasets object with columns "title", "text" and "embeddings", and it must have a faiss index
  >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", indexed_dataset=dataset)
  
  >>> # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py
  >>> from transformers import RagRetriever
  
  >>> dataset_path = "path/to/my/dataset"  # dataset saved via *dataset.save_to_disk(...)*
  >>> index_path = "path/to/my/index.faiss"  # faiss index saved via *dataset.get_index("embeddings").save(...)*
  >>> retriever = RagRetriever.from_pretrained(
  ...     "facebook/dpr-ctx_encoder-single-nq-base",
  ...     index_name="custom",
  ...     passages_path=dataset_path,
  ...     index_path=index_path,
  ... )
  
  >>> # To load the legacy index built originally for Rag's paper
  >>> from transformers import RagRetriever
  
  >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", index_name="legacy")
  ```
class_docstr_tok: ["Retriever","used","to","get","documents","from","vector","queries",".","It","retrieves","the","documents","embeddings","as","well","as","the","documents","contents",",","and","it","formats","them","to","be","used","with","a","RagModel",".","Args",":","config","(","[","`","RagConfig","`","]",")",":","The","configuration","of","the","RAG","model","this","Retriever","is","used","with",".","Contains","parameters","indicating","which","`","Index","`","to","build",".","You","can","load","your","own","custom","dataset","with","`","config.index_name=","''","custom","''","`","or","use","a","canonical","one","(","default",")","from","the","datasets","library","with","`","config.index_name=","''","wiki_dpr","''","`","for","example",".","question_encoder_tokenizer","(","[","`","PreTrainedTokenizer","`","]",")",":","The","tokenizer","that","was","used","to","tokenize","the","question",".","It","is","used","to","decode","the","question","and","then","use","the","generator_tokenizer",".","generator_tokenizer","(","[","`","PreTrainedTokenizer","`","]",")",":","The","tokenizer","used","for","the","generator","part","of","the","RagModel",".","index","(","[","`","~models.rag.retrieval_rag.Index","`","]",",","optional",",","defaults","to","the","one","defined","by","the","configuration",")",":","If","specified",",","use","this","index","instead","of","the","one","built","using","the","configuration","Examples",":","``","`","python",">",">",">","#","To","load","the","default","``","wiki_dpr","''","dataset","with","21M","passages","from","wikipedia","(","index","name","is","'compressed","'","or","'exact","'",")",">",">",">","from","transformers","import","RagRetriever",">",">",">","retriever","=","RagRetriever.from_pretrained","(","...","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","dataset=","''","wiki_dpr","''",",","index_name=","''","compressed","''","...",")",">",">",">","#","To","load","your","own","indexed","dataset","built","with","the","datasets","library",".","More","info","on","how","to","build","the","indexed","dataset","in","examples\/rag\/use_own_knowledge_dataset.py",">",">",">","from","transformers","import","RagRetriever",">",">",">","dataset","=","(","...","...","...",")","#","dataset","must","be","a","datasets.Datasets","object","with","columns","``","title","''",",","``","text","''","and","``","embeddings","''",",","and","it","must","have","a","faiss","index",">",">",">","retriever","=","RagRetriever.from_pretrained","(","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","indexed_dataset=dataset",")",">",">",">","#","To","load","your","own","indexed","dataset","built","with","the","datasets","library","that","was","saved","on","disk",".","More","info","in","examples\/rag\/use_own_knowledge_dataset.py",">",">",">","from","transformers","import","RagRetriever",">",">",">","dataset_path","=","``","path\/to\/my\/dataset","''","#","dataset","saved","via","*","dataset.save_to_disk","(","...",")","*",">",">",">","index_path","=","``","path\/to\/my\/index.faiss","''","#","faiss","index","saved","via","*","dataset.get_index","(","``","embeddings","''",")",".save","(","...",")","*",">",">",">","retriever","=","RagRetriever.from_pretrained","(","...","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","...","index_name=","''","custom","''",",","...","passages_path=dataset_path",",","...","index_path=index_path",",","...",")",">",">",">","#","To","load","the","legacy","index","built","originally","for","Rag","'s","paper",">",">",">","from","transformers","import","RagRetriever",">",">",">","retriever","=","RagRetriever.from_pretrained","(","``","facebook\/dpr-ctx_encoder-single-nq-base","''",",","index_name=","''","legacy","''",")","``","`"]
docstr: Retrieves documents for specified `question_hidden_states`.
  
  Args:
      question_input_ids (`List[List[int]]`) batch of input ids
      question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`:
          A batch of query vectors to retrieve with.
      prefix (`str`, *optional*):
          The prefix used by the generator's tokenizer.
      n_docs (`int`, *optional*):
          The number of docs retrieved per query.
      return_tensors (`str` or [`~utils.TensorType`], *optional*, defaults to "pt"):
          If set, will return tensors instead of list of python integers. Acceptable values are:
  
          - `'tf'`: Return TensorFlow `tf.constant` objects.
          - `'pt'`: Return PyTorch `torch.Tensor` objects.
          - `'np'`: Return Numpy `np.ndarray` objects.
  
  Returns: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:
  
      - **context_input_ids** -- List of token ids to be fed to a model.
  
        [What are input IDs?](../glossary#input-ids)
  
      - **context_attention_mask** -- List of indices specifying which tokens should be attended to by the model
      (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`).
  
        [What are attention masks?](../glossary#attention-mask)
  
      - **retrieved_doc_embeds** -- List of embeddings of the retrieved documents
      - **doc_ids** -- List of ids of the retrieved documents
code: def __call__(
      self,
      question_input_ids: List[List[int]],
      question_hidden_states: np.ndarray,
      prefix=None,
      n_docs=None,
      return_tensors=None,
  ) -> BatchEncoding:
      """
      Retrieves documents for specified `question_hidden_states`.
  
      Args:
          question_input_ids (`List[List[int]]`) batch of input ids
          question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`:
              A batch of query vectors to retrieve with.
          prefix (`str`, *optional*):
              The prefix used by the generator's tokenizer.
          n_docs (`int`, *optional*):
              The number of docs retrieved per query.
          return_tensors (`str` or [`~utils.TensorType`], *optional*, defaults to "pt"):
              If set, will return tensors instead of list of python integers. Acceptable values are:
  
              - `'tf'`: Return TensorFlow `tf.constant` objects.
              - `'pt'`: Return PyTorch `torch.Tensor` objects.
              - `'np'`: Return Numpy `np.ndarray` objects.
  
      Returns: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:
  
          - **context_input_ids** -- List of token ids to be fed to a model.
  
            [What are input IDs?](../glossary#input-ids)
  
          - **context_attention_mask** -- List of indices specifying which tokens should be attended to by the model
          (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`).
  
            [What are attention masks?](../glossary#attention-mask)
  
          - **retrieved_doc_embeds** -- List of embeddings of the retrieved documents
          - **doc_ids** -- List of ids of the retrieved documents
      """
  
      n_docs = n_docs if n_docs is not None else self.n_docs
      prefix = (
          prefix if prefix is not None else self.config.generator.prefix
      )
      retrieved_doc_embeds, doc_ids, docs = self.retrieve(
          question_hidden_states, n_docs
      )
  
      input_strings = self.question_encoder_tokenizer.batch_decode(
          question_input_ids, skip_special_tokens=True
      )
      context_input_ids, context_attention_mask = self.postprocess_docs(
          docs,
          input_strings,
          prefix,
          n_docs,
          return_tensors=return_tensors,
      )
  
      if self.return_tokenized_docs:
          retrieved_doc_text = []
          retrieved_doc_title = []
  
          for b_idx in range(len(docs)):
              for doc_idx in range(n_docs):
                  retrieved_doc_text.append(
                      docs[b_idx]["text"][doc_idx]
                  )
                  retrieved_doc_title.append(
                      docs[b_idx]["title"][doc_idx]
                  )
  
          tokenized_docs = self.ctx_encoder_tokenizer(
              retrieved_doc_title,
              retrieved_doc_text,
              truncation=True,
              padding="longest",
              return_tensors=return_tensors,
          )
  
          return BatchEncoding(
              {
                  "context_input_ids": context_input_ids,
                  "context_attention_mask": context_attention_mask,
                  "retrieved_doc_embeds": retrieved_doc_embeds,
                  "doc_ids": doc_ids,
                  "tokenized_doc_ids": tokenized_docs["input_ids"],
                  "tokenized_doc_attention_mask": tokenized_docs[
                      "attention_mask"
                  ],
              },
              tensor_type=return_tensors,
          )
  
      else:
          return BatchEncoding(
              {
                  "context_input_ids": context_input_ids,
                  "context_attention_mask": context_attention_mask,
                  "retrieved_doc_embeds": retrieved_doc_embeds,
                  "doc_ids": doc_ids,
              },
              tensor_type=return_tensors,
          )
  
imports: import os
  import pickle
  import time
  from typing import Iterable, List, Optional, Tuple
  import numpy
  from ...tokenization_utils import PreTrainedTokenizer
  from ...tokenization_utils_base import BatchEncoding
  from ...utils import cached_file, is_datasets_available, is_faiss_available, logging, requires_backends, strtobool
  from .configuration_rag import RagConfig
  from .tokenization_rag import RagTokenizer
label_desc: Use image node_id 11 for calling the RagRetriever obj's underlying member method code with example usage: obj.__call__(question_input_ids, question_hidden_states, prefix, n_docs, return_tensors) and returns: BatchEncoding, BatchEncoding