class_docstr: A distributed retriever built on top of the ``torch.distributed`` communication package. During training all workers
  initialize their own instance of the retriever, however, only the main worker loads the index into memory. The index is stored
  in cpu memory. The index will also work well in a non-distributed setup.
  
  Args:
      config (:class:`~transformers.RagConfig`):
          The configuration of the RAG model this Retriever is used with. Contains parameters indicating which ``Index`` to build.
      question_encoder_tokenizer (:class:`~transformers.PreTrainedTokenizer`):
          The tokenizer that was used to tokenize the question.
          It is used to decode the question and then use the generator_tokenizer.
      generator_tokenizer (:class:`~transformers.PreTrainedTokenizer`):
          The tokenizer used for the generator part of the RagModel.
      index (:class:`~transformers.models.rag.retrieval_rag.Index`, optional, defaults to the one defined by the configuration):
          If specified, use this index instead of the one built using the configuration
docstr: Retriever initialization function, needs to be called from the training process. The function sets some common parameters
  and environment variables. On top of that, (only) the main process in the process group loads the index into memory.
  
  Args:
      distributed_port (:obj:`int`):
          The port on which the main communication of the training run is carried out. We set the port for retrieval-related
          communication as ``distributed_port + 1``.
docstr_tok: ["Retriever","initialization","function",",","needs","to","be","called","from","the","training","process",".","The","function","sets","some","common","parameters","and","environment","variables",".","On","top","of","that",",","(","only",")","the","main","process","in","the","process","group","loads","the","index","into","memory",".","Args",":","distributed_port","(",":","obj",":","`","int","`",")",":","The","port","on","which","the","main","communication","of","the","training","run","is","carried","out",".","We","set","the","port","for","retrieval-related","communication","as","``","distributed_port","+","1","``","."]
code: def init_retrieval(self, distributed_port: int):
      """
      Retriever initialization function, needs to be called from the training process. The function sets some common parameters
      and environment variables. On top of that, (only) the main process in the process group loads the index into memory.
  
      Args:
          distributed_port (:obj:`int`):
              The port on which the main communication of the training run is carried out. We set the port for retrieval-related
              communication as ``distributed_port + 1``.
      """
  
      logger.info("initializing retrieval")
  
      # initializing a separate process group for retrieval as the default
      # nccl backend doesn't support gather/scatter operations while gloo
      # is too slow to replace nccl for the core gpu communication
      if dist.is_initialized():
          logger.info("dist initialized")
          # needs to be set manually
          os.environ["GLOO_SOCKET_IFNAME"] = self._infer_socket_ifname()
          # avoid clash with the NCCL port
          os.environ["MASTER_PORT"] = str(distributed_port + 1)
          self.process_group = dist.new_group(
              ranks=None, backend="gloo"
          )
  
      # initialize retriever only on the main worker
      if not dist.is_initialized() or self._is_main():
          logger.info("dist not initialized / main")
          self.index.init_index()
  
      # all processes wait untill the retriever is initialized by the main process
      if dist.is_initialized():
          torch.distributed.barrier(group=self.process_group)
  
file_path: transformers/examples/research_projects/rag/distributed_pytorch_retriever.py
imports: import logging
  import os
  from typing import List, Tuple
  import numpy
  import psutil
  import torch
  import torch.distributed
  from transformers import RagRetriever
label_desc: Use image node_id 2 for calling the RagPyTorchDistributedRetriever obj's underlying member method code with example usage: obj.init_retrieval(distributed_port) without return types