class_docstr: Parallelformers end-point function
  
  Args:
      model (nn.Module): Huggingface pre-trained transformer model.
      fp16: (bool): whether use FP16 or not.
      num_gpus (int): number of GPU for parallelization.
      master_addr (str): master process address for process communication (default='127.0.0.1')
      master_port (int): master process port for process communication (default=29500)
      backend (str): distributed backend (default='nccl')
      verbose (str): logging current gpu states (one of ['detail', 'simple', None]
      init_method (str): multiprocess initialization method. (It is safe to set `init_method` to `spawn`.)
      daemon (bool): whether make process daemon or not (default=True)
  
  Notes:
      We want to use this object as a simple function rather than a class.
      So we broke the PEP8 rules and set class names that start with a lowercase letter.
  
  Examples:
      >>> # 1. Import Huggingface and Parallelformers modules
      >>> from transformers import AutoModelForCausalLM, AutoTokenizer
      >>> from parallelformers import parallelize
  
      >>> # 2. Create Huggingface model and tokenizer.
      >>> model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-2.7B")
      >>> tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-2.7B")
  
      >>> # 3. Parallelize using Parallelformers.
      >>> parallelize(model, num_gpus=4, fp16=True)
  
      >>> # 4. Do inference as usual.
      >>> inputs = tokenizer("Parallelformers is", return_tensors="pt")
      >>> outputs = model.generate(**inputs, num_beams=5, no_repeat_ngram_size=4)
      >>> print(f"Output: {tokenizer.batch_decode(outputs)[0]}")
      'Output: Parallelformers is an open-source library for parallel programming ...'
code: def __init__(
      self,
      model: nn.Module,
      fp16: bool,
      num_gpus: int,
      custom_policies=None,
      master_addr: str = "127.0.0.1",
      master_port: int = 29500,
      backend="nccl",
      verbose: str = None,
      init_method: str = "spawn",
      daemon: bool = True,
      seed: int = None,
  ):
      self.init_environments(
          num_gpus,
          master_addr,
          master_port,
      )
  
      # Using fork often leads to deadlock.
      if mp.get_start_method() != init_method:
          with suppress(Exception):
              mp.set_start_method(init_method, force=True)
  
      for param in model.parameters():
          param.requires_grad = False
          param.detach()
  
      self.preprocess_for_wav2vec(model)
      self.model = model.half() if fp16 else model
      self.model.eval()
      self.fp16 = fp16
      self.num_gpus = num_gpus
      self.backend = backend
      self.daemon = daemon
      self.verbose = verbose
      self.seed = seed
      self.custom_policies = custom_policies
  
      self.processes = []
      self.parallel_mutexes = []
      self.inference_mutexes = []
      self.inputs_queues = []
      self.outputs_queues = []
      self.orig_methods = {}
  
      hijack_methods = ["generate", "forward", "to", "cpu", "cuda"]
  
      for attr in hijack_methods:
          if hasattr(self.model, attr):
              fn = getattr(self.model, attr)
              self.orig_methods[attr] = fn
  
      self.parallelize()
      for attr in hijack_methods:
          if hasattr(self.model, attr):
              self.register_hijack_methods(attr)
  
      self._memory_logger = [
          "memory_reserved",
          "memory_allocated",
          "memory_cached",
      ]
  
      for attr in self._memory_logger:
          self.register_memory_methods(attr)
  
imports: import os
  import traceback
  from contextlib import suppress
  from dataclasses import _is_dataclass_instance
  from typing import Any, Dict
  import torch
  import torch.multiprocessing
  from dacite import Config, from_dict
  from torch import nn
  from transformers.file_utils import ModelOutput
  from parallelformers.parallel.process import ParallelProcess
  from parallelformers.utils import rgetattr, rsetattr
label_desc: Use image node_id 1 to create a new parallelize object from inherited base classes: object with example: obj = parallelize(model, fp16, num_gpus, custom_policies, master_addr, master_port, backend, verbose, init_method, daemon, seed)