class_docstr: Actor to perform distributed training.
class_docstr_tok: ["Actor","to","perform","distributed","training","."]
docstr: Add a hook to the original save_checkpoint function.
  
  This checks if there are new computational resources available.
  If so, raise exception to restart the training process and
  make use of the new resources.
docstr_tok: ["Add","a","hook","to","the","original","save_checkpoint","function",".","This","checks","if","there","are","new","computational","resources","available",".","If","so",",","raise","exception","to","restart","the","training","process","and","make","use","of","the","new","resources","."]
code: def add_checkpoint_hook(self, args):
      """Add a hook to the original save_checkpoint function.
  
      This checks if there are new computational resources available.
      If so, raise exception to restart the training process and
      make use of the new resources.
      """
  
      if args.cpu:
          original_n_cpus = args.distributed_world_size
  
          def _new_save_checkpoint(*args, **kwargs):
              _original_save_checkpoint(*args, **kwargs)
              n_cpus = int(ray.cluster_resources()["CPU"])
              if n_cpus > original_n_cpus:
                  raise Exception(
                      "New CPUs find (original %d CPUs, now %d CPUs)"
                      % (original_n_cpus, n_cpus)
                  )
  
      else:
          original_n_gpus = args.distributed_world_size
  
          def _new_save_checkpoint(*args, **kwargs):
              _original_save_checkpoint(*args, **kwargs)
              n_gpus = int(ray.cluster_resources().get("GPU", 0))
              if n_gpus > original_n_gpus:
                  raise Exception(
                      "New GPUs find (original %d GPUs, now %d GPUs)"
                      % (original_n_gpus, n_gpus)
                  )
  
      fairseq.checkpoint_utils.save_checkpoint = _new_save_checkpoint
  
file_path: ray/doc/source/ray-core/examples/lm/ray_train.py
imports: import copy
  import math
  import socket
  import time
  from contextlib import closing
  import fairseq
  from fairseq import options
  from fairseq_cli.train import main
  import ray
label_desc: Use image node_id 2 for calling the RayDistributedActor obj's underlying member method code with example usage: obj.add_checkpoint_hook(args) without return types