class_docstr: PreemptionCallback class creates a callback that checks for preemption during training at the end of every step.
  Upon preemption the callback provides a function to gracefully exit the training immediately and also saves the current state in a checkpoint as *last.ckpt. 
  (to be able to start from the same step without wasting any compute while resuming the next time).
  
  PreemptionCallback is always enabled by default via the arg create_preemption_callback under ExpManagerConfig. To disable please pass
  create_preemption_callback: False in your config file.
class_docstr_tok: ["PreemptionCallback","class","creates","a","callback","that","checks","for","preemption","during","training","at","the","end","of","every","step",".","Upon","preemption","the","callback","provides","a","function","to","gracefully","exit","the","training","immediately","and","also","saves","the","current","state","in","a","checkpoint","as","*","last.ckpt",".","(","to","be","able","to","start","from","the","same","step","without","wasting","any","compute","while","resuming","the","next","time",")",".","PreemptionCallback","is","always","enabled","by","default","via","the","arg","create_preemption_callback","under","ExpManagerConfig",".","To","disable","please","pass","create_preemption_callback",":","False","in","your","config","file","."]
docstr: Defines custom handlers at the beginning of training to be executed when the 
  preemption signal is received.
docstr_tok: ["Defines","custom","handlers","at","the","beginning","of","training","to","be","executed","when","the","preemption","signal","is","received","."]
code: def on_train_start(self, trainer, pl_module):
      """
      Defines custom handlers at the beginning of training to be executed when the
      preemption signal is received.
      """
  
      # Check if torch distributed is initialised, as its needed for broadcasting the preemption signal to all the ranks
      if not (
          torch.distributed.is_available()
          and torch.distributed.is_initialized()
      ):
          logging.info(
              "Preemption requires torch distributed to be initialized, disabling preemption"
          )
      else:
          self.preemption_enabled = True
          # Bool var that's initialized to false and made True upon receving the preemption signal
          self._interrupted = False
          self.released = False
          self.original_handler = signal.getsignal(self.sig)
  
          # Master handler executed only by rank 0 when the preemption siganal is received, to avoid deadlock conditions
          def master_handler(signum, frame):
              self.release()
              self._interrupted = True
  
          # Handler executed by the non zero ranks
          def ignoring_handler(signum, frame):
              self.release()
  
          self.private_rank = torch.distributed.get_rank()
          if self.private_rank == 0:
              signal.signal(self.sig, master_handler)
          else:
              signal.signal(self.sig, ignoring_handler)
  
      return self
  
file_path: NeMo/nemo/utils/callbacks/preemption.py
imports: import signal
  import sys
  import torch
  from pytorch_lightning.callbacks import Callback
  from nemo.utils import logging
label_desc: Use image node_id 3 for calling the PreemptionCallback obj's underlying member method code with example usage: obj.on_train_start(trainer, pl_module) and returns: self