class_docstr: Abstract class for trainable models, functions, etc.
  
  A call to ``train()`` on a trainable will execute one logical iteration of
  training. As a rule of thumb, the execution time of one train call should
  be large enough to avoid overheads (i.e. more than a few seconds), but
  short enough to report progress periodically (i.e. at most a few minutes).
  
  Calling ``save()`` should save the training state of a trainable to disk,
  and ``restore(path)`` should restore a trainable to the given state.
  
  Generally you only need to implement ``setup``, ``step``,
  ``save_checkpoint``, and ``load_checkpoint`` when subclassing Trainable.
  
  Other implementation methods that may be helpful to override are
  ``log_result``, ``reset_config``, ``cleanup``, and ``_export_model``.
  
  Tune will convert this class into a Ray actor, which runs on a separate process.
  By default, Tune will also change the current working directory of this process to
  its corresponding trial-level log directory ``self.logdir``.
  This is designed so that different trials that run on the same physical node won't
  accidentally write to the same location and overstep each other.
  
  The behavior of changing the working directory can be disabled by setting the
  `RAY_CHDIR_TO_TRIAL_DIR=0` environment variable. This allows access to files
  in the original working directory, but relative paths should be used for read only
  purposes, and you must make sure that the directory is synced on all nodes if
  running on multiple machines.
  
  The `TUNE_ORIG_WORKING_DIR` environment variable was the original workaround for
  accessing paths relative to the original working directory. This environment
  variable is deprecated, and the `RAY_CHDIR_TO_TRIAL_DIR` environment variable
  described above should be used instead.
  
  This class supports checkpointing to and restoring from remote storage.
docstr: Restores training state from a given model checkpoint.
  
  These checkpoints are returned from calls to save().
  
  Subclasses should override ``load_checkpoint()`` instead to
  restore state.
  This method restores additional metadata saved with the checkpoint.
  
  `checkpoint_path` should match with the return from ``save()``.
  
  Args:
      checkpoint_path: Path to restore checkpoint from. If this
          path does not exist on the local node, it will be fetched
          from external (cloud) storage if available, or restored
          from a remote node.
      checkpoint_node_ip: If given, try to restore
          checkpoint from this node if it doesn't exist locally or
          on cloud storage.
      fallback_to_latest: If True, will try to recover the
          latest available checkpoint if the given ``checkpoint_path``
          could not be found.
docstr_tok: ["Restores","training","state","from","a","given","model","checkpoint",".","These","checkpoints","are","returned","from","calls","to","save","(",")",".","Subclasses","should","override","``","load_checkpoint","(",")","``","instead","to","restore","state",".","This","method","restores","additional","metadata","saved","with","the","checkpoint",".","`","checkpoint_path","`","should","match","with","the","return","from","``","save","(",")","``",".","Args",":","checkpoint_path",":","Path","to","restore","checkpoint","from",".","If","this","path","does","not","exist","on","the","local","node",",","it","will","be","fetched","from","external","(","cloud",")","storage","if","available",",","or","restored","from","a","remote","node",".","checkpoint_node_ip",":","If","given",",","try","to","restore","checkpoint","from","this","node","if","it","does","n't","exist","locally","or","on","cloud","storage",".","fallback_to_latest",":","If","True",",","will","try","to","recover","the","latest","available","checkpoint","if","the","given","``","checkpoint_path","``","could","not","be","found","."]
code: def restore(
      self, checkpoint_path: Union[str, Checkpoint, _TrainingResult]
  ):
      """Restores training state from a given model checkpoint.
  
      These checkpoints are returned from calls to save().
  
      Subclasses should override ``load_checkpoint()`` instead to
      restore state.
      This method restores additional metadata saved with the checkpoint.
  
      `checkpoint_path` should match with the return from ``save()``.
  
      Args:
          checkpoint_path: Path to restore checkpoint from. If this
              path does not exist on the local node, it will be fetched
              from external (cloud) storage if available, or restored
              from a remote node.
          checkpoint_node_ip: If given, try to restore
              checkpoint from this node if it doesn't exist locally or
              on cloud storage.
          fallback_to_latest: If True, will try to recover the
              latest available checkpoint if the given ``checkpoint_path``
              could not be found.
  
      """
      # TODO(justinvyu): Clean up this interface
      if isinstance(checkpoint_path, str):
          checkpoint_path = Checkpoint.from_directory(checkpoint_path)
      if isinstance(checkpoint_path, Checkpoint):
          checkpoint_result = _TrainingResult(
              checkpoint=checkpoint_path, metrics={}
          )
      else:
          checkpoint_result: _TrainingResult = checkpoint_path
  
      assert isinstance(checkpoint_result, _TrainingResult), type(
          checkpoint_result
      )
      checkpoint = checkpoint_result.checkpoint
      checkpoint_metrics = checkpoint_result.metrics
      self._iteration = checkpoint_metrics.get(TRAINING_ITERATION, 0)
      self._time_total = checkpoint_metrics.get(TIME_TOTAL_S, 0)
      self._time_since_restore = 0.0
      self._iterations_since_restore = 0
  
      # TODO(justinvyu): This stuff should be moved to rllib.
      self._timesteps_total = checkpoint_metrics.get(TIMESTEPS_TOTAL)
      self._timesteps_since_restore = 0
      self._episodes_total = checkpoint_metrics.get(EPISODES_TOTAL)
  
      if not _exists_at_fs_path(checkpoint.filesystem, checkpoint.path):
          raise ValueError(
              f"Could not recover from checkpoint as it does not exist on "
              f"storage anymore. "
              f"Got storage fs type `{checkpoint.filesystem.type_name}` and "
              f"path: {checkpoint.path}"
          )
  
      # TODO(justinvyu): [cls_trainable_support]
      # This is to conform to the public class Trainable `load_checkpoint` API.
      if not isinstance(self, ray.tune.trainable.FunctionTrainable):
          # Need to convert Checkpoint -> local path or dict
          # (depending on what the output of save_checkpoint was)
          with checkpoint.as_directory() as checkpoint_dir:
              checkpoint_path = Path(checkpoint_dir)
              dict_checkpoint_file = (
                  checkpoint_path / _DICT_CHECKPOINT_FILE_NAME
              )
              if dict_checkpoint_file.exists():
                  # If this was a dict checkpoint, load it as a dict
                  with open(dict_checkpoint_file, "rb") as f:
                      checkpoint_dict = ray_pickle.load(f)
                  self.load_checkpoint(checkpoint_dict)
              else:
                  self.load_checkpoint(checkpoint_dir)
      else:
          # TODO(justinvyu): The Function Trainable case doesn't conform
          # to the load_checkpoint API at the moment.
          self.load_checkpoint(checkpoint_result)
  
      self._restored = True
  
      logger.info(
          f"Restored on {self._local_ip} from checkpoint: {checkpoint}"
      )
  
imports: import copy
  from datetime import datetime
  import logging
  import os
  from pathlib import Path
  import platform
  import sys
  import tempfile
  import time
  from contextlib import redirect_stderr, redirect_stdout
  from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
  import ray
  import ray.cloudpickle
  from ray.air._internal.util import skip_exceptions, exception_cause
  from ray.air.constants import TIMESTAMP, TIME_THIS_ITER_S, TRAINING_ITERATION
  from ray.train._internal.checkpoint_manager import _TrainingResult
  from ray.train._internal.storage import StorageContext, _exists_at_fs_path
  from ray.train import Checkpoint
  from ray.tune.result import DEBUG_METRICS, DEFAULT_RESULTS_DIR, DONE, EPISODES_THIS_ITER, EPISODES_TOTAL, HOSTNAME, NODE_IP, PID, RESULT_DUPLICATE, SHOULD_CHECKPOINT, STDERR_FILE, STDOUT_FILE, TIME_TOTAL_S, TIMESTEPS_THIS_ITER, TIMESTEPS_TOTAL, TRIAL_ID, TRIAL_INFO
  from ray.tune.utils import UtilMonitor
  from ray.tune.utils.log import disable_ipython
  from ray.tune.utils.util import Tee
  from ray.tune.execution.placement_groups import PlacementGroupFactory
  from ray.util.annotations import DeveloperAPI, PublicAPI
label_desc: Use image node_id 12 for calling the Trainable obj's underlying member method code with example usage: obj.restore(checkpoint_path) without return types