class_docstr: Training helper that restores from checkpoint and creates session.
  
  This class is a small wrapper that takes care of session creation and
  checkpoint recovery. It also provides functions that to facilitate
  coordination among multiple training threads or processes.
  
  * Checkpointing trained variables as the training progresses.
  * Initializing variables on startup, restoring them from the most recent
    checkpoint after a crash, or wait for checkpoints to become available.
  
  ### Usage:
  
  ```python
  with tf.Graph().as_default():
     ...add operations to the graph...
    # Create a SessionManager that will checkpoint the model in '/tmp/mydir'.
    sm = SessionManager()
    sess = sm.prepare_session(master, init_op, saver, checkpoint_dir)
    # Use the session to train the graph.
    while True:
      sess.run(<my_train_op>)
  ```
  
  `prepare_session()` initializes or restores a model. It requires `init_op`
  and `saver` as an argument.
  
  A second process could wait for the model to be ready by doing the following:
  
  ```python
  with tf.Graph().as_default():
     ...add operations to the graph...
    # Create a SessionManager that will wait for the model to become ready.
    sm = SessionManager()
    sess = sm.wait_for_session(master)
    # Use the session to train the graph.
    while True:
      sess.run(<my_train_op>)
  ```
  
  `wait_for_session()` waits for a model to be initialized by other processes.
class_docstr_tok: ["Training","helper","that","restores","from","checkpoint","and","creates","session",".","This","class","is","a","small","wrapper","that","takes","care","of","session","creation","and","checkpoint","recovery",".","It","also","provides","functions","that","to","facilitate","coordination","among","multiple","training","threads","or","processes",".","*","Checkpointing","trained","variables","as","the","training","progresses",".","*","Initializing","variables","on","startup",",","restoring","them","from","the","most","recent","checkpoint","after","a","crash",",","or","wait","for","checkpoints","to","become","available",".","#","#","#","Usage",":","``","`","python","with","tf.Graph","(",")",".as_default","(",")",":","...","add","operations","to","the","graph","...","#","Create","a","SessionManager","that","will","checkpoint","the","model","in","'\/tmp\/mydir","'",".","sm","=","SessionManager","(",")","sess","=","sm.prepare_session","(","master",",","init_op",",","saver",",","checkpoint_dir",")","#","Use","the","session","to","train","the","graph",".","while","True",":","sess.run","(","<","my_train_op",">",")","``","`","`","prepare_session","(",")","`","initializes","or","restores","a","model",".","It","requires","`","init_op","`","and","`","saver","`","as","an","argument",".","A","second","process","could","wait","for","the","model","to","be","ready","by","doing","the","following",":","``","`","python","with","tf.Graph","(",")",".as_default","(",")",":","...","add","operations","to","the","graph","...","#","Create","a","SessionManager","that","will","wait","for","the","model","to","become","ready",".","sm","=","SessionManager","(",")","sess","=","sm.wait_for_session","(","master",")","#","Use","the","session","to","train","the","graph",".","while","True",":","sess.run","(","<","my_train_op",">",")","``","`","`","wait_for_session","(",")","`","waits","for","a","model","to","be","initialized","by","other","processes","."]
docstr: Creates a `Session`, and tries to restore a checkpoint.
  
  
  Args:
    master: `String` representation of the TensorFlow master to use.
    saver: A `Saver` object used to restore a model.
    checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the
      dir will be used to restore.
    checkpoint_filename_with_path: Full file name path to the checkpoint file.
    wait_for_checkpoint: Whether to wait for checkpoint to become available.
    max_wait_secs: Maximum time to wait for checkpoints to become available.
    config: Optional `ConfigProto` proto used to configure the session.
  
  Returns:
    A pair (sess, is_restored) where 'is_restored' is `True` if
    the session could be restored, `False` otherwise.
  
  Raises:
    ValueError: If both checkpoint_dir and checkpoint_filename_with_path are
      set.
docstr_tok: ["Creates","a","`","Session","`",",","and","tries","to","restore","a","checkpoint",".","Args",":","master",":","`","String","`","representation","of","the","TensorFlow","master","to","use",".","saver",":","A","`","Saver","`","object","used","to","restore","a","model",".","checkpoint_dir",":","Path","to","the","checkpoint","files",".","The","latest","checkpoint","in","the","dir","will","be","used","to","restore",".","checkpoint_filename_with_path",":","Full","file","name","path","to","the","checkpoint","file",".","wait_for_checkpoint",":","Whether","to","wait","for","checkpoint","to","become","available",".","max_wait_secs",":","Maximum","time","to","wait","for","checkpoints","to","become","available",".","config",":","Optional","`","ConfigProto","`","proto","used","to","configure","the","session",".","Returns",":","A","pair","(","sess",",","is_restored",")","where","'is_restored","'","is","`","True","`","if","the","session","could","be","restored",",","`","False","`","otherwise",".","Raises",":","ValueError",":","If","both","checkpoint_dir","and","checkpoint_filename_with_path","are","set","."]
code: def _restore_checkpoint(
      self,
      master: str,
      saver: saver_lib.Saver = None,
      checkpoint_dir: str = None,
      checkpoint_filename_with_path: str = None,
      wait_for_checkpoint=False,
      max_wait_secs=7200,
      config=None,
  ) -> Tuple[session.Session, bool]:
      """Creates a `Session`, and tries to restore a checkpoint.
  
  
      Args:
        master: `String` representation of the TensorFlow master to use.
        saver: A `Saver` object used to restore a model.
        checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the
          dir will be used to restore.
        checkpoint_filename_with_path: Full file name path to the checkpoint file.
        wait_for_checkpoint: Whether to wait for checkpoint to become available.
        max_wait_secs: Maximum time to wait for checkpoints to become available.
        config: Optional `ConfigProto` proto used to configure the session.
  
      Returns:
        A pair (sess, is_restored) where 'is_restored' is `True` if
        the session could be restored, `False` otherwise.
  
      Raises:
        ValueError: If both checkpoint_dir and checkpoint_filename_with_path are
          set.
      """
      self._target = master
  
      # This is required to so that we initialize the TPU device before
      # restoring from checkpoint since we'll be placing variables on the device
      # and TPUInitialize wipes out the memory of the device.
      strategy = distribute_lib.get_strategy()
      if strategy and hasattr(
          strategy.extended, "_experimental_initialize_system"
      ):
          strategy.extended._experimental_initialize_system()  # pylint: disable=protected-access
  
      sess = session.Session(
          self._target, graph=self._graph, config=config
      )
      if checkpoint_dir and checkpoint_filename_with_path:
          raise ValueError(
              "Can not provide both checkpoint_dir and "
              "checkpoint_filename_with_path."
          )
      # If either saver or checkpoint_* is not specified, cannot restore. Just
      # return.
      if not saver or not (
          checkpoint_dir or checkpoint_filename_with_path
      ):
          return sess, False
  
      if checkpoint_filename_with_path:
          _restore_checkpoint_and_maybe_run_saved_model_initializers(
              sess, saver, checkpoint_filename_with_path
          )
          return sess, True
  
      # Waits up until max_wait_secs for checkpoint to become available.
      wait_time = 0
      ckpt = checkpoint_management.get_checkpoint_state(checkpoint_dir)
      while not ckpt or not ckpt.model_checkpoint_path:
          if wait_for_checkpoint and wait_time < max_wait_secs:
              logging.info("Waiting for checkpoint to be available.")
              time.sleep(self._recovery_wait_secs)
              wait_time += self._recovery_wait_secs
              ckpt = checkpoint_management.get_checkpoint_state(
                  checkpoint_dir
              )
          else:
              return sess, False
  
      # Loads the checkpoint.
      _restore_checkpoint_and_maybe_run_saved_model_initializers(
          sess, saver, ckpt.model_checkpoint_path
      )
      saver.recover_last_checkpoints(ckpt.all_model_checkpoint_paths)
      return sess, True
  
imports: import time
  from typing import Optional, Tuple
  import numpy
  from tensorflow.python.checkpoint import checkpoint_management
  from tensorflow.python.client import session
  from tensorflow.python.distribute import distribute_lib
  from tensorflow.python.framework import errors
  from tensorflow.python.framework import ops
  from tensorflow.python.platform import tf_logging
  from tensorflow.python.training import saver
  from tensorflow.python.util.tf_export import tf_export
label_desc: Use image node_id 2 for calling the SessionManager obj's underlying member method code with example usage: obj._restore_checkpoint(master, saver, checkpoint_dir, checkpoint_filename_with_path, wait_for_checkpoint, max_wait_secs, config) and returns: sess,  True, sess,  False, sess,  True, sess,  False