class_docstr: Training helper that restores from checkpoint and creates session.
  
  This class is a small wrapper that takes care of session creation and
  checkpoint recovery. It also provides functions that to facilitate
  coordination among multiple training threads or processes.
  
  * Checkpointing trained variables as the training progresses.
  * Initializing variables on startup, restoring them from the most recent
    checkpoint after a crash, or wait for checkpoints to become available.
  
  ### Usage:
  
  ```python
  with tf.Graph().as_default():
     ...add operations to the graph...
    # Create a SessionManager that will checkpoint the model in '/tmp/mydir'.
    sm = SessionManager()
    sess = sm.prepare_session(master, init_op, saver, checkpoint_dir)
    # Use the session to train the graph.
    while True:
      sess.run(<my_train_op>)
  ```
  
  `prepare_session()` initializes or restores a model. It requires `init_op`
  and `saver` as an argument.
  
  A second process could wait for the model to be ready by doing the following:
  
  ```python
  with tf.Graph().as_default():
     ...add operations to the graph...
    # Create a SessionManager that will wait for the model to become ready.
    sm = SessionManager()
    sess = sm.wait_for_session(master)
    # Use the session to train the graph.
    while True:
      sess.run(<my_train_op>)
  ```
  
  `wait_for_session()` waits for a model to be initialized by other processes.
class_docstr_tok: ["Training","helper","that","restores","from","checkpoint","and","creates","session",".","This","class","is","a","small","wrapper","that","takes","care","of","session","creation","and","checkpoint","recovery",".","It","also","provides","functions","that","to","facilitate","coordination","among","multiple","training","threads","or","processes",".","*","Checkpointing","trained","variables","as","the","training","progresses",".","*","Initializing","variables","on","startup",",","restoring","them","from","the","most","recent","checkpoint","after","a","crash",",","or","wait","for","checkpoints","to","become","available",".","#","#","#","Usage",":","``","`","python","with","tf.Graph","(",")",".as_default","(",")",":","...","add","operations","to","the","graph","...","#","Create","a","SessionManager","that","will","checkpoint","the","model","in","'\/tmp\/mydir","'",".","sm","=","SessionManager","(",")","sess","=","sm.prepare_session","(","master",",","init_op",",","saver",",","checkpoint_dir",")","#","Use","the","session","to","train","the","graph",".","while","True",":","sess.run","(","<","my_train_op",">",")","``","`","`","prepare_session","(",")","`","initializes","or","restores","a","model",".","It","requires","`","init_op","`","and","`","saver","`","as","an","argument",".","A","second","process","could","wait","for","the","model","to","be","ready","by","doing","the","following",":","``","`","python","with","tf.Graph","(",")",".as_default","(",")",":","...","add","operations","to","the","graph","...","#","Create","a","SessionManager","that","will","wait","for","the","model","to","become","ready",".","sm","=","SessionManager","(",")","sess","=","sm.wait_for_session","(","master",")","#","Use","the","session","to","train","the","graph",".","while","True",":","sess.run","(","<","my_train_op",">",")","``","`","`","wait_for_session","(",")","`","waits","for","a","model","to","be","initialized","by","other","processes","."]
docstr: Creates a new `Session` and waits for model to be ready.
  
  Creates a new `Session` on 'master'.  Waits for the model to be
  initialized or recovered from a checkpoint.  It's expected that
  another thread or process will make the model ready, and that this
  is intended to be used by threads/processes that participate in a
  distributed training configuration where a different thread/process
  is responsible for initializing or recovering the model being trained.
  
  NB: The amount of time this method waits for the session is bounded
  by max_wait_secs. By default, this function will wait indefinitely.
  
  Args:
    master: `String` representation of the TensorFlow master to use.
    config: Optional ConfigProto proto used to configure the session.
    max_wait_secs: Maximum time to wait for the session to become available.
  
  Returns:
    A `Session`. May be None if the operation exceeds the timeout
    specified by config.operation_timeout_in_ms.
  
  Raises:
    tf.DeadlineExceededError: if the session is not available after
      max_wait_secs.
docstr_tok: ["Creates","a","new","`","Session","`","and","waits","for","model","to","be","ready",".","Creates","a","new","`","Session","`","on","'master","'",".","Waits","for","the","model","to","be","initialized","or","recovered","from","a","checkpoint",".","It","'s","expected","that","another","thread","or","process","will","make","the","model","ready",",","and","that","this","is","intended","to","be","used","by","threads\/processes","that","participate","in","a","distributed","training","configuration","where","a","different","thread\/process","is","responsible","for","initializing","or","recovering","the","model","being","trained",".","NB",":","The","amount","of","time","this","method","waits","for","the","session","is","bounded","by","max_wait_secs",".","By","default",",","this","function","will","wait","indefinitely",".","Args",":","master",":","`","String","`","representation","of","the","TensorFlow","master","to","use",".","config",":","Optional","ConfigProto","proto","used","to","configure","the","session",".","max_wait_secs",":","Maximum","time","to","wait","for","the","session","to","become","available",".","Returns",":","A","`","Session","`",".","May","be","None","if","the","operation","exceeds","the","timeout","specified","by","config.operation_timeout_in_ms",".","Raises",":","tf.DeadlineExceededError",":","if","the","session","is","not","available","after","max_wait_secs","."]
code: def wait_for_session(
      self, master: str, config=None, max_wait_secs=float("Inf")
  ) -> Optional[session.Session]:
      """Creates a new `Session` and waits for model to be ready.
  
      Creates a new `Session` on 'master'.  Waits for the model to be
      initialized or recovered from a checkpoint.  It's expected that
      another thread or process will make the model ready, and that this
      is intended to be used by threads/processes that participate in a
      distributed training configuration where a different thread/process
      is responsible for initializing or recovering the model being trained.
  
      NB: The amount of time this method waits for the session is bounded
      by max_wait_secs. By default, this function will wait indefinitely.
  
      Args:
        master: `String` representation of the TensorFlow master to use.
        config: Optional ConfigProto proto used to configure the session.
        max_wait_secs: Maximum time to wait for the session to become available.
  
      Returns:
        A `Session`. May be None if the operation exceeds the timeout
        specified by config.operation_timeout_in_ms.
  
      Raises:
        tf.DeadlineExceededError: if the session is not available after
          max_wait_secs.
      """
      self._target = master
  
      if max_wait_secs is None:
          max_wait_secs = float("Inf")
      timer = _CountDownTimer(max_wait_secs)
  
      while True:
          sess = session.Session(
              self._target, graph=self._graph, config=config
          )
          not_ready_msg = None
          not_ready_local_msg = None
          (
              local_init_success,
              not_ready_local_msg,
          ) = self._try_run_local_init_op(sess)
          if local_init_success:
              # Successful if local_init_op is None, or ready_for_local_init_op passes
              is_ready, not_ready_msg = self._model_ready(sess)
              if is_ready:
                  return sess
  
          self._safe_close(sess)
  
          # Do we have enough time left to try again?
          remaining_ms_after_wait = (
              timer.secs_remaining() - self._recovery_wait_secs
          )
          if remaining_ms_after_wait < 0:
              raise errors.DeadlineExceededError(
                  None,
                  None,
                  "Session was not ready after waiting %d secs."
                  % (max_wait_secs,),
              )
  
          logging.info(
              "Waiting for model to be ready.  "
              "Ready_for_local_init_op:  %s, ready: %s",
              not_ready_local_msg,
              not_ready_msg,
          )
          time.sleep(self._recovery_wait_secs)
  
imports: import time
  from typing import Optional, Tuple
  import numpy
  from tensorflow.python.checkpoint import checkpoint_management
  from tensorflow.python.client import session
  from tensorflow.python.distribute import distribute_lib
  from tensorflow.python.framework import errors
  from tensorflow.python.framework import ops
  from tensorflow.python.platform import tf_logging
  from tensorflow.python.training import saver
  from tensorflow.python.util.tf_export import tf_export
label_desc: Use image node_id 5 for calling the SessionManager obj's underlying member method code with example usage: obj.wait_for_session(master, config, max_wait_secs) and returns: sess