class_docstr: A simple out-of-graph Replay Buffer.
  
  Stores transitions, state, action, reward, next_state, terminal (and any
  extra contents specified) in a circular buffer and provides a uniform
  transition sampling function.
  
  When the states consist of stacks of observations storing the states is
  inefficient. This class writes observations and constructs the stacked states
  at sample time.
  
  Attributes:
    add_count: int, counter of how many transitions have been added (including
      the blank ones at the beginning of an episode).
    invalid_range: np.array, an array with the indices of cursor-related invalid
      transitions
    episode_end_indices: set[int], a set of indices corresponding to the
      end of an episode.
docstr: Initializes OutOfGraphReplayBuffer.
  
  Args:
    observation_shape: tuple of ints.
    stack_size: int, number of frames to use in state stack.
    replay_capacity: int, number of transitions to keep in memory.
    batch_size: int.
    update_horizon: int, length of update ('n' in n-step update).
    gamma: float, the discount factor.
    max_sample_attempts: int, the maximum number of attempts allowed to
      get a sample.
    extra_storage_types: list of ReplayElements defining the type of the extra
      contents that will be stored and returned by sample_transition_batch.
    observation_dtype: np.dtype, type of the observations. Defaults to
      np.uint8 for Atari 2600.
    terminal_dtype: np.dtype, type of the terminals. Defaults to np.uint8 for
      Atari 2600.
    action_shape: tuple of ints, the shape for the action vector. Empty tuple
      means the action is a scalar.
    action_dtype: np.dtype, type of elements in the action.
    reward_shape: tuple of ints, the shape of the reward vector. Empty tuple
      means the reward is a scalar.
    reward_dtype: np.dtype, type of elements in the reward.
    checkpoint_duration: int, how many iterations a checkpoint is kept for.
    keep_every: Optional (int or None), keep all checkpoints == 0 % this
      number. Set to None to disable.
  
  Raises:
    ValueError: If replay_capacity is too small to hold at least one
      transition.
docstr_tok: ["Initializes","OutOfGraphReplayBuffer",".","Args",":","observation_shape",":","tuple","of","ints",".","stack_size",":","int",",","number","of","frames","to","use","in","state","stack",".","replay_capacity",":","int",",","number","of","transitions","to","keep","in","memory",".","batch_size",":","int",".","update_horizon",":","int",",","length","of","update","(","'n","'","in","n-step","update",")",".","gamma",":","float",",","the","discount","factor",".","max_sample_attempts",":","int",",","the","maximum","number","of","attempts","allowed","to","get","a","sample",".","extra_storage_types",":","list","of","ReplayElements","defining","the","type","of","the","extra","contents","that","will","be","stored","and","returned","by","sample_transition_batch",".","observation_dtype",":","np.dtype",",","type","of","the","observations",".","Defaults","to","np.uint8","for","Atari","2600.","terminal_dtype",":","np.dtype",",","type","of","the","terminals",".","Defaults","to","np.uint8","for","Atari","2600.","action_shape",":","tuple","of","ints",",","the","shape","for","the","action","vector",".","Empty","tuple","means","the","action","is","a","scalar",".","action_dtype",":","np.dtype",",","type","of","elements","in","the","action",".","reward_shape",":","tuple","of","ints",",","the","shape","of","the","reward","vector",".","Empty","tuple","means","the","reward","is","a","scalar",".","reward_dtype",":","np.dtype",",","type","of","elements","in","the","reward",".","checkpoint_duration",":","int",",","how","many","iterations","a","checkpoint","is","kept","for",".","keep_every",":","Optional","(","int","or","None",")",",","keep","all","checkpoints","==","0","%","this","number",".","Set","to","None","to","disable",".","Raises",":","ValueError",":","If","replay_capacity","is","too","small","to","hold","at","least","one","transition","."]
code: def __init__(
      self,
      observation_shape,
      stack_size,
      replay_capacity,
      batch_size,
      update_horizon=1,
      gamma=0.99,
      max_sample_attempts=1000,
      extra_storage_types=None,
      observation_dtype=np.uint8,
      terminal_dtype=np.uint8,
      action_shape=(),
      action_dtype=np.int32,
      reward_shape=(),
      reward_dtype=np.float32,
      checkpoint_duration=4,
      keep_every=None,
  ):
      """Initializes OutOfGraphReplayBuffer.
  
      Args:
        observation_shape: tuple of ints.
        stack_size: int, number of frames to use in state stack.
        replay_capacity: int, number of transitions to keep in memory.
        batch_size: int.
        update_horizon: int, length of update ('n' in n-step update).
        gamma: float, the discount factor.
        max_sample_attempts: int, the maximum number of attempts allowed to
          get a sample.
        extra_storage_types: list of ReplayElements defining the type of the extra
          contents that will be stored and returned by sample_transition_batch.
        observation_dtype: np.dtype, type of the observations. Defaults to
          np.uint8 for Atari 2600.
        terminal_dtype: np.dtype, type of the terminals. Defaults to np.uint8 for
          Atari 2600.
        action_shape: tuple of ints, the shape for the action vector. Empty tuple
          means the action is a scalar.
        action_dtype: np.dtype, type of elements in the action.
        reward_shape: tuple of ints, the shape of the reward vector. Empty tuple
          means the reward is a scalar.
        reward_dtype: np.dtype, type of elements in the reward.
        checkpoint_duration: int, how many iterations a checkpoint is kept for.
        keep_every: Optional (int or None), keep all checkpoints == 0 % this
          number. Set to None to disable.
  
      Raises:
        ValueError: If replay_capacity is too small to hold at least one
          transition.
      """
      assert isinstance(observation_shape, tuple)
      if replay_capacity < update_horizon + stack_size:
          raise ValueError(
              "There is not enough capacity to cover "
              "update_horizon and stack_size."
          )
  
      logging.info(
          "Creating a %s replay memory with the following parameters:",
          self.__class__.__name__,
      )
      logging.info("\t observation_shape: %s", str(observation_shape))
      logging.info("\t observation_dtype: %s", str(observation_dtype))
      logging.info("\t terminal_dtype: %s", str(terminal_dtype))
      logging.info("\t stack_size: %d", stack_size)
      logging.info("\t replay_capacity: %d", replay_capacity)
      logging.info("\t batch_size: %d", batch_size)
      logging.info("\t update_horizon: %d", update_horizon)
      logging.info("\t gamma: %f", gamma)
      logging.info("\t checkpoint_duration: %d", checkpoint_duration)
      logging.info("\t keep_every: %s", str(keep_every))
  
      self._action_shape = action_shape
      self._action_dtype = action_dtype
      self._reward_shape = reward_shape
      self._reward_dtype = reward_dtype
      self._observation_shape = observation_shape
      self._stack_size = stack_size
      self._state_shape = self._observation_shape + (self._stack_size,)
      self._replay_capacity = replay_capacity
      self._batch_size = batch_size
      self._update_horizon = update_horizon
      self._gamma = gamma
      self._observation_dtype = observation_dtype
      self._terminal_dtype = terminal_dtype
      self._max_sample_attempts = max_sample_attempts
      if extra_storage_types:
          self._extra_storage_types = extra_storage_types
      else:
          self._extra_storage_types = []
      self._create_storage()
      self.add_count = np.array(0)
      self.invalid_range = np.zeros((self._stack_size))
      # When the horizon is > 1, we compute the sum of discounted rewards as a dot
      # product using the precomputed vector <gamma^0, gamma^1, ..., gamma^{n-1}>.
      self._cumulative_discount_vector = np.array(
          [math.pow(self._gamma, n) for n in range(update_horizon)],
          dtype=np.float32,
      )
      self._next_experience_is_episode_start = True
      self.episode_end_indices = set()
      self._checkpoint_duration = checkpoint_duration
      self._keep_every = keep_every
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  import gzip
  import math
  import os
  import pickle
  from absl import logging
  import gin.tf
  import numpy
  import tensorflow
label_desc: Use image node_id 1 to create a new OutOfGraphReplayBuffer object from inherited base classes: object with example: obj = OutOfGraphReplayBuffer(observation_shape, stack_size, replay_capacity, batch_size, update_horizon, gamma, max_sample_attempts, extra_storage_types, observation_dtype, terminal_dtype, action_shape, action_dtype, reward_shape, reward_dtype, checkpoint_duration, keep_every)