class_docstr: PCL (Path Consistency Learning).
  
  Not only the batch PCL algorithm proposed in the paper but also its
  asynchronous variant is implemented.
  
  See https://arxiv.org/abs/1702.08892
  
  Args:
      model (chainer.Link): Model to train. It must be a callable that
          accepts a batch of observations as input and return two values:
  
          - action distributions (Distribution)
          - state values (chainer.Variable)
      optimizer (chainer.Optimizer): optimizer used to train the model
      t_max (int or None): The model is updated after every t_max local
          steps. If set None, the model is updated after every episode.
      gamma (float): Discount factor [0,1]
      tau (float): Weight coefficient for the entropy regularizaiton term.
      phi (callable): Feature extractor function
      pi_loss_coef (float): Weight coefficient for the loss of the policy
      v_loss_coef (float): Weight coefficient for the loss of the value
          function
      rollout_len (int): Number of rollout steps
      batchsize (int): Number of episodes or sub-trajectories used for an
          update. The total number of transitions used will be
          (batchsize x t_max).
      disable_online_update (bool): If set true, disable online on-policy
          update and rely only on experience replay.
      n_times_replay (int): Number of times experience replay is repeated per
          one time of online update.
      replay_start_size (int): Experience replay is disabled if the number of
          transitions in the replay buffer is lower than this value.
      normalize_loss_by_steps (bool): If set true, losses are normalized by
          the number of steps taken to accumulate the losses
      act_deterministically (bool): If set true, choose most probable actions
          in act method.
      average_loss_decay (float): Decay rate of average loss. Used only
          to record statistics.
      average_entropy_decay (float): Decay rate of average entropy. Used only
          to record statistics.
      average_value_decay (float): Decay rate of average value. Used only
          to record statistics.
      explorer (Explorer or None): If not None, this explorer is used for
          selecting actions.
      logger (None or Logger): Logger to be used
      batch_states (callable): Method which makes a batch of observations.
          default is `chainerrl.misc.batch_states.batch_states`
      backprop_future_values (bool): If set True, value gradients are
          computed not only wrt V(s_t) but also V(s_{t+d}).
      train_async (bool): If set True, use a process-local model to compute
          gradients and update the globally shared model.
code: def __init__(
      self,
      model,
      optimizer,
      replay_buffer=None,
      t_max=None,
      gamma=0.99,
      tau=1e-2,
      phi=lambda x: x,
      pi_loss_coef=1.0,
      v_loss_coef=0.5,
      rollout_len=10,
      batchsize=1,
      disable_online_update=False,
      n_times_replay=1,
      replay_start_size=10**2,
      normalize_loss_by_steps=True,
      act_deterministically=False,
      average_loss_decay=0.999,
      average_entropy_decay=0.999,
      average_value_decay=0.999,
      explorer=None,
      logger=None,
      batch_states=batch_states,
      backprop_future_values=True,
      train_async=False,
  ):
      if train_async:
          # Globally shared model
          self.shared_model = model
  
          # Thread specific model
          self.model = copy.deepcopy(self.shared_model)
          async_.assert_params_not_shared(self.shared_model, self.model)
      else:
          self.model = model
      self.xp = self.model.xp
  
      self.optimizer = optimizer
  
      self.replay_buffer = replay_buffer
      self.t_max = t_max
      self.gamma = gamma
      self.tau = tau
      self.phi = phi
      self.pi_loss_coef = pi_loss_coef
      self.v_loss_coef = v_loss_coef
      self.rollout_len = rollout_len
      if not self.xp.isscalar(batchsize):
          batchsize = self.xp.int32(batchsize)
          """Fix Chainer Issue #2807
  
              batchsize should (look to) be scalar.
              """
      self.batchsize = batchsize
      self.normalize_loss_by_steps = normalize_loss_by_steps
      self.act_deterministically = act_deterministically
      self.disable_online_update = disable_online_update
      self.n_times_replay = n_times_replay
      self.replay_start_size = replay_start_size
      self.average_loss_decay = average_loss_decay
      self.average_value_decay = average_value_decay
      self.average_entropy_decay = average_entropy_decay
      self.logger = logger if logger else getLogger(__name__)
      self.batch_states = batch_states
      self.backprop_future_values = backprop_future_values
      self.train_async = train_async
  
      self.t = 0
      self.last_state = None
      self.last_action = None
      self.explorer = explorer
      self.online_batch_losses = []
  
      # Stats
      self.average_loss = 0
      self.average_value = 0
      self.average_entropy = 0
  
      self.init_history_data_for_online_update()
  
imports: import copy
  from logging import getLogger
  import chainer
  from chainer import functions
  import chainerrl
  from chainerrl import agent
  from chainerrl.agents import a3c
  from chainerrl.misc import async_
  from chainerrl.misc.batch_states import batch_states
  from chainerrl.misc import copy_param
  from chainerrl.recurrent import Recurrent
  from chainerrl.recurrent import state_kept
  from chainerrl.recurrent import state_reset
  from chainerrl.replay_buffer import batch_experiences
label_desc: Use image node_id 1 to create a new PCL object from inherited base classes: agent, agent with example: obj = PCL(model, optimizer, replay_buffer, t_max, gamma, tau, phi, pi_loss_coef, v_loss_coef, rollout_len, batchsize, disable_online_update, n_times_replay, replay_start_size, normalize_loss_by_steps, act_deterministically, average_loss_decay, average_entropy_decay, average_value_decay, explorer, logger, batch_states, backprop_future_values, train_async)