class_docstr: William's episodic REINFORCE.
  
  Args:
      model (Policy): Model to train. It must be a callable that accepts
          observations as input and return action distributions
          (Distribution).
      optimizer (torch.optim.Optimizer): optimizer used to train the model
      gpu (int): GPU device id if not None nor negative
      beta (float): Weight coefficient for the entropy regularizaiton term.
      phi (callable): Feature extractor function
      act_deterministically (bool): If set true, choose most probable actions
          in act method.
      batchsize (int): Number of episodes used for each update
      backward_separately (bool): If set true, call backward separately for
          each episode and accumulate only gradients.
      average_entropy_decay (float): Decay rate of average entropy. Used only
          to record statistics.
      batch_states (callable): Method which makes a batch of observations.
          default is `pfrl.utils.batch_states`
      recurrent (bool): If set to True, `model` is assumed to implement
          `pfrl.nn.Recurrent` and update in a recurrent
          manner.
      max_grad_norm (float or None): Maximum L2 norm of the gradient used for
          gradient clipping. If set to None, the gradient is not clipped.
      logger (logging.Logger): Logger to be used.
code: def _observe_train(self, obs, reward, done, reset):
      self.reward_sequences[-1].append(reward)
      self.t += 1
  
      if done or reset:
          if not done:
              warnings.warn(
                  "Since REINFORCE supports episodic environments only, "
                  "reset=True with done=False will throw away the last episode."
              )
              self.reward_sequences[-1] = []
              self.log_prob_sequences[-1] = []
              self.entropy_sequences[-1] = []
          elif done:
              self.reward_sequences[-1].append(reward)
              if self.backward_separately:
                  self.accumulate_grad()
                  if self.n_backward == self.batchsize:
                      self.update_with_accumulated_grad()
              else:
                  if len(self.reward_sequences) == self.batchsize:
                      self.batch_update()
                  else:
                      # Prepare for the next episode
                      self.reward_sequences.append([])
                      self.log_prob_sequences.append([])
                      self.entropy_sequences.append([])
          self.train_recurrent_states = None
  
imports: import warnings
  from logging import getLogger
  import numpy
  import torch
  import pfrl
  from pfrl import agent
  from pfrl.utils import clip_l2_grad_norm_
  from pfrl.utils.mode_of_distribution import mode_of_distribution
  from pfrl.utils.recurrent import one_step_forward
label_desc: Use image node_id 5 for calling the REINFORCE obj's underlying member method code with example usage: obj._observe_train(obs, reward, done, reset) without return types