class_docstr: William's episodic REINFORCE.
  
  Args:
      model (Policy): Model to train. It must be a callable that accepts
          observations as input and return action distributions
          (Distribution).
      optimizer (torch.optim.Optimizer): optimizer used to train the model
      gpu (int): GPU device id if not None nor negative
      beta (float): Weight coefficient for the entropy regularizaiton term.
      phi (callable): Feature extractor function
      act_deterministically (bool): If set true, choose most probable actions
          in act method.
      batchsize (int): Number of episodes used for each update
      backward_separately (bool): If set true, call backward separately for
          each episode and accumulate only gradients.
      average_entropy_decay (float): Decay rate of average entropy. Used only
          to record statistics.
      batch_states (callable): Method which makes a batch of observations.
          default is `pfrl.utils.batch_states`
      recurrent (bool): If set to True, `model` is assumed to implement
          `pfrl.nn.Recurrent` and update in a recurrent
          manner.
      max_grad_norm (float or None): Maximum L2 norm of the gradient used for
          gradient clipping. If set to None, the gradient is not clipped.
      logger (logging.Logger): Logger to be used.
code: def __init__(
      self,
      model,
      optimizer,
      gpu=None,
      beta=0,
      phi=lambda x: x,
      batchsize=1,
      act_deterministically=False,
      average_entropy_decay=0.999,
      backward_separately=False,
      batch_states=pfrl.utils.batch_states,
      recurrent=False,
      max_grad_norm=None,
      logger=None,
  ):
      self.model = model
      if gpu is not None and gpu >= 0:
          assert torch.cuda.is_available()
          self.device = torch.device("cuda:{}".format(gpu))
          self.model.to(self.device)
      else:
          self.device = torch.device("cpu")
      self.optimizer = optimizer
      self.beta = beta
      self.phi = phi
      self.batchsize = batchsize
      self.backward_separately = backward_separately
      self.act_deterministically = act_deterministically
      self.average_entropy_decay = average_entropy_decay
      self.batch_states = batch_states
      self.recurrent = recurrent
      self.max_grad_norm = max_grad_norm
      self.logger = logger or getLogger(__name__)
  
      # Recurrent states of the model
      self.train_recurrent_states = None
      self.test_recurrent_states = None
  
      # Statistics
      self.average_entropy = 0
  
      self.t = 0
      self.reward_sequences = [[]]
      self.log_prob_sequences = [[]]
      self.entropy_sequences = [[]]
      self.n_backward = 0
  
imports: import warnings
  from logging import getLogger
  import numpy
  import torch
  import pfrl
  from pfrl import agent
  from pfrl.utils import clip_l2_grad_norm_
  from pfrl.utils.mode_of_distribution import mode_of_distribution
  from pfrl.utils.recurrent import one_step_forward
label_desc: Use image node_id 1 to create a new REINFORCE object from inherited base classes: agent, agent with example: obj = REINFORCE(model, optimizer, gpu, beta, phi, batchsize, act_deterministically, average_entropy_decay, backward_separately, batch_states, recurrent, max_grad_norm, logger)