class_docstr: Trust Region Policy Optimization.
  
  A given stochastic policy is optimized by the TRPO algorithm. A given
  value function is also trained to predict by the TD(lambda) algorithm and
  used for Generalized Advantage Estimation (GAE).
  
  Since the policy is optimized via the conjugate gradient method and line
  search while the value function is optimized via SGD, these two models
  should be separate.
  
  Since TRPO requires second-order derivatives to compute Hessian-vector
  products, Chainer v3.0.0 or newer is required. In addition, your policy
  must contain only functions that support second-order derivatives.
  
  See https://arxiv.org/abs/1502.05477 for TRPO.
  See https://arxiv.org/abs/1506.02438 for GAE.
  
  Args:
      policy (Policy): Stochastic policy. Its forward computation must
          contain only functions that support second-order derivatives.
      vf (ValueFunction): Value function.
      vf_optimizer (chainer.Optimizer): Optimizer for the value function.
      obs_normalizer (chainerrl.links.EmpiricalNormalization or None):
          If set to chainerrl.links.EmpiricalNormalization, it is used to
          normalize observations based on the empirical mean and standard
          deviation of observations. These statistics are updated after
          computing advantages and target values and before updating the
          policy and the value function.
      gamma (float): Discount factor [0, 1]
      lambd (float): Lambda-return factor [0, 1]
      phi (callable): Feature extractor function
      entropy_coef (float): Weight coefficient for entropoy bonus [0, inf)
      update_interval (int): Interval steps of TRPO iterations. Every after
          this amount of steps, this agent updates the policy and the value
          function using data from these steps.
      vf_epochs (int): Number of epochs for which the value function is
          trained on each TRPO iteration.
      vf_batch_size (int): Batch size of SGD for the value function.
      standardize_advantages (bool): Use standardized advantages on updates
      line_search_max_backtrack (int): Maximum number of backtracking in line
          search to tune step sizes of policy updates.
      conjugate_gradient_max_iter (int): Maximum number of iterations in
          the conjugate gradient method.
      conjugate_gradient_damping (float): Damping factor used in the
          conjugate gradient method.
      act_deterministically (bool): If set to True, choose most probable
          actions in the act method instead of sampling from distributions.
      value_stats_window (int): Window size used to compute statistics
          of value predictions.
      entropy_stats_window (int): Window size used to compute statistics
          of entropy of action distributions.
      kl_stats_window (int): Window size used to compute statistics
          of KL divergence between old and new policies.
      policy_step_size_stats_window (int): Window size used to compute
          statistics of step sizes of policy updates.
  
  Statistics:
      average_value: Average of value predictions on non-terminal states.
          It's updated before the value function is updated.
      average_entropy: Average of entropy of action distributions on
          non-terminal states. It's updated on act_and_train.
      average_kl: Average of KL divergence between old and new policies.
          It's updated after the policy is updated.
      average_policy_step_size: Average of step sizes of policy updates
          It's updated after the policy is updated.
class_docstr_tok: ["Trust","Region","Policy","Optimization",".","A","given","stochastic","policy","is","optimized","by","the","TRPO","algorithm",".","A","given","value","function","is","also","trained","to","predict","by","the","TD","(","lambda",")","algorithm","and","used","for","Generalized","Advantage","Estimation","(","GAE",")",".","Since","the","policy","is","optimized","via","the","conjugate","gradient","method","and","line","search","while","the","value","function","is","optimized","via","SGD",",","these","two","models","should","be","separate",".","Since","TRPO","requires","second-order","derivatives","to","compute","Hessian-vector","products",",","Chainer","v3.0.0","or","newer","is","required",".","In","addition",",","your","policy","must","contain","only","functions","that","support","second-order","derivatives",".","See","https",":","\/\/arxiv.org\/abs\/1502.05477","for","TRPO",".","See","https",":","\/\/arxiv.org\/abs\/1506.02438","for","GAE",".","Args",":","policy","(","Policy",")",":","Stochastic","policy",".","Its","forward","computation","must","contain","only","functions","that","support","second-order","derivatives",".","vf","(","ValueFunction",")",":","Value","function",".","vf_optimizer","(","chainer.Optimizer",")",":","Optimizer","for","the","value","function",".","obs_normalizer","(","chainerrl.links.EmpiricalNormalization","or","None",")",":","If","set","to","chainerrl.links.EmpiricalNormalization",",","it","is","used","to","normalize","observations","based","on","the","empirical","mean","and","standard","deviation","of","observations",".","These","statistics","are","updated","after","computing","advantages","and","target","values","and","before","updating","the","policy","and","the","value","function",".","gamma","(","float",")",":","Discount","factor","[","0",",","1","]","lambd","(","float",")",":","Lambda-return","factor","[","0",",","1","]","phi","(","callable",")",":","Feature","extractor","function","entropy_coef","(","float",")",":","Weight","coefficient","for","entropoy","bonus","[","0",",","inf",")","update_interval","(","int",")",":","Interval","steps","of","TRPO","iterations",".","Every","after","this","amount","of","steps",",","this","agent","updates","the","policy","and","the","value","function","using","data","from","these","steps",".","vf_epochs","(","int",")",":","Number","of","epochs","for","which","the","value","function","is","trained","on","each","TRPO","iteration",".","vf_batch_size","(","int",")",":","Batch","size","of","SGD","for","the","value","function",".","standardize_advantages","(","bool",")",":","Use","standardized","advantages","on","updates","line_search_max_backtrack","(","int",")",":","Maximum","number","of","backtracking","in","line","search","to","tune","step","sizes","of","policy","updates",".","conjugate_gradient_max_iter","(","int",")",":","Maximum","number","of","iterations","in","the","conjugate","gradient","method",".","conjugate_gradient_damping","(","float",")",":","Damping","factor","used","in","the","conjugate","gradient","method",".","act_deterministically","(","bool",")",":","If","set","to","True",",","choose","most","probable","actions","in","the","act","method","instead","of","sampling","from","distributions",".","value_stats_window","(","int",")",":","Window","size","used","to","compute","statistics","of","value","predictions",".","entropy_stats_window","(","int",")",":","Window","size","used","to","compute","statistics","of","entropy","of","action","distributions",".","kl_stats_window","(","int",")",":","Window","size","used","to","compute","statistics","of","KL","divergence","between","old","and","new","policies",".","policy_step_size_stats_window","(","int",")",":","Window","size","used","to","compute","statistics","of","step","sizes","of","policy","updates",".","Statistics",":","average_value",":","Average","of","value","predictions","on","non-terminal","states",".","It","'s","updated","before","the","value","function","is","updated",".","average_entropy",":","Average","of","entropy","of","action","distributions","on","non-terminal","states",".","It","'s","updated","on","act_and_train",".","average_kl",":","Average","of","KL","divergence","between","old","and","new","policies",".","It","'s","updated","after","the","policy","is","updated",".","average_policy_step_size",":","Average","of","step","sizes","of","policy","updates","It","'s","updated","after","the","policy","is","updated","."]
code: def _update_vf_recurrent(self, dataset):
      for epoch in range(self.vf_epochs):
          random.shuffle(dataset)
          for (
              minibatch
          ) in _yield_subset_of_sequences_with_fixed_number_of_items(  # NOQA
              dataset, self.vf_batch_size
          ):
              self._update_vf_once_recurrent(minibatch)
  
imports: import collections
  import itertools
  from logging import getLogger
  import random
  import chainer
  import chainer.functions
  import numpy
  import chainerrl
  from chainerrl import agent
  from chainerrl.agents.ppo import _compute_explained_variance
  from chainerrl.agents.ppo import _make_dataset
  from chainerrl.agents.ppo import _make_dataset_recurrent
  from chainerrl.agents.ppo import _yield_subset_of_sequences_with_fixed_number_of_items
  from chainerrl.misc.batch_states import batch_states
label_desc: Use image node_id 7 for calling the TRPO obj's underlying member method code with example usage: obj._update_vf_recurrent(dataset) without return types