class_docstr: Algorithm for policy-based reinforcement learning.
  This is a wrapper of algorithms provided in tianshou (PPO by default),
  and can be easily customized with other algorithms that inherit ``BasePolicy``
  (e.g., `REINFORCE <https://link.springer.com/content/pdf/10.1007/BF00992696.pdf>`__
  as in `this paper <https://arxiv.org/abs/1611.01578>`__).
  
  Parameters
  ----------
  samples_per_update
      How many models (trajectories) each time collector collects.
      After each collect, trainer will sample batch from replay buffer and do the update.
  replay_buffer_size
      Size of replay buffer.
      If it's none, the size will be the expected trajectory length times ``samples_per_update``.
  reward_for_invalid
      The reward for a sample that didn't pass validation, or the training doesn't return a metric.
      If not provided, failed models will be simply ignored as if nothing happened.
  policy_fn
      Since environment is created on the fly, the policy needs to be a factory function that creates a policy on-the-fly.
      It takes :class:`~nni.nas.strategy._rl_impl.TuningEnvironment` as input and returns a policy.
      By default, it will use the policy returned by :func:`~nni.nas.strategy._rl_impl.default_policy_fn`.
  update_kwargs
      Keyword arguments for ``policy.update``. See tianshou's BasePolicy for details.
      There is a special key ``"update_times"`` that can be used to specify how many times ``policy.update`` is called,
      which can be used to sufficiently exploit the current available trajectories in the replay buffer
      (for example when actor and critic needs to be updated alternatively multiple times).
      By default, it's ``{'batch_size': 32, 'repeat': 5, 'update_times': 5}``.
class_docstr_tok: ["Algorithm","for","policy-based","reinforcement","learning",".","This","is","a","wrapper","of","algorithms","provided","in","tianshou","(","PPO","by","default",")",",","and","can","be","easily","customized","with","other","algorithms","that","inherit","``","BasePolicy","``","(","e.g.",",","`","REINFORCE","<","https",":","\/\/link.springer.com\/content\/pdf\/10.1007\/BF00992696.pdf",">","`","__","as","in","`","this","paper","<","https",":","\/\/arxiv.org\/abs\/1611.01578",">","`","__",")",".","Parameters","--","--","--","--","--","samples_per_update","How","many","models","(","trajectories",")","each","time","collector","collects",".","After","each","collect",",","trainer","will","sample","batch","from","replay","buffer","and","do","the","update",".","replay_buffer_size","Size","of","replay","buffer",".","If","it","'s","none",",","the","size","will","be","the","expected","trajectory","length","times","``","samples_per_update","``",".","reward_for_invalid","The","reward","for","a","sample","that","did","n't","pass","validation",",","or","the","training","does","n't","return","a","metric",".","If","not","provided",",","failed","models","will","be","simply","ignored","as","if","nothing","happened",".","policy_fn","Since","environment","is","created","on","the","fly",",","the","policy","needs","to","be","a","factory","function","that","creates","a","policy","on-the-fly",".","It","takes",":","class",":","`","~nni.nas.strategy._rl_impl.TuningEnvironment","`","as","input","and","returns","a","policy",".","By","default",",","it","will","use","the","policy","returned","by",":","func",":","`","~nni.nas.strategy._rl_impl.default_policy_fn","`",".","update_kwargs","Keyword","arguments","for","``","policy.update","``",".","See","tianshou","'s","BasePolicy","for","details",".","There","is","a","special","key","``","``","update_times","''","``","that","can","be","used","to","specify","how","many","times","``","policy.update","``","is","called",",","which","can","be","used","to","sufficiently","exploit","the","current","available","trajectories","in","the","replay","buffer","(","for","example","when","actor","and","critic","needs","to","be","updated","alternatively","multiple","times",")",".","By","default",",","it","'s","``","{","'batch_size","'",":","32",",","'repeat","'",":","5",",","'update_times","'",":","5","}","``","."]
code: def _initialize(
      self, model_space: ExecutableModelSpace, engine: ExecutionEngine
  ) -> ExecutableModelSpace:
      generator = TuningTrajectoryGenerator(model_space, self.policy_fn)
      self._policy = generator.policy
  
      if self.replay_buffer_size is None:
          replay_buffer_size = (
              generator.expected_trajectory_length
              * self.samples_per_update
          )
      else:
          replay_buffer_size = self.replay_buffer_size
  
      self._replay_buffer = ReplayBuffer(replay_buffer_size)
  
      return model_space
  
file_path: nni/nni/nas/strategy/rl.py
imports: from __future__ import annotations
  import logging
  import warnings
  from typing import Optional, TYPE_CHECKING
  from nni.mutable import SampleValidationError
  from nni.nas.execution import ExecutionEngine
  from nni.nas.space import ExecutableModelSpace
  from .base import Strategy
label_desc: Use image node_id 6 for calling the PolicyBasedRL obj's underlying member method code with example usage: obj._initialize(model_space, engine) and returns: model_space