rllib/agents/ppo/ppo.py
def training(
        self,
        *,
        lr_schedule: Optional[List[List[Union[int, float]]]] = None,
        use_critic: Optional[bool] = None,
        use_gae: Optional[bool] = None,
        lambda_: Optional[float] = None,
        kl_coeff: Optional[float] = None,
        sgd_minibatch_size: Optional[int] = None,
        num_sgd_iter: Optional[int] = None,
        shuffle_sequences: Optional[bool] = None,
        vf_loss_coeff: Optional[float] = None,
        entropy_coeff: Optional[float] = None,
        entropy_coeff_schedule: Optional[List[List[Union[int, float]]]] = None,
        clip_param: Optional[float] = None,
        vf_clip_param: Optional[float] = None,
        grad_clip: Optional[float] = None,
        kl_target: Optional[float] = None,
        **kwargs,
    ) -> "PPOConfig":
        """Sets the training related configuration.

        Args:
            lr_schedule: Learning rate schedule. In the format of
                [[timestep, lr-value], [timestep, lr-value], ...]
                Intermediary timesteps will be assigned to interpolated learning rate
                values. A schedule should normally start from timestep 0.
            use_critic: Should use a critic as a baseline (otherwise don't use value
                baseline; required for using GAE).
            use_gae: If true, use the Generalized Advantage Estimator (GAE)
                with a value function, see https://arxiv.org/pdf/1506.02438.pdf.
            lambda_: The GAE (lambda) parameter.
            kl_coeff: Initial coefficient for KL divergence.
            sgd_minibatch_size: Total SGD batch size across all devices for SGD.
                This defines the minibatch size within each epoch.
            num_sgd_iter: Number of SGD iterations in each outer loop (i.e., number of
                epochs to execute per train batch).
            shuffle_sequences: Whether to shuffle sequences in the batch when training
                (recommended).
            vf_loss_coeff: Coefficient of the value function loss. IMPORTANT: you must
                tune this if you set vf_share_layers=True inside your model's config.
            entropy_coeff: Coefficient of the entropy regularizer.
            entropy_coeff_schedule: Decay schedule for the entropy regularizer.
            clip_param: PPO clip parameter.
            vf_clip_param: Clip param for the value function. Note that this is
                sensitive to the scale of the rewards. If your expected V is large,
                increase this.
            grad_clip: If specified, clip the global norm of gradients by this amount.
            kl_target: Target value for KL divergence.

        Returns:
            This updated TrainerConfig object.
        """
        # Pass kwargs onto super's `training()` method.
        super().training(**kwargs)

        if lr_schedule is not None:
            self.lr_schedule = lr_schedule
        if use_critic is not None:
            self.use_critic = use_critic
        if use_gae is not None:
            self.use_gae = use_gae
        if lambda_ is not None:
            self.lambda_ = lambda_
        if kl_coeff is not None:
            self.kl_coeff = kl_coeff
        if sgd_minibatch_size is not None:
            self.sgd_minibatch_size = sgd_minibatch_size
        if num_sgd_iter is not None:
            self.num_sgd_iter = num_sgd_iter
        if shuffle_sequences is not None:
            self.shuffle_sequences = shuffle_sequences
        if vf_loss_coeff is not None:
            self.vf_loss_coeff = vf_loss_coeff
        if entropy_coeff is not None:
            self.entropy_coeff = entropy_coeff
        if entropy_coeff_schedule is not None:
            self.entropy_coeff_schedule = entropy_coeff_schedule
        if clip_param is not None:
            self.clip_param = clip_param
        if vf_clip_param is not None:
            self.vf_clip_param = vf_clip_param
        if grad_clip is not None:
            self.grad_clip = grad_clip
        if kl_target is not None:
            self.kl_target = kl_target

        return self
Sets the training related configuration.

        Args:
            lr_schedule: Learning rate schedule. In the format of
                [[timestep, lr-value], [timestep, lr-value], ...]
                Intermediary timesteps will be assigned to interpolated learning rate
                values. A schedule should normally start from timestep 0.
            use_critic: Should use a critic as a baseline (otherwise don't use value
                baseline; required for using GAE).
            use_gae: If true, use the Generalized Advantage Estimator (GAE)
                with a value function, see https://arxiv.org/pdf/1506.02438.pdf.
            lambda_: The GAE (lambda) parameter.
            kl_coeff: Initial coefficient for KL divergence.
            sgd_minibatch_size: Total SGD batch size across all devices for SGD.
                This defines the minibatch size within each epoch.
            num_sgd_iter: Number of SGD iterations in each outer loop (i.e., number of
                epochs to execute per train batch).
            shuffle_sequences: Whether to shuffle sequences in the batch when training
                (recommended).
            vf_loss_coeff: Coefficient of the value function loss. IMPORTANT: you must
                tune this if you set vf_share_layers=True inside your model's config.
            entropy_coeff: Coefficient of the entropy regularizer.
            entropy_coeff_schedule: Decay schedule for the entropy regularizer.
            clip_param: PPO clip parameter.
            vf_clip_param: Clip param for the value function. Note that this is
                sensitive to the scale of the rewards. If your expected V is large,
                increase this.
            grad_clip: If specified, clip the global norm of gradients by this amount.
            kl_target: Target value for KL divergence.

        Returns:
            This updated TrainerConfig object.
        
Sets the training related configuration.
Sets the training related configuration .
def training(
        self,
        *,
        lr_schedule: Optional[List[List[Union[int, float]]]] = None,
        use_critic: Optional[bool] = None,
        use_gae: Optional[bool] = None,
        lambda_: Optional[float] = None,
        kl_coeff: Optional[float] = None,
        sgd_minibatch_size: Optional[int] = None,
        num_sgd_iter: Optional[int] = None,
        shuffle_sequences: Optional[bool] = None,
        vf_loss_coeff: Optional[float] = None,
        entropy_coeff: Optional[float] = None,
        entropy_coeff_schedule: Optional[List[List[Union[int, float]]]] = None,
        clip_param: Optional[float] = None,
        vf_clip_param: Optional[float] = None,
        grad_clip: Optional[float] = None,
        kl_target: Optional[float] = None,
        **kwargs,
    ) -> "PPOConfig":
        super().training(**kwargs)
        if lr_schedule is not None:
            self.lr_schedule = lr_schedule
        if use_critic is not None:
            self.use_critic = use_critic
        if use_gae is not None:
            self.use_gae = use_gae
        if lambda_ is not None:
            self.lambda_ = lambda_
        if kl_coeff is not None:
            self.kl_coeff = kl_coeff
        if sgd_minibatch_size is not None:
            self.sgd_minibatch_size = sgd_minibatch_size
        if num_sgd_iter is not None:
            self.num_sgd_iter = num_sgd_iter
        if shuffle_sequences is not None:
            self.shuffle_sequences = shuffle_sequences
        if vf_loss_coeff is not None:
            self.vf_loss_coeff = vf_loss_coeff
        if entropy_coeff is not None:
            self.entropy_coeff = entropy_coeff
        if entropy_coeff_schedule is not None:
            self.entropy_coeff_schedule = entropy_coeff_schedule
        if clip_param is not None:
            self.clip_param = clip_param
        if vf_clip_param is not None:
            self.vf_clip_param = vf_clip_param
        if grad_clip is not None:
            self.grad_clip = grad_clip
        if kl_target is not None:
            self.kl_target = kl_target
        return self
def training ( self , * , lr_schedule : Optional [ List [ List [ Union [ int , float ] ] ] ] = None , use_critic : Optional [ bool ] = None , use_gae : Optional [ bool ] = None , lambda_ : Optional [ float ] = None , kl_coeff : Optional [ float ] = None , sgd_minibatch_size : Optional [ int ] = None , num_sgd_iter : Optional [ int ] = None , shuffle_sequences : Optional [ bool ] = None , vf_loss_coeff : Optional [ float ] = None , entropy_coeff : Optional [ float ] = None , entropy_coeff_schedule : Optional [ List [ List [ Union [ int , float ] ] ] ] = None , clip_param : Optional [ float ] = None , vf_clip_param : Optional [ float ] = None , grad_clip : Optional [ float ] = None , kl_target : Optional [ float ] = None , ** kwargs , ) -> "PPOConfig" : super ( ) . training ( ** kwargs ) if lr_schedule is not None : self . lr_schedule = lr_schedule if use_critic is not None : self . use_critic = use_critic if use_gae is not None : self . use_gae = use_gae if lambda_ is not None : self . lambda_ = lambda_ if kl_coeff is not None : self . kl_coeff = kl_coeff if sgd_minibatch_size is not None : self . sgd_minibatch_size = sgd_minibatch_size if num_sgd_iter is not None : self . num_sgd_iter = num_sgd_iter if shuffle_sequences is not None : self . shuffle_sequences = shuffle_sequences if vf_loss_coeff is not None : self . vf_loss_coeff = vf_loss_coeff if entropy_coeff is not None : self . entropy_coeff = entropy_coeff if entropy_coeff_schedule is not None : self . entropy_coeff_schedule = entropy_coeff_schedule if clip_param is not None : self . clip_param = clip_param if vf_clip_param is not None : self . vf_clip_param = vf_clip_param if grad_clip is not None : self . grad_clip = grad_clip if kl_target is not None : self . kl_target = kl_target return self
Sets the training related configuration.
Sets the training related configuration .
"""Sets the training related configuration.

        Args:
            lr_schedule: Learning rate schedule. In the format of
                [[timestep, lr-value], [timestep, lr-value], ...]
                Intermediary timesteps will be assigned to interpolated learning rate
                values. A schedule should normally start from timestep 0.
            use_critic: Should use a critic as a baseline (otherwise don't use value
                baseline; required for using GAE).
            use_gae: If true, use the Generalized Advantage Estimator (GAE)
                with a value function, see https://arxiv.org/pdf/1506.02438.pdf.
            lambda_: The GAE (lambda) parameter.
            kl_coeff: Initial coefficient for KL divergence.
            sgd_minibatch_size: Total SGD batch size across all devices for SGD.
                This defines the minibatch size within each epoch.
            num_sgd_iter: Number of SGD iterations in each outer loop (i.e., number of
                epochs to execute per train batch).
            shuffle_sequences: Whether to shuffle sequences in the batch when training
                (recommended).
            vf_loss_coeff: Coefficient of the value function loss. IMPORTANT: you must
                tune this if you set vf_share_layers=True inside your model's config.
            entropy_coeff: Coefficient of the entropy regularizer.
            entropy_coeff_schedule: Decay schedule for the entropy regularizer.
            clip_param: PPO clip parameter.
            vf_clip_param: Clip param for the value function. Note that this is
                sensitive to the scale of the rewards. If your expected V is large,
                increase this.
            grad_clip: If specified, clip the global norm of gradients by this amount.
            kl_target: Target value for KL divergence.

        Returns:
            This updated TrainerConfig object.
        """ # Pass kwargs onto super's `training()` method.
This updated TrainerConfig object.
This updated TrainerConfig object .
self
lr_schedule
Optional[List[List[Union[int, float]]]]
Learning rate schedule. In the format of
[[timestep, lr-value], [timestep, lr-value], ...]
Intermediary timesteps will be assigned to interpolated learning rate
values. A schedule should normally start from timestep 0.
Learning rate schedule . In the format of [[ timestep lr - value ] [ timestep lr - value ] ... ] Intermediary timesteps will be assigned to interpolated learning rate values . A schedule should normally start from timestep 0 .
use_critic
Optional[bool]
Should use a critic as a baseline (otherwise don't use value
baseline; required for using GAE).
Should use a critic as a baseline ( otherwise don ' t use value baseline ; required for using GAE ) .
use_gae
Optional[bool]
None
lambda_
Optional[float]
The GAE (lambda) parameter.
The GAE ( lambda ) parameter .
kl_coeff
Optional[float]
Initial coefficient for KL divergence.
Initial coefficient for KL divergence .
sgd_minibatch_size
Optional[int]
Total SGD batch size across all devices for SGD.
This defines the minibatch size within each epoch.
Total SGD batch size across all devices for SGD . This defines the minibatch size within each epoch .
num_sgd_iter
Optional[int]
Number of SGD iterations in each outer loop .
Number of SGD iterations in each outer loop .
shuffle_sequences
Optional[bool]
Whether to shuffle sequences in the batch when training
(recommended).
Whether to shuffle sequences in the batch when training ( recommended ) .
vf_loss_coeff
Optional[float]
Coefficient of the value function loss. IMPORTANT: you must
tune this if you set vf_share_layers=True inside your model's config.
Coefficient of the value function loss . IMPORTANT : you must tune this if you set vf_share_layers = True inside your model ' s config .
entropy_coeff
Optional[float]
Coefficient of the entropy regularizer.
Coefficient of the entropy regularizer .
entropy_coeff_schedule
Optional[List[List[Union[int, float]]]]
Decay schedule for the entropy regularizer.
Decay schedule for the entropy regularizer .
clip_param
Optional[float]
PPO clip parameter.
PPO clip parameter .
vf_clip_param
Optional[float]
Clip param for the value function. Note that this is
sensitive to the scale of the rewards. If your expected V is large,
increase this.
Clip param for the value function . Note that this is sensitive to the scale of the rewards . If your expected V is large increase this .
grad_clip
Optional[float]
If specified, clip the global norm of gradients by this amount.
If specified clip the global norm of gradients by this amount .
kl_target
Optional[float]
Target value for KL divergence.
Target value for KL divergence .