scripts/grad_estimators.py
def estimate_grad_forward_fd(self, expl_directions, rewards):
        """
        Forward finite differences gradient estimator, with optional scaling by the stdev of the rewards.

        :param expl_directions: The set of perturbations.
        :param rewards: Rewards obtained from perturbed models: a tensor of size (num_expl_directions + 1).
                        The last element of rewards is the 'baseline', i.e. the reward of the unperturbed
                        model. The rest are rewards of perturbations +epsilon.
        """
        rewards_pos = rewards[:-1]

        if self.use_only_best:
            sort_idx = torch.argsort(rewards_pos, descending=True)
            rewards_pos = rewards_pos[sort_idx][:self.use_only_best]
            expl_directions = expl_directions[sort_idx][:self.use_only_best]

        return (rewards_pos - rewards[-1]) @ expl_directions / (len(rewards_pos) * self.normalizer(rewards_pos))

        Forward finite differences gradient estimator, with optional scaling by the stdev of the rewards.

        :param expl_directions: The set of perturbations.
        :param rewards: Rewards obtained from perturbed models: a tensor of size (num_expl_directions + 1).
                        The last element of rewards is the 'baseline', i.e. the reward of the unperturbed
                        model. The rest are rewards of perturbations +epsilon.
        
Forward finite differences gradient estimator, with optional scaling by the stdev of the rewards.
Forward finite differences gradient estimator with optional scaling by the stdev of the rewards .
def estimate_grad_forward_fd(self, expl_directions, rewards):
        rewards_pos = rewards[:-1]
        if self.use_only_best:
            sort_idx = torch.argsort(rewards_pos, descending=True)
            rewards_pos = rewards_pos[sort_idx][:self.use_only_best]
            expl_directions = expl_directions[sort_idx][:self.use_only_best]
        return (rewards_pos - rewards[-1]) @ expl_directions / (len(rewards_pos) * self.normalizer(rewards_pos))
def estimate_grad_forward_fd ( self , expl_directions , rewards ) : rewards_pos = rewards [ : - 1 ] if self . use_only_best : sort_idx = torch . argsort ( rewards_pos , descending = True ) rewards_pos = rewards_pos [ sort_idx ] [ : self . use_only_best ] expl_directions = expl_directions [ sort_idx ] [ : self . use_only_best ] return ( rewards_pos - rewards [ - 1 ] ) @ expl_directions / ( len ( rewards_pos ) * self . normalizer ( rewards_pos ) )
Forward finite differences gradient estimator, with optional scaling by the stdev of the rewards.
Forward finite differences gradient estimator with optional scaling by the stdev of the rewards .
"""
        Forward finite differences gradient estimator, with optional scaling by the stdev of the rewards.

        :param expl_directions: The set of perturbations.
        :param rewards: Rewards obtained from perturbed models: a tensor of size (num_expl_directions + 1).
                        The last element of rewards is the 'baseline', i.e. the reward of the unperturbed
                        model. The rest are rewards of perturbations +epsilon.
        """
self
expl_directions
The set of perturbations.
The set of perturbations .
rewards
Rewards obtained from perturbed models: a tensor of size (num_expl_directions + 1).
The last element of rewards is the 'baseline', i.e. the reward of the unperturbed
model. The rest are rewards of perturbations +epsilon.
Rewards obtained from perturbed models : a tensor of size ( num_expl_directions + 1 ) . The last element of rewards is the ' baseline ' i . e . the reward of the unperturbed model . The rest are rewards of perturbations + epsilon .