∇ ( , ) in ( ) is the term responsible for the entropy regularization mechanism that prevents the policy from converging too early, where ( , ) is the entropy of the action probability distribution in state and the hyperparameter determines the extent to which the updating rule depends on entropy regularization relative to the reward: = 0, if the entropy ≥ 0.8, 1, if the entropy < 0.8.