How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?
Loss. In the context of Deep Learning and Deep Reinforcement Learning, &quot;training&quot; is just a fancy word for &quot;optimization&quot;. You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to maximize the expected return - while most of the Deep Learning classification problems are minimizing some loss.
Most of the modern Deep Learning programming frameworks was created with the classification problems in mind, so they call the optimization goals &quot;losses&quot;: tf.keras, pytorch. The good news is that you can easily convert return-maximization problem to loss-minimization problem by just changing the sign of your target function.
Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign.
Value Function. Given an agent's state, Reinforcement Learning tries to find the best agent's action that maximized the expected return. The problem is that the expected return is also unknown. Note on the difference between &quot;reward&quot; and &quot;return&quot;. Rewards $r_t$ are immediate bits of positive/negative reinforcement that the agent receives obtain after taking an action. The &quot;return&quot; $G_t$ is the function of a particular trajectory - it is the discounted sum of future rewards starting from step $t$:
$$G_t = \sum_k \gamma^k r_{t+k+1}$$
The expectation of the return at state $s$ given a policy $\pi$ is called value $V^\pi(s)$:
$$V^\pi(s) = \mathbb{E}_{a\sim\pi}\left[ G_t\right|\left. s_t=s\right]$$
In principle, it is possible to estimate $V^\pi(s)$ by doing Monte Carlo: start from the state $s$ and run many simulations following $\pi$ and average all the returns you've got. For most practical cases this approach is infeasible. Modern Deep Reinforcement Learning algorithms try to reduce computational load by using neural networks to approximate $V(s)$.
Having the value function approximation $V(s)$, one can then try to optimize the policy approximation function $\pi(a|s)$ using the policy gradient update. Such setup: with one approximator for policy and another approximator for value function is called an &quot;Actor-Critic&quot; family of Reinforcement Learning algorithms.
BTW, it doesn't look like your value function &quot;stabilizes&quot; during training. And it also doesn't change too much if you look at the vertical scale of your plot.
Surrogate Loss. In practice, the policy gradient optimization step above suffers from instabilities. The gradient step tends to change the policy too strongly, which makes the approximation error of the value function too large.
To deal with this, the TRPO/PPO algorithms introduce a kind of clipping of the loss gradient that constrains the policy gradient changes from being too large. The new function that produces these clipped/constrained gradients is called a &quot;surrogate loss&quot;.
mean_noise_std. This doesn't look like a standard name for anything, but I suspect that this is a version of PPO with artificial noise added for better exploration of the agent.