What is the surrogate loss function in imitation learning, and how is it different from the true cost?
A surrogate loss is a loss than you use &quot;instead of&quot;, &quot;in place of&quot;, &quot;as a proxy for&quot; or &quot;as a substitute for&quot; another loss, which is typically the &quot;true&quot; loss.
Surrogate losses are actually common in machine learning (although almost nobody realizes that they are surrogate losses). For example, the empirical risk (which the mean squared error is an instance of) is a surrogate for the expected risk, which is incomputable in almost all cases, given that you do not know the underlying probability distribution. See An Overview of Statistical Learning Theory by V. N. Vapnik for more details. In fact, discussions on generalization arise because of this issue, i.e. you use surrogate losses rather than true losses.
The term &quot;surrogate&quot; is also used in conjunction with the term &quot;model&quot;, i.e. &quot;surrogate model&quot;, for example, in the context of Bayesian optimization, where a Gaussian process is the surrogate model for the unknown model/function that you want to know about, i.e. you use the Gaussian process to approximate the unknown function/model.
Regarding the excerpt you are quoting and your specific concerns, although I didn't read the paper and I am not an expert in imitation learning, let me try to explain what I understand from this excerpt. Essentially, in imitation learning, you use the expert's policy $\pi^*$ to train the agent, rather than letting him just explore and exploit the environment. So, what you know is $\pi^*$ and you can calculate the &quot;loss&quot; between $\pi^*$ and $\pi$ (the current agent's policy), denoted by $l$. However, this loss $l$ that you calculate is not necessarily the &quot;true&quot; loss (i.e. it is a surrogate loss), given that our goal is not really to imitate the &quot;expert&quot; but to learn an optimal policy to behave in the environment. If the goal was to just imitate the &quot;expert&quot;, then $C$ and $l$ would coincide, because, in that case, $l$ would represent the &quot;discrepancy&quot; or &quot;loss&quot; between $\pi$ and the expert's policy $\pi^*$.