loss. For ease of exposition, we will disregard the effect of
the conditioning variablez and consider one dimensional
representations. Denote the representation Eθ (y) by
a random variable Y . The optimal predictor under
equation (1) is thus given by the following functional
expression,
P⋆ (Eθ (x)) = argminP ∥P(Eθ (x)) − Y ∥1
= median(Y |Eθ (x)).
Substituting this expression for the optimal predictor
into the loss function and evaluating the expected gradi-
ent of the encoder gives
∇θ E∥P⋆ (Eθ (x)) − Y ∥1 = ∇θ MAD(Y |Eθ (x)),
where MAD(· |Eθ (x)) is the median absolute deviation
of a random variable conditioned onEθ (x). Thus, in the
case where the predictor is optimal, the encoder must
learn to capture as much information about the video
as possible to minimize the deviation of the target. The
hypothesis is that incorporating an exponential moving
average to compute the representation ofy ensures that
the predictor evolves faster than the encoder and remains
close to optimal, thereby preventing collapse.
3