\section{Introduction}
Generative modeling can be thought of as inverting an inference process.
If the inference process is invertible, then one can focus on transforming the data into a tractable distribution \citep{dinh2016density}.
If the inference process is deterministic yet non-invertible, one could learn to invert it stochastically \citep{dinh2019rad, nielsen2020survae}.
Most generally, both inference and generation can be stochastic.
This is known as the variational autoencoder \citep[VAE]{kingma2014auto, rezende2014stochastic}.

Under the variational framework, one has a lot of flexibility in choosing the generative and inference models.
Recent work on diffusion-based modeling \citep{sohl2015deep, ho2020denoising} can be thought of as removing one degree of freedom, by freezing the inference path.
The inference model is a fixed discrete-time Markov chain, that slowly transforms the data into a tractable prior, such as the standard normal distribution.
The generative model is another Markov chain that is trained to revert this process iteratively.
Diffusion-based models have been shown to perform remarkably well on image synthesis \citep{dhariwal2021diffusion}, rivaling the performance of state-of-the-art Generative Adversarial Networks \citep{brock2018large}.

\citet{song2021score} connect diffusion-based model and \emph{score matching} \citep{hyvarinen2005estimation}, by looking at the stochastic differential equation (SDE) associated with the inference process.
They realize that the dynamic of the inference process can be inverted if one has access to the score function of the perturbed data, by solving another SDE reversed in time.
They then propose to learn the score function of the inference process and substitute the approximate score into the formula of the reverse SDE to obtain a generative model.
We call the resulting generative model the plug-in reverse SDE.

Conceptually simple as this learning procedure may seem, little is known about how the score matching loss relates to the plug-in reverse SDE.
In this paper, we propose a variational framework suitable for likelihood estimation for general generative diffusion processes, and use this framework to connect score matching with maximum likelihood.
We do so by combining two important theorems in stochastic calculus: the \emph{Feynman-Kac formula} for representing the marginal density of the generative diffusion as an expectation (Section \ref{sec:density}), and the \emph{Girsanov theorem} for performing inference in function space (Section \ref{sec:inference}).
We derive a functional evidence lower bound that consistently extends discrete-time diffusion models to have infinite depth, {\it i.e.}\@\xspace the number of layers goes to infinity (Section \ref{sec:infvae}).
Finally, by reparameterizing our generative and inference SDEs, we obtain a training objective equivalent to minimizing the (implicit) score matching loss (Section \ref{sec:score}).
Our theory suggests that by matching the score, one actually maximizes a lower bound on the log marginal density of the plug-in reverse SDE, laying a theoretical foundation for this learning procedure.
We further generalize our result to a family of \emph{marginal-equivalent} plug-in reverse SDEs, including an equivalent ODE as a limiting case.

\begin{figure}
    \centering
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_diffusion_sigma0.0.png}
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_diffusion.png}
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_inference.png}
    \caption{
    Three special cases of generative SDEs.
    The stars indicate the initial values, followed by some random sample paths.
    \emph{Left}: trained with no diffusion $\sigma=0$ ({\it i.e.}\@\xspace neural ODE).
    \emph{Middle}: trained with some fixed diffusion $\sigma>0$.
    \emph{Right}: trained with a fixed inference process, $f$ and $g$ ({\it i.e.}\@\xspace the plug-in reverse SDE).
    }
    \label{fig:generative_sde}
\end{figure}

\paragraph{Notation: }
We use $({Y}_s, s)$ to denote the inference process (where ${Y}_0$ is the data), and $({X}_t, t)$ to denote the generative process (where ${X}_0$ is a random variable following an unstructured prior).
We use $s$ and $t$ to distinguish the two directions, and always integrate the differential equations from $0$ to $T>0$ (different from the literature, where sometimes one might see integration from $T$ to $0$).
$\hat{B}_s$ and $B_t$ denote the Brownian motions associated with the inference and generative SDEs, respectively.
$B_s'$ is a reparameterization of $\hat{B}_s$ (see Section \ref{sec:inference}).
$q(y, s)$ and $p(x, t)$ denote the probability density functions of ${Y}_s$ and ${X}_t$, respectively.
We let $\mathsfit{s}_\theta$ denote a time-indexed parameterized function that will be used to approximate the score $\nabla \log q(y, s)$.
$\nabla$ is the gradient wrt the spatial variable ($x$ or $y$, which we sometimes call position), $\partt{ }$, $\parts{ }$ and $\partxj{i}$ are partial derivatives, and $H_{*}$ denotes Hessian.

\section{Background}
\label{sec:background}
Assume ${Y}_0$
follows the data distribution $q(y,0)$, and ${Y}_s$ satisfies the It\^{o} SDE \citep{oksendal2003stochastic}
\begin{align}
    \mathrm{d}{Y} = f({Y}, s) \,\mathrm{d}s + g({Y}, s) \,\mathrm{d}{\hat{B}}_s,
    \label{eq:fixed-inference-sde}
\end{align}
where $f$ and $g$ are chosen such that the density $q(y,s)$ will converge to some tractable prior $p_0$ as $s\rightarrow T$.
Following \citet{song2021score}, we assume $g$ is position-independent.
It is possible to find a ``reverse'' SDE, whose marginal density evolves according to $q(y,s)$, reversed in time, for example\footnote{See Appendix {\ref{app:equiv}} for a family of equivalent (reverse) SDEs indexed by some parameter $\lambda$, of which equation (\ref{eq:reverse-sde}) is a special case with $\lambda=0$. }
\begin{align}
    \mathrm{d}{X} = (gg^\top \nabla \log q({X}, T-t) - f) \,\mathrm{d}t + g \,\mathrm{d}B_t.
    \label{eq:reverse-sde}
\end{align}
If ${X}_0\sim p_0$, then the density $p(x, t)$ of ${X}_t$ is equal to $q(x, T-t)$.
This means that if we have access to the score function $\nabla \log q$, we can solve the above SDE to obtain ${X}_T\overset{d}{=}{Y}_0$.
\citet{song2021score} propose to approximate the score via a parameterized score function $\mathsfit{s}_\theta$ by minimizing
\begin{align*}
    \int_0^T \mathbb{E}_{{Y}_s}\left[\frac{1}{2}||\mathsfit{s}_\theta({Y}_s , s) - \nabla \log q({Y}_s, s) ||_{\Lambda(s)}^2\right] \, \mathrm{d}s
\end{align*}
where the expectation in the integral is known as the explicit score matching (ESM) loss $\gL_\text{ESM}$, and $\Lambda(s)$ is a positive definite matrix\footnote{We use this matrix to induce a Mahalanobis norm $||x||_{\Lambda}^2 := x^\top \Lambda x$, which will be used in Section \ref{sec:score}.} that serves as a weighting function for the overall loss.
$\gL_\text{ESM}$ is not immediately useful, since we do not have access to the ground truth score $\nabla \log q$.
A few alternative losses can be used, which are all equal to one another up to a constant, including implicit score matching \citep[ISM]{hyvarinen2005estimation}, sliced score matching \citep[SSM]{song2020sliced}, and denoising score matching \citep[DSM]{vincent2011connection}.
The losses are summarized in Table \ref{tab:score-matching-losses}, and are related through the following identity (see Appendix \ref{app:score-matching} for the derivation):
\begin{align}
\gL_\text{ESM} - \frac{1}{2} {\mathcal{I}}(q(y_s,s))
= \gL_\text{ISM} = \gL_\text{SSM} = \gL_\text{DSM} - \frac{1}{2}\mathbb{E}_{{Y}_0}[ {\mathcal{I}}(q(y_s|y_0))
],
\label{eq:sm-identity}
\end{align}
where %
${\mathcal{I}}(q) = \mathbb{E}[||\nabla \log q||_\Lambda^2]$ is a constant.
After training, \citet{song2021score} plug $\mathsfit{s}_\theta$ into (\ref{eq:reverse-sde}) to define a generative model.
We refer to this SDE as the \emph{plug-in reverse SDE}.
The plug-in reverse SDE has been demonstrated to have impressive empirical results, but a theoretical underpinning of this learning framework is still lacking.
For example, it is unclear how the training objective (minimizing the score matching loss) relates to the sampling procedure, {\it e.g.}\@\xspace whether the probability distribution induced by the plug-in reverse SDE gets closer to the data distribution in the sense of any statistical divergence or metric.
We seek to answer the following question in this paper:
\emph{How will minimizing the score-matching loss impact the plug-in reverse SDE?}
We first provide a framework to estimate the likelihood of generative SDEs, and then get back to this question in Section \ref{sec:score}.

\begin{table*}[]
    \centering
    \begin{minipage}[c]{0.50\textwidth}
    \centering
    \begin{tabular}{cc}
     \toprule
     Method & Loss \\
     \midrule
     $\gL_\text{ESM}$ & $\frac{1}{2}\mathbb{E}[||\mathsfit{s}_\theta({Y}_s, s)-\nabla \log q({Y}_s)||_\Lambda^2]$ \\
     $\gL_\text{ISM}$ & $\mathbb{E}[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s)||_\Lambda^2  + \nabla\cdot(\Lambda^\top\mathsfit{s}_\theta)]$ \\
     $\gL_\text{SSM}$ & $\mathbb{E}[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s)||_\Lambda^2  + v^\top\nabla(\Lambda^\top\mathsfit{s}_\theta)v]$ \\
     $\gL_\text{DSM}$ & $\frac{1}{2}\mathbb{E}[||\mathsfit{s}_\theta({Y}_s, s)-\nabla \log q({Y}_s|{Y}_0)||_\Lambda^2]$ \\
     \bottomrule
    \end{tabular}
    \caption{
    Score matching losses.
    $v$ follows the Rademacher distribution.
    }
    \label{tab:score-matching-losses}
    \end{minipage}
    \begin{minipage}[c]{0.45\textwidth}
    \centering
    \begin{tabular}{cc}
    \toprule
    F-K & F-P \\
    \midrule
    $v(y, \varsigma)$ & $p(y, T-\varsigma)$ \\
    \midrule
    $c(y, \varsigma)$ & $-\nabla \cdot \mu(y, T-\varsigma)$ \\
    $b(y, \varsigma)$ & $-\mu(y, T-\varsigma)$ \\
    $\eta(y,\varsigma)$ & $\sigma(T-\varsigma)$ \\
    $g(y)$ & $p_0(y)$ \\
    \bottomrule
    \end{tabular}
    \caption{Feynman-Kac coefficients.}
    \label{tab:fk-fp}
    \end{minipage}
\end{table*}

\section{Marginal density
and stochastic instantaneous change of variable
}
\label{sec:density}

Let ${X}_t$ be a diffusion process solving the following It\^{o} SDE\footnote{For generality, we use the notation $\mu$ and $\sigma$ to describe a generative SDE, which will be set to $g^2\mathsfit{s}_\theta -f$ and $g$ when we come back to the discussion of the plug-in reverse SDE in Section \ref{sec:score}.}:
\begin{align}
    \mathrm{d}{X} = \mu({X}, t) \,\mathrm{d}t+ \sigma({X}, t) \,\mathrm{d}B_t
    \label{eq:gen-sde}
\end{align}
with the initial condition ${X}_0\sim p_0$, which induces a family of densities ${X}_t\sim p(\cdot, t)$.
We use this SDE as the generative SDE, and we are interested in $\log p(x, T)$ for maximum likelihood.
The density $p(x, t)$ follows the
\emph{Kolmogorov forward} (or the \emph{Fokker Planck}) \emph{equation}:
\begin{align}
    \partt{p(x, t)} = - \sum_j \partxj{j}[\mu_j(x, t)\,p(x, t)] + \sum_{i,j} \partxxij{i}{j}[D_{ij}(x, t)\, p(x, t)]
    \label{eq:fp}
\end{align}
with the initial value $p(\cdot, 0)=p_0(\cdot)$, where $D = \frac{1}{2}\sigma \sigma^T$ is the diffusion matrix.
We can expand the Fokker Planck and rearrange the terms to obtain
\begin{align}
    \partt{p(x, t)} =
        &\left[ -\nabla\cdot\mu(x, t) + \sum_{i,j} \partxxij{i}{j} D_{ij}(x, t)  \right] p(x, t) \,\, + \nonumber \\
        &\sum_i\left[ - \mu_i(x, t) + 2 \sum_j \partxj{i} D_{ij}(x, t)\right] \partxj{i} p(x, t)  \,\, + %
        \sum_{i,j} D_{ij}(x,t) \partxxij{i}{j}p(x, t)
    \label{eq:fp-rearranged}
\end{align}
so that all coefficients of the same order are grouped together.
For simplicity, we assume the diffusion term $\sigma$ is independent of $x$ throughout the paper.
Then (\ref{eq:fp-rearranged}) reduces to
\begin{align}
    \partt{p(x, t)} =
        -\left(\nabla\cdot\mu(x, t)\right)\,p(x, t)
        - \mu(x, t)^\top  \nabla p(x, t)
        + D(t) : H_p(x, t)
    \label{eq:fp-simplified}
\end{align}
where $:$ denotes the Frobenius inner product between matrices.
Even with this simplification, solving (\ref{eq:fp-simplified}) is not trivial.
Fortunately, we can estimate this quantity using the \emph{Feynman-Kac formula}, which tells us that the solution of certain second-order linear partial differential equations have a probabilistic representation.

\begin{restatable}[\textbf{Feynman-Kac representation}, Chapter 5.7 of \citet{karatzas2014brownian}]{thm}{feynmankac}
Let $T>0$.
Let $y$ and $\varsigma$ be the spatial and temporal arguments to the function $v \in C^{2, 1}(\mathbb{R}^d\times [0,T])$ solving
\begin{align}
\partial_{\varsigma}{v} + c v + b^\top \nabla v + A : H_v = 0
\label{eq:fk-pde}
\end{align}
with the terminal condition $v(y, T)=h(y)$, where $A = \frac{1}{2}\eta\eta^\top$ for some matrix-valued function $\eta(y, \varsigma)$.
Under the assumption stated in Appendix \ref{app:fk-assumption}, if $B_s'$ is a Brownian motion and ${Y}_s$ solves
\begin{align}
    \mathrm{d}{Y} = b({Y}, s) \,\mathrm{d}s + \eta({Y}, s) \,\mathrm{d}B_s',
    \label{eq:fk-dynamic}
\end{align}
with the initial datum ${Y}_\varsigma = y$, then
\begin{align}
    v(y, \varsigma) = \mathbb{E}\left[h({Y}_T) \exp\left(\int_{\varsigma}^T c({Y}_{s}, s) \,\mathrm{d}s \right)\, \Bigg\vert \, {Y}_{\varsigma} = y \right].
    \label{eq:fk-rep}
\end{align}
\end{restatable}

To estimate the density $p(\cdot, T)$ of (\ref{eq:fp-simplified}),
we can apply the change of variable $p(x, t) := v(x, T-t)$ by letting the Feynman-Kac (F-K) coefficients correspond to their Fokker-Planck (F-P) counterparts according to Table~\ref{tab:fk-fp}.
This way, solving (\ref{eq:fk-pde}) backward is equivalent to solving (\ref{eq:fp-simplified}) forward, and we have the following representation of the marginal density at $T$:
\begin{align}
    p(x,T) = \mathbb{E}\left[p_0({Y}_T) \exp\left(\int_0^T -\nabla \cdot \mu({Y}_{s}, T-s) \,\mathrm{d}s\right) \, \Bigg\vert \, {Y}_0 = x\right],
    \label{eq:fk-fp}
\end{align}
where ${Y}_s$ is a diffusion process solving
\begin{align}
    \mathrm{d} {Y} = -\mu({Y}, T-s)\,\mathrm{d}s + \sigma(T-s) \,\mathrm{d} B_s'.
    \label{eq:fk-fp-dynamic}
\end{align}

\begin{restatable}[\textbf{Marginalization}]{rmk}{}
\label{rm:marginalization}
This representation can be interpreted as a mixture of continuous time flows.
Assume a sample path of the Brownian motion is given, and we are interested in how the density evolves following the dynamic (\ref{eq:gen-sde}).
In the infinitesimal setting, it can been seen as applying the invertible map $x\mapsto x + \mu(x,t)\Delta t + \sigma(t)\Delta B_i$, where $\Delta B_i := B_{(i+1)\Delta t} - B_{i\Delta t}$ is the Brownian increment.
Since the diffusion term is independent of the spatial variable, it can be seen as a constant additive transformation, which is volume preserving, so it will not be taken into account when computing the change of density.
The only contribution to the change of density will be from the log-determinant of the Jacobian of $\text{id} + \mu\Delta t$, which means we can simply apply the instantaneous change of variable formula \citep{chen2018neural}.
This will be the conditional density given the entire $\{B_t:t\geq0\}$, and marginalizing it out results in the expectation in (\ref{eq:fk-fp}).
See Appendix \ref{app:mixture} for details.
\end{restatable}

Our framework also works with the general case where $\sigma$ depends on $x$, but the formulae need to be adapted to account for the spatial partial derivatives.
See Appendix \ref{app:density} for the derivation.

\section{Inferring latent Brownian motion}
\label{sec:inference}

As our goal is to estimate likelihood, we would like to compute the log density value using (\ref{eq:fk-fp}).
However, this involves integrating out all possible Brownian paths, which is intractable.
To resolve this, we view the Brownian motion as a latent variable, and perform inference by assigning higher probability to sample paths that are more likely to generate the observation.
One can view this as a VAE, except we have an infinite dimensional latent variable.

Formally, let $(\Omega, {\mathcal{F}},{\mathbb{P}})$ be the underlying probability space for which $B_s'$ is a Brownian motion.
Suppose ${\mathbb{Q}}$ is another probability measure on $(\Omega, {\mathcal{F}})$ \emph{equivalent} to ${\mathbb{P}}$; that is, ${\mathbb{P}}$ and ${\mathbb{Q}}$ are similar in the sense that they have the same measure zero sets.
This allows us to apply the change-of-measure trick and lower bound the log-likelihood with a finite quantity using Jensen's inequality:
\begin{align}
    \log p(x, T)
    &\geq  \mathbb{E}_{\mathbb{Q}}\left[ \log \frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}} + \log p_0({Y}_T) - \int_0^T \nabla\cdot\mu \,\mathrm{d}s \,\bigg|\, {Y}_0=x\right].
    \label{eq:elbo}
\end{align}
Note that $\frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}}$ is the \emph{Radon-Nikodym} derivative of ${\mathbb{P}}$ wrt ${\mathbb{Q}}$.
When both measures are absolutely continuous wrt a third measure, say Lebesgue, then the derivative can be expressed as the ratio of the two densities.
However, since we are dealing with an infinite dimensional space, we are immediately faced with the following problems:
\begin{enumerate}
    \item Is there a measure ${\mathbb{Q}}$ (equiv. to ${\mathbb{P}}$) for which $\frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}}$ can be easily computed, or at least numerically approximated?
    \item Can we find a reparameterization (similar to the Gaussian reparameterization) of $B_s'$ under the new law ${\mathbb{Q}}$ to estimate the gradient needed for training?
\end{enumerate}

We resort to the \emph{Girsanov theorem}, which describes a general framework for dealing with the change of measure of Gaussian random variables under additive perturbation.
It allows us to consider the law of a diffusion process as ${\mathbb{Q}}$.
See Appendix \ref{app:1d-girsanov} for an explanation using the more familiar notion of probability densities.

\begin{restatable}[\textbf{Girsanov theorem}, Theorem 8.6.3 of \citet{oksendal2003stochastic}]{thm}{girsanov}
Let $\hat{B}_s$ be an It\^o process solving
\begin{align}
    \mathrm{d}\hat{B}_s = - a(\omega, s) \,\mathrm{d}s + \mathrm{d}B_s',
    \label{eq:girsanov-standardize}
\end{align}
for $\omega\in\Omega$, $0\leq s\leq T$ and $\hat{B}_0=0$,
where $a(\omega, s)$ satisfies the Novikov's condition $\mathbb{E}\left[\exp\left(\frac{1}{2}\int_0^T a^2 \,\mathrm{d}s\right)\right] < \infty$.
Then $\hat{B}_s$ is a Brownian motion wrt ${\mathbb{Q}}$ where
\begin{align}
    \frac{\mathrm{d}{\mathbb{Q}}}{\mathrm{d}{\mathbb{P}}}(\omega) := \exp\left(\int_0^T a(\omega, s) \cdot \mathrm{d}B_s' - \frac{1}{2} \int_0^T ||a(\omega, s)||_2^2 \,\mathrm{d}s\right).
    \label{eq:girsanov-density}
\end{align}
\end{restatable}
Equation (\ref{eq:girsanov-standardize}) provides a standarization formula of $B_s'$ under ${\mathbb{Q}}$, which means we can ``invert'' it to reparameterize $B_s'$.
This leads to the following lower bound.

\begin{restatable}[\textbf{Continuous-time ELBO}]{thm}{ctelbo}
Let ${\mathbb{Q}}$ be defined via the density (\ref{eq:girsanov-density}).
Then the RHS of (\ref{eq:elbo}) can be rewritten as
\begin{align}
    \mathbb{E}\left[
    - \frac{1}{2} \int_0^T ||a(\omega, s)||_2^2 \,\mathrm{d}s + \log p_0({Y}_T) - \int_0^T \nabla\cdot\mu \,\mathrm{d}s
    \, \Bigg\vert \, {Y}_0=x
    \right]  =: {\mathcal{E}}^\infty,
    \label{eq:continuous:elbo}
\end{align}
where the expectation is taken wrt the Brownian motion $\hat{B}_s$, and ${Y}_s$ solves\footnote{Note that $\mu$ and $\sigma$ run backward in time from $T$, whereas $a$ runs forward.}
\begin{align}
    \mathrm{d} {Y} = (-\mu + \sigma a)  \,\mathrm{d}s + \sigma \mathrm{d} \hat{B}_s.
    \label{eq:fk-fp-dynamic-reparameterized}
\end{align}
\end{restatable}

We call ${Y}_s$ solving (\ref{eq:fk-fp-dynamic-reparameterized}) the inference SDE, and ${\mathcal{E}}^\infty$ the continuous-time ELBO (CT-ELBO).

\begin{restatable}[\bf Computation]{rmk}{}
\label{rm:computation}
This lower bound can be numerically estimated by using any black box SDE solver, by augmenting the dynamic of $y$ with the accumulation of $||a||^2$ and $\nabla\cdot\mu$.
Computing the divergence term $\nabla\cdot\mu$ directly can be expensive, but it can be efficiently estimated using the Hutchinson trace estimator \citep{hutchinson1989stochastic} along with reverse-mode automatic differentiation, similar to \citet{grathwohl2018ffjord}.
As the parameters of both the generative and inference models are decoupled from the random variable $\hat{B}_s$, their gradients can be estimated via the reparameterization trick \citep{kingma2014auto, rezende2014stochastic}.
Furthermore, backpropagation can be computed using an adjoint method with a constant memory cost \citep{li2020scalable}.
\end{restatable}

\begin{restatable}[\bf Drift $a$]{rmk}{}
(i) In general, the drift term of the approximate posterior to the latent Brownian motion can be amortized, so that it will encode the information of individual datum $x$.
(ii) The regularization $||a||^2$ ensures that $a$ is kept close to $0$, since it represents the deviation of the measure it induces ({\it i.e.}\@\xspace ${\mathbb{Q}}$) from the classical Wiener measure (which is a centered Gaussian measure).
(iii) When the diffusion coefficient $\sigma$ is $0$, the inference SDE reduces to the reverse dynamic of the generative ODE, and if $a\equiv 0$ in this case, the lower bound is tight.
(iv) There is generally no constraint on the form of $a(\omega, s)$, so one can potentially augment it with additional dimensions to have a non-Markovian inference SDE.
For simplicity, we let the inference SDE be a Markovian model, i.e. $a=a(y, s)$. This is justified by the following theorem.
\end{restatable}

\begin{restatable}[\textbf{Variational gap and optimal inference SDE}]{thm}{optinf}
\label{thm:optinf}
The variational gap can be written as
\begin{align}
    \log p(x,T) - {\mathcal{E}}^\infty = \int_0^T\mathbb{E}\left[||a(\omega, s) - \sigma^\top \nabla \log p({Y}_s, T-s)||^2\right] \,\mathrm{d}s.
\end{align}
In particular,
${\mathcal{E}}^\infty=\log p(x,T)$ if and only if
$a(\omega, s)$ can be written as $a(\omega, s) = a({Y}_s(\omega), s)$ for almost every $s\in[0,T]$ and $\omega\in\Omega$, and
$a(y, s)=\sigma^\top \nabla \log p(y, T-s)$ almost everywhere.
\end{restatable}

\begin{restatable}[\bf Variational gap]{rmk}{}
Even though the inference SDE seemingly takes a simple form, it is sufficiently flexible in that
this type of variational problem can be generally solved by taking the supremem over all progressively measurable processes $a(\omega, s)$ \citep{boue1998variational}.
In fact, the above theorem shows that
${\mathcal{E}}^\infty = \log p(x,T)$ if and only if $a(y, s) = \sigma^\top \nabla \log p(y, T-s)$.
This means a non-amortized Markovian inference process is powerful enough.
\end{restatable}

\section{Infinitely deep hierarchical VAE}
\label{sec:infvae}
Before we make the connection to score matching, we formally address the common belief that ``diffusion models can be viewed as the continuous limit of hierarchical VAEs'' \citep{tzen2019neural}, and show that the CT-ELBO consistently extends their discrete-time counterpart.
We do so by inspecting the ELBO of a hierarchical VAE defined as discretized\footnote{We follow the Euler-Maruyama (EM) scheme. Other discretization scheme may also work; we leave that for future work.} generative and inference SDEs.
We assume the generative model ({\it i.e.}\@\xspace the decoder) follows the transitional distributions
\begin{gather}
    p(x_{i+1} | x_{i}) = {\mathcal{N}}(x_{i+1} ; \tilde{\mu}_{i}(x_{i}), \tilde{\sigma}^2_{i}) \\
    \tilde{\mu}_{i}(x) = x + \Delta t\mu(x, i\Delta t) \qquad
    \tilde{\sigma}^2_{i} = \Delta t\sigma^2(i\Delta t),
    \label{eq:discrete-transition-p}
\end{gather}
where $\Delta t = T/L$ is the step size and $L$ is the number of layers.
For the inference model ({\it i.e.}\@\xspace the encoder), we assume
\begin{gather}
    q(x_{i} | x_{i+1}) = {\mathcal{N}}(x_{i} ; \hat{\mu}_{i+1}(x_{i+1}), \hat{\sigma}^2_{i+1}) \\
    \hat{\mu}_{i}(x) = x + \Delta t (- \mu(x, i\Delta t) + \sigma(i\Delta t) a(x,T-i\Delta t)) \qquad
    \hat{\sigma}^2_{i} = \Delta t\sigma^2(i\Delta t).
    \label{eq:discrete-transition-q}
\end{gather}

These transition kernels constitute a hierarchical variational autoencoder of $L$ stochastic layers, whose marginal likelihood can be lower bounded by
\begin{align}
    \log p(x_{L}) \geq \mathbb{E}_q\left[\log p(x_0)+\sum_{i=0}^{L-1}\log \frac{p(x_{i+1} |x_i)}{ q(x_i|x_{i+1})}\right] =: {\mathcal{E}}^L,
    \label{eq:discrete-elbo}
\end{align}
which we refer to as the discrete-time ELBO (DT-ELBO).
The reconstruction error of the stochastic layer can be seen as some form of finite difference approximation to differentiation, which gives rise to $\nabla\cdot\mu$ in the CT-ELBO in the infinitesimal limit (as $\Delta t$ approaches $0$).
The regularization of $||a||^2$ pops up when we compare the difference between $\tilde{\mu}_i$ and $\hat{\mu}_i$ using the Gaussian reparameterization to compute the reconstruction error.
We formalize this idea in the following theorem.

\begin{restatable}[\textbf{Consistency}]{thm}{consistency}
Assume $\mu$, $\sigma$, $\sigma^{-2}$, $a$, $||a||^2$ and their derivatives up to the fourth order are all bounded and continuous, and that $\sigma$ is non-singular. Then ${\mathcal{E}}^L\rightarrow{\mathcal{E}}^\infty$ as $L\rightarrow\infty$.
\end{restatable}
This theorem tells us that the CT-ELBO we derive for continuous-time diffusion models is not that different from the traditional ELBO, and that maximizing the CT-ELBO can be seen as training an infinitely deep hierarchical VAE.
We present the proof in Appendix \ref{app:proofs}, which formalizes the above intuition, using Taylor's theorem to control the polynomial approximation error, which will go to $0$ as the step size $\Delta t$ vanishes when the number of layers $L$ increases to infinity.

\section{Score-based generative modeling}
\label{sec:score}

\begin{figure}
    \centering
    \includegraphics[width=0.50\textwidth]{figures/equivalent-plug-in-reverse-sdes-main.png}
    \hfill
    \includegraphics[width=0.44\textwidth]{figures/lambda_lower_bound.pdf}
    \caption{
    \emph{Left}: Samples from plug-in reverse SDEs with different $\lambda$ values (rows).
    We use the same score function $\mathsfit{s}_\theta$ trained on the Swiss roll dataset, and plug it into (\ref{eq:lambda-sde-pair}).
    For generation, we use the Euler Maruyama method with a step size of $\Delta t = 1/1000$.
    We visualize the samples for the $i$-th iterates (columns), which approximately represent the same marginal distribution when the score function is well trained.
    \emph{Right}:
    Lower bound on the marginal likelihood of a continuum of plug-in reverse SDEs.
    The lower bound is optimized when the score matching loss is minimized, which will push up the entire dark blue curve.
    }
    \label{fig:equiv-plug-in-lambda-elbo}
\end{figure}
Recall that our goal is to analyze the plug-in reverse SDE and draw connection to score matching.
To this end, we reparameterize the generative (\ref{eq:gen-sde}) and inference (\ref{eq:fk-fp-dynamic-reparameterized}) SDEs as
\begin{align}
    \mathrm{d}{X} = (gg^\top \mathsfit{s}_\theta -f)\,\mathrm{d}t + g\,\mathrm{d}B_t \,\,\text{ and }\,\, \mathrm{d}{Y} = f\,\mathrm{d}s + g\,\mathrm{d}\hat{B}_s,
    \label{eq:plug-in-reverse-forward}
\end{align}
by letting $a=g^\top\mathsfit{s}_\theta$, where the time variable is reversed ($T-t$) for the generative process, and forward in time ($s$) for inference.
The ELBO (\ref{eq:continuous:elbo}) can be rewritten as
\begin{align}
    {\mathcal{E}}^\infty
    = \mathbb{E}_{{Y}_T}[\log p_0({Y}_T)\,|\,{Y}_0=x] - \int_0^T \mathbb{E}_{{Y}_s}\left[ \frac{1}{2} ||\mathsfit{s}_\theta||_{gg^\top}^2 + \nabla \cdot (gg^\top \mathsfit{s}_\theta - f) \,\bigg|\, {Y}_0=x \right] \,\mathrm{d}s.
    \label{eq:continuous-elbo-ism}
\end{align}

Comparing the integrand to the implicit score matching loss in Table \ref{tab:score-matching-losses}, we immediately see that the network $\mathsfit{s}_\theta$ approximates $\nabla \log q(y, s)$, the score function of the marginal density of ${Y}_s$.
That is, \emph{matching the score of $q(y,t)$ amounts to maximizing the lower bound on the marginal likelihood of the plug-in reverse SDE}. %

Recently, \citet{durkan2021maximum}\footnote{This refers to the \href{https://arxiv.org/abs/2101.09258v1}{v1} of the paper on arXiv. This version was later on replaced with a new version where they derived a similar bound as ours.} also attempt to establish the equivalency between maximum likelihood and score matching, by showing the following relationship between the forward KL divergence and a weighted sum of score matching loss (aka the Fisher divergence):
\begin{align}
    D_{\mathrm{KL}}(q(y,0) ||r(y, 0)) = \frac{1}{2}\int_0^T \mathbb{E}_{q(\cdot, s)}\left[||\nabla \log r({Y}_s, s) - \nabla \log q({Y}_s, s)||_{gg^\top}^2\right]\,\mathrm{d}s,
\end{align}

where $r(y, s)$ is the density of ${Y}_s$ solving the same inference SDE with the initial condition $y_0\sim r(\cdot, 0)$, assuming $q(y,T)=r(y,T)$.
However, it is inaccurate to claim that score matching is equivalent to maximum likelihood.
This is because if we simply let $r(y, 0)= p(y, T)$, {\it i.e.}\@\xspace the density of the generative SDE evaluated at $y$, $r(y, s)$ will not necessarily be the same as either $p(y, T-s)$ or $\mathsfit{s}_\theta(y, s)$.
This means the KL divergence is not equal to the integral of the weighted score matching loss $\mathbb{E}[\frac{1}{2}||\mathsfit{s}_\theta - \nabla \log q||_{gg^\top}^2]$.
In fact, the latter corresponds to a lower bound on the likelihood (the cross-entropy term of the KL) up to some constant, as equation (\ref{eq:continuous-elbo-ism}) suggests.

More generally, we can apply our analysis to a family of plug-in reverse SDEs indexed by some parameter $\lambda\leq1$:
\begin{align}
    \resizebox{0.99\textwidth}{!}{%
    $\mathrm{d} {X} =
    \left(\left(1-\frac{\lambda}{2}\right)g^2 \mathsfit{s}_\theta - f\right) \,\mathrm{d}t + \sqrt{1-\lambda} g \,\mathrm{d}B_t
    \,\,\text{ and }\,\,
    \mathrm{d} {Y} = \left(f - \frac{\lambda}{2} g^2 \nabla \log q\right) \,\mathrm{d}s + \sqrt{1-\lambda} g \,\mathrm{d}\hat{B}_s,
    $}
    \label{eq:lambda-sde-pair}
\end{align}
where we assume $g$ is diagonal for simplicity.
We defer the formal discussion to Appendix \ref{app:equiv}, but the essence is that this inference SDE induces the same marginal distribution as (\ref{eq:fixed-inference-sde}), and the generative SDE is its corresponding plug-in reverse.
Equation (\ref{eq:lambda-sde-pair}) includes the original plug-in reverse SDE (\ref{eq:plug-in-reverse-forward}) and an equivalent ODE as special cases with $\lambda=0$ and $\lambda=1$.
Denote its corresponding CT-ELBO by ${\mathcal{E}}^\infty_\lambda$.
Specifically, (\ref{eq:continuous-elbo-ism}) becomes ${\mathcal{E}}^\infty_0$.
Then we have the following relationship.

\begin{restatable}[\bf Plug-in reverse SDE ELBO, abridged]{thm}{elboscoreshort}
For $\lambda < 1$,
\begin{align}
    \mathbb{E}_{{Y}_0}[{\mathcal{E}}^\infty_\lambda] = \mathbb{E}_{{Y}_0}[{\mathcal{E}}^\infty_0] - \left(\frac{\lambda^2}{4(1-\lambda)}\right) \int_{0}^T \mathbb{E}_{{Y}_s}\left[\frac{1}{2}||\mathsfit{s}_\theta({Y}_s, s) - \nabla \log q({Y}_s, s)||_{g^2}^2\right] \,\mathrm{d}s
\end{align}
\end{restatable}

We state the full theorem in Appendix \ref{app:score-plug-in-likelihood}, where we rearrange the terms to show that the average CT-ELBO of the $\lambda$-plug-in reverse SDE is also equivalent to the ISM loss, similarly to (\ref{eq:continuous-elbo-ism}) but up to some multiplying and additive constants.
The implication is that \emph{while minimizing the score matching loss, we implicitly maximize the likelihood of a continuum of plug-in reverse SDEs which include the ODE as a limiting case ($\lambda\rightarrow1$)}.
See Figure \ref{fig:equiv-plug-in-lambda-elbo} (right) for illustration.
This suggests the likelihood of the equivalent ODE can be improved by minimizing the score matching loss, as the ODE's likelihood will be close to plug-in reverse SDEs with $\lambda\approx1$,
which explains the good likelihood of the equivalent ODE reported in \citet{song2021score}.
In practice, we can only estimate the ELBO of the case $\lambda=0$ since otherwise there will be some constant we do not have access to, but their gradients can all be estimated via score matching.

\subsection{Computational trade-off}
\label{subsec:computation}
Having a general framework for estimating the likelihood of diffusion processes allows us to compare a wide family of models, including continuous-time flows and plug-in reverse SDEs trained by score matching.
We compare the two by measuring the negative ELBO throughout training to highlight their computation-estimation trade-off.
We train the models on the Swiss roll toy data.
For continuous-time flow, we set $\sigma=0$, using the Hutchinson trace estimator following \citet{grathwohl2018ffjord}.
The ELBO in this case is tight since $a$ will be penalized to be 0.
We use the \texttt{torchdiffeq} library \citep{chen2018neural} for numerical integration for fairer comparison\footnote{Black-box SDE solvers such as \texttt{torchsde} \citep{li2020scalable} might not be optimized for the deterministic case, since their stochastic adjoint method scales ${\mathcal{O}}(L\log L)$ in time whereas deterministic numerical solvers are usually faster. This matters for our runtime comparison.}.
For plug-in reverse SDE, we train the drift network $a$ using SSM and DSM (for DSM the loss is weighted to reduce variance, which introduces some bias; see the next subsection).
We use the variance-preserving inference SDE from \citet{song2021score}, which allows us to sample ${Y}_s$ using a closed form formula, for $s$ sampled uniformly between $[0,T]$.
The trained models are visualized in Figure \ref{fig:generative_sde}, the learning curves presented in Figure \ref{fig:ode-sde}.

From the learning curve figures, we see that neg-likelihood decreases rapidly for the continuous-time flow in terms of the number of parameter updates.
But once the x-axis is normalized by runtime, the convergence speed becomes almost indistinguishable.
This is because for continuous-time flows, numerical integration takes time, whereas for plug-in reverse SDEs, we train on a random time step $s$; that is, within a fixed amount of time the latter can make more parameter updates at the cost of noisier gradients.
Note that both models have constant memory cost (wrt $T$ or $L$, the number of integration steps), so a large batch size can be used to reduce variance for training.

\subsection{Bias and variance trade-off}
\label{subsec:bias-variance}
The integral in equation (\ref{eq:continuous-elbo-ism}) can be estimated by sampling $({Y}_s, s)$, and using the Hutchinson trace estimator to estimate the divergence, which corresponds to implicit score matching.
However, in practice the variance of this estimator is very high when the norm of the Jacobian $\nabla \mathsfit{s}_\theta$ is large.
Another popular approach is to use the denoising estimator (recall the identity from (\ref{eq:sm-identity})),
\begin{align}
    \mathbb{E}_{{Y}_s}\left[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s) - \nabla \log q({Y}_s|{Y}_0)||_{gg^\top}^2 \,\bigg|\, {Y}_0=x\right].
    \label{eq:dsm}
\end{align}
The inference SDE is typically chosen so that ${Y}_s$ can be easily sampled, e.g. following ${\mathcal{N}}(\mu_s, \sigma_s^2)$, where $\mu_s$ and $\sigma_s$ are functions of ${Y}_0$ and $s$.
In this case, if we reparameterize ${Y}_s=\mu_s + \sigma_s \epsilon$ where $\epsilon\sim{\mathcal{N}}(0,\mathbf{I})$, then the score becomes
$\nabla \log q = - \frac{\epsilon}{\sigma_s}$.
Since $\sigma_s\rightarrow0$ as $s\rightarrow0$, this estimator normally has unbounded variance.
\citet{song2019generative, song2021score} propose to remedy
this by multiplying the DSM loss by $\sigma_s^2/g^2$ (assuming $g$ is a scalar for simplicity), so that the target has constant magnitude on average $\mathbb{E}[\frac{1}{2}||\sigma_s\mathsfit{s}_\theta + \epsilon||^2]$, which would result in a biased gradient estimate with much smaller variance.
We can debias this estimator by sampling $s\sim q(s)\propto g^2/\sigma_s^2$.
This ratio, however, is usally not normalizable in practice (as it integrates to $\infty$).
As an alternative, we consider the following unnormalized density
$\tilde{q}_\epsilon(s) = g^2(s_\epsilon)/\sigma_{s_\epsilon}^2$ for $s\in[0, s_\epsilon]$, and $\tilde{q}_\epsilon(s) = g^2(s)/\sigma_s^2$ for $s\in[s_\epsilon, T]$.
We experiment with this debiased procedure by sampling $s\sim q_\epsilon\propto\tilde{q}_\epsilon$, for $f$ and $g$ chosen to be the variance-preserving SDE.
$s_\epsilon$ is small so that the bias is negligible.

We train the model on MNIST \citep{lecun1998gradient} and CIFAR10 \citep{krizhevsky2009learning}.
We present the learning curves and the standard error of the estimate of the ELBO in Figure \ref{fig:debias}.
The lower bound is estimated using the Hutchinson trace estimator with $s$ sampled uniformly from $[0,T]$, with the same batch size, so the only thing that will affect the dispersion is the magnitude of $\nabla \mathsfit{s}_\theta$.
Since smaller values of $s$ are more likely to be sampled under $q_\epsilon$, the debiased model will see samples with less perturbation more often.
On the contrary, sampling $s$ uniformly will bias the model to learn from noisier data, causing the learned score to be smoother.
We also experiment with parameterizing $\mathsfit{s}_\theta$ vs parameterizing $a$.
We find the latter parameterization to be helpful since the relationship $\mathsfit{s}_\theta=g^{-1}a$ has the effect of negating the multiplier $\sigma_s$ in the reweighted loss, {\it i.e.}\@\xspace $\mathbb{E}[\frac{1}{2}||\frac{\sigma_s}{g}a + \epsilon||^2]$.
This is similar to the noise conditioning technique introduced in \citet{song2020improved}.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.49\linewidth]{figures/ODE-SDE-iterations.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/ODE-SDE-seconds.pdf}
    \caption{Neural ODE vs plug-in reverse SDE (denoising or slice score matching). The learning curves are presented as a function of iterations (left) and runtime (right) to emphasize the computational distinction between the two families of models.}
    \label{fig:ode-sde}
\end{figure}
\begin{figure}
    \centering
    \includegraphics[width=0.49\linewidth]{figures/mnist_bpd_aggregated.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/mnist_bpd_std_err_aggregated.pdf}
    \includegraphics[width=0.49\linewidth]{figures/cifar_bpd_aggregated.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/cifar_bpd_std_err_aggregated.pdf}
    \caption{Likelihood estimation on MNIST (first row) and CIFAR10 (second row). $\mathsfit{s}_\theta$ and $a$ denote which model we parameterize. Y-axes are bits-per-dim and the standard error of BPD of the test set.
    The debiased curves improve upon the original biased gradient estimator \citep{song2021score} since it maximizes a proper ELBO.
    Shaded area reflects the uncertainty estimated by 3 random seeds.}
    \label{fig:debias}
\end{figure}

\section{Related work}
\label{sec:related}

\paragraph{Diffusion-based generative models}
Our work lays a theoretical foundation for \citet{song2021score}, which recognizes that conditional denoising score matching \citep{song2019generative, song2020improved} and discrete-time diffusion-based generative models \citep{sohl2015deep, goyal2017variational, ho2020denoising} can be viewed as learning to revert an inference process (using the plug-in reverse SDE).
Different from \citet{ho2020denoising}, which shows the ELBO of discrete time diffusion process can be likened to DSM (Section 3.2 of the paper), we show that ISM loss naturally arises from the Fokker-Planck equation of the marginal density, via the Fenman-Kac representation and the Girsanov change of measure.
This line of work has been successfully applied to modeling high dimensional natural images \citep{dhariwal2021diffusion, saharia2021image}, audio \citep{kong2020diffwave}, 3D point cloud \citep{cai2020learning, zhou20213d}, and discrete data \citep{hoogeboom2021argmax}.

\paragraph{Time-reversal of diffusion processes}
Plenty of works have studied the reverse-time diffusion processes (\ref{eq:reverse-sde}), including \citet{anderson1982reverse, follmer1985entropy, elliott1985reverse, haussmann1986time}.
These are different from our marginal-equivalent (reverse) processes (\ref{eq:lambda-sde-pair}) when $\mathsfit{s}_\theta=\nabla \log q$, since the latter is related by the marginals only.

\paragraph{Score matching for energy-based models}
Besides the connection to diffusion models, score matching is also often used as a method for learning energy based models (EBM)---~see \citet{song2021train} for a comprehensive review on useful techniques---.
When used as an EBM, sampling from the conditional score model can be achieved by running the annealed Langevin diffusion \citep{neal2001annealed}, which is connected to free-energy estimaton in physics \citep{jarzynski1997equilibrium}, wherein the \emph{path integral} is essentially a Feynman Kac representation.

\paragraph{De Bruijn's identity}
To connect maximum likelihood and score matching, \citet{durkan2021maximum} shows that KL divergence can be represented as an integral of weighted Fisher divergence, generalizing the case of \citet{lyu2009interpretation} where the inference perturbation is a simple Brownian motion.
This type of formulas fall into the category of de Bruijn's identity \citep{cover1999elements} for relative entropy.
A similar differential form result can be found in \citet{wibisono2017information}.

\paragraph{Learning SDEs}
\citet{tzen2019neural, li2020scalable} also propose to learn a neural SDE by applying Girsanov's theorem.
The key difference is that they treat the SDE entirely as a latent variable, with an additional emission probability, whereas we use the Feynman-Kac formula to directly express the marginal density as an expectation, side-stepping the need to smooth out the density using the emission probability (which will be a Dirac point mass in our case).
In their case, the inference direction is the same as the generative direction, since they infer the latent SDE directly, whereas we apply Girsanov to the Feynman-Kac diffusion (opposite the generative direction).
\citet{xu2021infinitely} further apply neural SDE as an infinitely deep Bayesian neural network.

\section{Conclusion and Discussion}
In this work, we derive a general variational framework for estimating the marginal likelihood of continuous-time diffusion models.
This framework allows us to study a wide spectrum of models, including continuous-time normalizing flows and score-based generative models.
Using our framework, we show that performing score matching with a particular choice of mixture weighting is equivalent to maximizing a lower bound on the marginal likelihood of a family of plug-in reverse SDEs, of which the one used in \citet{song2021score} and the equivalent ODE are special cases.
Empirically, we validate our theory by monitoring the ELBO while performing score matching, and discuss the implication of the choice of mixture weighting and the potential of debiasing via non-uniform sampling.
We emphasize that our theory does not explain the impressive sample quality of this family of models, which is still an open research problem and we leave it for future work.

This work introduces a general framework to estimate the likelihood of diffusion-based models, which allows the parameters of both the generative and inference SDEs to be learned, using a numerical solver with constant memory cost (as per Remark \ref{rm:computation}).
The training time can be reduced via the connection to score matching and the reverse-time parameterization (\ref{eq:plug-in-reverse-forward}) as long as $f$ and $g$ take a simple form (so that ${Y}_t$ can be sampled without numerical integration).
For example, one can generalize the Ornstein-Uhlenbeck process to have non-linear (in time) $f$ and $g$, similar to the variance-presering SDE, by parameterizing the integral of $f$ using a monotone network \citep{sill1998monotonic,kay2000estimating,daniels2010monotone,huang2018neural}.
This has been explored in a concurrent work by \citet{kingma2021variational} in a different framework.

\section*{Acknowledgements}
We would like to thank
David Kanaa, Ricky Chen, Simon Verret, Rémi Piché-Taillefer, Alexia Jolicoeur-Martineau,  and Faruk Ahmed for giving their feedback on this manuscript.
We would also like to thank the INNF+ 2021 reviewers and NeurIPS 2021 reviewers for their constructive suggestions, which help us improve the clarity of the paper.
Chin-Wei is supported by the Google PhD fellowship.

We also acknowledge the Python community \citep{van1995python, oliphant2007python} for developing the tools that enabled this work, including
numpy \citep{oliphant2006guide,van2011numpy, walt2011numpy, harris2020array},
PyTorch \citep{paszke2019pytorch},
Matplotlib \citep{hunter2007matplotlib},
seaborn \citep{seaborn}, and
SciPy \citep{jones2014scipy}.

\section{Introduction}
Generative modeling can be thought of as inverting an inference process.
If the inference process is invertible, then one can focus on transforming the data into a tractable distribution \citep{dinh2016density}.
If the inference process is deterministic yet non-invertible, one could learn to invert it stochastically \citep{dinh2019rad, nielsen2020survae}.
Most generally, both inference and generation can be stochastic.
This is known as the variational autoencoder \citep[VAE]{kingma2014auto, rezende2014stochastic}.

Under the variational framework, one has a lot of flexibility in choosing the generative and inference models.
Recent work on diffusion-based modeling \citep{sohl2015deep, ho2020denoising} can be thought of as removing one degree of freedom, by freezing the inference path.
The inference model is a fixed discrete-time Markov chain, that slowly transforms the data into a tractable prior, such as the standard normal distribution.
The generative model is another Markov chain that is trained to revert this process iteratively.
Diffusion-based models have been shown to perform remarkably well on image synthesis \citep{dhariwal2021diffusion}, rivaling the performance of state-of-the-art Generative Adversarial Networks \citep{brock2018large}.

\citet{song2021score} connect diffusion-based model and \emph{score matching} \citep{hyvarinen2005estimation}, by looking at the stochastic differential equation (SDE) associated with the inference process.
They realize that the dynamic of the inference process can be inverted if one has access to the score function of the perturbed data, by solving another SDE reversed in time.
They then propose to learn the score function of the inference process and substitute the approximate score into the formula of the reverse SDE to obtain a generative model.
We call the resulting generative model the plug-in reverse SDE.

Conceptually simple as this learning procedure may seem, little is known about how the score matching loss relates to the plug-in reverse SDE.
In this paper, we propose a variational framework suitable for likelihood estimation for general generative diffusion processes, and use this framework to connect score matching with maximum likelihood.
We do so by combining two important theorems in stochastic calculus: the \emph{Feynman-Kac formula} for representing the marginal density of the generative diffusion as an expectation (Section \ref{sec:density}), and the \emph{Girsanov theorem} for performing inference in function space (Section \ref{sec:inference}).
We derive a functional evidence lower bound that consistently extends discrete-time diffusion models to have infinite depth, {\it i.e.}\@\xspace the number of layers goes to infinity (Section \ref{sec:infvae}).
Finally, by reparameterizing our generative and inference SDEs, we obtain a training objective equivalent to minimizing the (implicit) score matching loss (Section \ref{sec:score}).
Our theory suggests that by matching the score, one actually maximizes a lower bound on the log marginal density of the plug-in reverse SDE, laying a theoretical foundation for this learning procedure.
We further generalize our result to a family of \emph{marginal-equivalent} plug-in reverse SDEs, including an equivalent ODE as a limiting case.

\begin{figure}
    \centering
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_diffusion_sigma0.0.png}
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_diffusion.png}
    \includegraphics[width=.32\linewidth]{figures/toy_SwissRoll_sde_with_paths_fixed_inference.png}
    \caption{
    Three special cases of generative SDEs.
    The stars indicate the initial values, followed by some random sample paths.
    \emph{Left}: trained with no diffusion $\sigma=0$ ({\it i.e.}\@\xspace neural ODE).
    \emph{Middle}: trained with some fixed diffusion $\sigma>0$.
    \emph{Right}: trained with a fixed inference process, $f$ and $g$ ({\it i.e.}\@\xspace the plug-in reverse SDE).
    }
    \label{fig:generative_sde}
\end{figure}

\paragraph{Notation: }
We use $({Y}_s, s)$ to denote the inference process (where ${Y}_0$ is the data), and $({X}_t, t)$ to denote the generative process (where ${X}_0$ is a random variable following an unstructured prior).
We use $s$ and $t$ to distinguish the two directions, and always integrate the differential equations from $0$ to $T>0$ (different from the literature, where sometimes one might see integration from $T$ to $0$).
$\hat{B}_s$ and $B_t$ denote the Brownian motions associated with the inference and generative SDEs, respectively.
$B_s'$ is a reparameterization of $\hat{B}_s$ (see Section \ref{sec:inference}).
$q(y, s)$ and $p(x, t)$ denote the probability density functions of ${Y}_s$ and ${X}_t$, respectively.
We let $\mathsfit{s}_\theta$ denote a time-indexed parameterized function that will be used to approximate the score $\nabla \log q(y, s)$.
$\nabla$ is the gradient wrt the spatial variable ($x$ or $y$, which we sometimes call position), $\partt{ }$, $\parts{ }$ and $\partxj{i}$ are partial derivatives, and $H_{*}$ denotes Hessian.

\section{Background}
\label{sec:background}
Assume ${Y}_0$
follows the data distribution $q(y,0)$, and ${Y}_s$ satisfies the It\^{o} SDE \citep{oksendal2003stochastic}
\begin{align}
    \mathrm{d}{Y} = f({Y}, s) \,\mathrm{d}s + g({Y}, s) \,\mathrm{d}{\hat{B}}_s,
    \label{eq:fixed-inference-sde}
\end{align}
where $f$ and $g$ are chosen such that the density $q(y,s)$ will converge to some tractable prior $p_0$ as $s\rightarrow T$.
Following \citet{song2021score}, we assume $g$ is position-independent.
It is possible to find a ``reverse'' SDE, whose marginal density evolves according to $q(y,s)$, reversed in time, for example\footnote{See Appendix {\ref{app:equiv}} for a family of equivalent (reverse) SDEs indexed by some parameter $\lambda$, of which equation (\ref{eq:reverse-sde}) is a special case with $\lambda=0$. }
\begin{align}
    \mathrm{d}{X} = (gg^\top \nabla \log q({X}, T-t) - f) \,\mathrm{d}t + g \,\mathrm{d}B_t.
    \label{eq:reverse-sde}
\end{align}
If ${X}_0\sim p_0$, then the density $p(x, t)$ of ${X}_t$ is equal to $q(x, T-t)$.
This means that if we have access to the score function $\nabla \log q$, we can solve the above SDE to obtain ${X}_T\overset{d}{=}{Y}_0$.
\citet{song2021score} propose to approximate the score via a parameterized score function $\mathsfit{s}_\theta$ by minimizing
\begin{align*}
    \int_0^T \mathbb{E}_{{Y}_s}\left[\frac{1}{2}||\mathsfit{s}_\theta({Y}_s , s) - \nabla \log q({Y}_s, s) ||_{\Lambda(s)}^2\right] \, \mathrm{d}s
\end{align*}
where the expectation in the integral is known as the explicit score matching (ESM) loss $\gL_\text{ESM}$, and $\Lambda(s)$ is a positive definite matrix\footnote{We use this matrix to induce a Mahalanobis norm $||x||_{\Lambda}^2 := x^\top \Lambda x$, which will be used in Section \ref{sec:score}.} that serves as a weighting function for the overall loss.
$\gL_\text{ESM}$ is not immediately useful, since we do not have access to the ground truth score $\nabla \log q$.
A few alternative losses can be used, which are all equal to one another up to a constant, including implicit score matching \citep[ISM]{hyvarinen2005estimation}, sliced score matching \citep[SSM]{song2020sliced}, and denoising score matching \citep[DSM]{vincent2011connection}.
The losses are summarized in Table \ref{tab:score-matching-losses}, and are related through the following identity (see Appendix \ref{app:score-matching} for the derivation):
\begin{align}
\gL_\text{ESM} - \frac{1}{2} {\mathcal{I}}(q(y_s,s))
= \gL_\text{ISM} = \gL_\text{SSM} = \gL_\text{DSM} - \frac{1}{2}\mathbb{E}_{{Y}_0}[ {\mathcal{I}}(q(y_s|y_0))
],
\label{eq:sm-identity}
\end{align}
where %
${\mathcal{I}}(q) = \mathbb{E}[||\nabla \log q||_\Lambda^2]$ is a constant.
After training, \citet{song2021score} plug $\mathsfit{s}_\theta$ into (\ref{eq:reverse-sde}) to define a generative model.
We refer to this SDE as the \emph{plug-in reverse SDE}.
The plug-in reverse SDE has been demonstrated to have impressive empirical results, but a theoretical underpinning of this learning framework is still lacking.
For example, it is unclear how the training objective (minimizing the score matching loss) relates to the sampling procedure, {\it e.g.}\@\xspace whether the probability distribution induced by the plug-in reverse SDE gets closer to the data distribution in the sense of any statistical divergence or metric.
We seek to answer the following question in this paper:
\emph{How will minimizing the score-matching loss impact the plug-in reverse SDE?}
We first provide a framework to estimate the likelihood of generative SDEs, and then get back to this question in Section \ref{sec:score}.

\begin{table*}[]
    \centering
    \begin{minipage}[c]{0.50\textwidth}
    \centering
    \begin{tabular}{cc}
     \toprule
     Method & Loss \\
     \midrule
     $\gL_\text{ESM}$ & $\frac{1}{2}\mathbb{E}[||\mathsfit{s}_\theta({Y}_s, s)-\nabla \log q({Y}_s)||_\Lambda^2]$ \\
     $\gL_\text{ISM}$ & $\mathbb{E}[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s)||_\Lambda^2  + \nabla\cdot(\Lambda^\top\mathsfit{s}_\theta)]$ \\
     $\gL_\text{SSM}$ & $\mathbb{E}[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s)||_\Lambda^2  + v^\top\nabla(\Lambda^\top\mathsfit{s}_\theta)v]$ \\
     $\gL_\text{DSM}$ & $\frac{1}{2}\mathbb{E}[||\mathsfit{s}_\theta({Y}_s, s)-\nabla \log q({Y}_s|{Y}_0)||_\Lambda^2]$ \\
     \bottomrule
    \end{tabular}
    \caption{
    Score matching losses.
    $v$ follows the Rademacher distribution.
    }
    \label{tab:score-matching-losses}
    \end{minipage}
    \begin{minipage}[c]{0.45\textwidth}
    \centering
    \begin{tabular}{cc}
    \toprule
    F-K & F-P \\
    \midrule
    $v(y, \varsigma)$ & $p(y, T-\varsigma)$ \\
    \midrule
    $c(y, \varsigma)$ & $-\nabla \cdot \mu(y, T-\varsigma)$ \\
    $b(y, \varsigma)$ & $-\mu(y, T-\varsigma)$ \\
    $\eta(y,\varsigma)$ & $\sigma(T-\varsigma)$ \\
    $g(y)$ & $p_0(y)$ \\
    \bottomrule
    \end{tabular}
    \caption{Feynman-Kac coefficients.}
    \label{tab:fk-fp}
    \end{minipage}
\end{table*}

\section{Marginal density
and stochastic instantaneous change of variable
}
\label{sec:density}

Let ${X}_t$ be a diffusion process solving the following It\^{o} SDE\footnote{For generality, we use the notation $\mu$ and $\sigma$ to describe a generative SDE, which will be set to $g^2\mathsfit{s}_\theta -f$ and $g$ when we come back to the discussion of the plug-in reverse SDE in Section \ref{sec:score}.}:
\begin{align}
    \mathrm{d}{X} = \mu({X}, t) \,\mathrm{d}t+ \sigma({X}, t) \,\mathrm{d}B_t
    \label{eq:gen-sde}
\end{align}
with the initial condition ${X}_0\sim p_0$, which induces a family of densities ${X}_t\sim p(\cdot, t)$.
We use this SDE as the generative SDE, and we are interested in $\log p(x, T)$ for maximum likelihood.
The density $p(x, t)$ follows the
\emph{Kolmogorov forward} (or the \emph{Fokker Planck}) \emph{equation}:
\begin{align}
    \partt{p(x, t)} = - \sum_j \partxj{j}[\mu_j(x, t)\,p(x, t)] + \sum_{i,j} \partxxij{i}{j}[D_{ij}(x, t)\, p(x, t)]
    \label{eq:fp}
\end{align}
with the initial value $p(\cdot, 0)=p_0(\cdot)$, where $D = \frac{1}{2}\sigma \sigma^T$ is the diffusion matrix.
We can expand the Fokker Planck and rearrange the terms to obtain
\begin{align}
    \partt{p(x, t)} =
        &\left[ -\nabla\cdot\mu(x, t) + \sum_{i,j} \partxxij{i}{j} D_{ij}(x, t)  \right] p(x, t) \,\, + \nonumber \\
        &\sum_i\left[ - \mu_i(x, t) + 2 \sum_j \partxj{i} D_{ij}(x, t)\right] \partxj{i} p(x, t)  \,\, + %
        \sum_{i,j} D_{ij}(x,t) \partxxij{i}{j}p(x, t)
    \label{eq:fp-rearranged}
\end{align}
so that all coefficients of the same order are grouped together.
For simplicity, we assume the diffusion term $\sigma$ is independent of $x$ throughout the paper.
Then (\ref{eq:fp-rearranged}) reduces to
\begin{align}
    \partt{p(x, t)} =
        -\left(\nabla\cdot\mu(x, t)\right)\,p(x, t)
        - \mu(x, t)^\top  \nabla p(x, t)
        + D(t) : H_p(x, t)
    \label{eq:fp-simplified}
\end{align}
where $:$ denotes the Frobenius inner product between matrices.
Even with this simplification, solving (\ref{eq:fp-simplified}) is not trivial.
Fortunately, we can estimate this quantity using the \emph{Feynman-Kac formula}, which tells us that the solution of certain second-order linear partial differential equations have a probabilistic representation.

\begin{restatable}[\textbf{Feynman-Kac representation}, Chapter 5.7 of \citet{karatzas2014brownian}]{thm}{feynmankac}
Let $T>0$.
Let $y$ and $\varsigma$ be the spatial and temporal arguments to the function $v \in C^{2, 1}(\mathbb{R}^d\times [0,T])$ solving
\begin{align}
\partial_{\varsigma}{v} + c v + b^\top \nabla v + A : H_v = 0
\label{eq:fk-pde}
\end{align}
with the terminal condition $v(y, T)=h(y)$, where $A = \frac{1}{2}\eta\eta^\top$ for some matrix-valued function $\eta(y, \varsigma)$.
Under the assumption stated in Appendix \ref{app:fk-assumption}, if $B_s'$ is a Brownian motion and ${Y}_s$ solves
\begin{align}
    \mathrm{d}{Y} = b({Y}, s) \,\mathrm{d}s + \eta({Y}, s) \,\mathrm{d}B_s',
    \label{eq:fk-dynamic}
\end{align}
with the initial datum ${Y}_\varsigma = y$, then
\begin{align}
    v(y, \varsigma) = \mathbb{E}\left[h({Y}_T) \exp\left(\int_{\varsigma}^T c({Y}_{s}, s) \,\mathrm{d}s \right)\, \Bigg\vert \, {Y}_{\varsigma} = y \right].
    \label{eq:fk-rep}
\end{align}
\end{restatable}

To estimate the density $p(\cdot, T)$ of (\ref{eq:fp-simplified}),
we can apply the change of variable $p(x, t) := v(x, T-t)$ by letting the Feynman-Kac (F-K) coefficients correspond to their Fokker-Planck (F-P) counterparts according to Table~\ref{tab:fk-fp}.
This way, solving (\ref{eq:fk-pde}) backward is equivalent to solving (\ref{eq:fp-simplified}) forward, and we have the following representation of the marginal density at $T$:
\begin{align}
    p(x,T) = \mathbb{E}\left[p_0({Y}_T) \exp\left(\int_0^T -\nabla \cdot \mu({Y}_{s}, T-s) \,\mathrm{d}s\right) \, \Bigg\vert \, {Y}_0 = x\right],
    \label{eq:fk-fp}
\end{align}
where ${Y}_s$ is a diffusion process solving
\begin{align}
    \mathrm{d} {Y} = -\mu({Y}, T-s)\,\mathrm{d}s + \sigma(T-s) \,\mathrm{d} B_s'.
    \label{eq:fk-fp-dynamic}
\end{align}

\begin{restatable}[\textbf{Marginalization}]{rmk}{}
\label{rm:marginalization}
This representation can be interpreted as a mixture of continuous time flows.
Assume a sample path of the Brownian motion is given, and we are interested in how the density evolves following the dynamic (\ref{eq:gen-sde}).
In the infinitesimal setting, it can been seen as applying the invertible map $x\mapsto x + \mu(x,t)\Delta t + \sigma(t)\Delta B_i$, where $\Delta B_i := B_{(i+1)\Delta t} - B_{i\Delta t}$ is the Brownian increment.
Since the diffusion term is independent of the spatial variable, it can be seen as a constant additive transformation, which is volume preserving, so it will not be taken into account when computing the change of density.
The only contribution to the change of density will be from the log-determinant of the Jacobian of $\text{id} + \mu\Delta t$, which means we can simply apply the instantaneous change of variable formula \citep{chen2018neural}.
This will be the conditional density given the entire $\{B_t:t\geq0\}$, and marginalizing it out results in the expectation in (\ref{eq:fk-fp}).
See Appendix \ref{app:mixture} for details.
\end{restatable}

Our framework also works with the general case where $\sigma$ depends on $x$, but the formulae need to be adapted to account for the spatial partial derivatives.
See Appendix \ref{app:density} for the derivation.

\section{Inferring latent Brownian motion}
\label{sec:inference}

As our goal is to estimate likelihood, we would like to compute the log density value using (\ref{eq:fk-fp}).
However, this involves integrating out all possible Brownian paths, which is intractable.
To resolve this, we view the Brownian motion as a latent variable, and perform inference by assigning higher probability to sample paths that are more likely to generate the observation.
One can view this as a VAE, except we have an infinite dimensional latent variable.

Formally, let $(\Omega, {\mathcal{F}},{\mathbb{P}})$ be the underlying probability space for which $B_s'$ is a Brownian motion.
Suppose ${\mathbb{Q}}$ is another probability measure on $(\Omega, {\mathcal{F}})$ \emph{equivalent} to ${\mathbb{P}}$; that is, ${\mathbb{P}}$ and ${\mathbb{Q}}$ are similar in the sense that they have the same measure zero sets.
This allows us to apply the change-of-measure trick and lower bound the log-likelihood with a finite quantity using Jensen's inequality:
\begin{align}
    \log p(x, T)
    &\geq  \mathbb{E}_{\mathbb{Q}}\left[ \log \frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}} + \log p_0({Y}_T) - \int_0^T \nabla\cdot\mu \,\mathrm{d}s \,\bigg|\, {Y}_0=x\right].
    \label{eq:elbo}
\end{align}
Note that $\frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}}$ is the \emph{Radon-Nikodym} derivative of ${\mathbb{P}}$ wrt ${\mathbb{Q}}$.
When both measures are absolutely continuous wrt a third measure, say Lebesgue, then the derivative can be expressed as the ratio of the two densities.
However, since we are dealing with an infinite dimensional space, we are immediately faced with the following problems:
\begin{enumerate}
    \item Is there a measure ${\mathbb{Q}}$ (equiv. to ${\mathbb{P}}$) for which $\frac{\mathrm{d}{\mathbb{P}}}{\mathrm{d}{\mathbb{Q}}}$ can be easily computed, or at least numerically approximated?
    \item Can we find a reparameterization (similar to the Gaussian reparameterization) of $B_s'$ under the new law ${\mathbb{Q}}$ to estimate the gradient needed for training?
\end{enumerate}

We resort to the \emph{Girsanov theorem}, which describes a general framework for dealing with the change of measure of Gaussian random variables under additive perturbation.
It allows us to consider the law of a diffusion process as ${\mathbb{Q}}$.
See Appendix \ref{app:1d-girsanov} for an explanation using the more familiar notion of probability densities.

\begin{restatable}[\textbf{Girsanov theorem}, Theorem 8.6.3 of \citet{oksendal2003stochastic}]{thm}{girsanov}
Let $\hat{B}_s$ be an It\^o process solving
\begin{align}
    \mathrm{d}\hat{B}_s = - a(\omega, s) \,\mathrm{d}s + \mathrm{d}B_s',
    \label{eq:girsanov-standardize}
\end{align}
for $\omega\in\Omega$, $0\leq s\leq T$ and $\hat{B}_0=0$,
where $a(\omega, s)$ satisfies the Novikov's condition $\mathbb{E}\left[\exp\left(\frac{1}{2}\int_0^T a^2 \,\mathrm{d}s\right)\right] < \infty$.
Then $\hat{B}_s$ is a Brownian motion wrt ${\mathbb{Q}}$ where
\begin{align}
    \frac{\mathrm{d}{\mathbb{Q}}}{\mathrm{d}{\mathbb{P}}}(\omega) := \exp\left(\int_0^T a(\omega, s) \cdot \mathrm{d}B_s' - \frac{1}{2} \int_0^T ||a(\omega, s)||_2^2 \,\mathrm{d}s\right).
    \label{eq:girsanov-density}
\end{align}
\end{restatable}
Equation (\ref{eq:girsanov-standardize}) provides a standarization formula of $B_s'$ under ${\mathbb{Q}}$, which means we can ``invert'' it to reparameterize $B_s'$.
This leads to the following lower bound.

\begin{restatable}[\textbf{Continuous-time ELBO}]{thm}{ctelbo}
Let ${\mathbb{Q}}$ be defined via the density (\ref{eq:girsanov-density}).
Then the RHS of (\ref{eq:elbo}) can be rewritten as
\begin{align}
    \mathbb{E}\left[
    - \frac{1}{2} \int_0^T ||a(\omega, s)||_2^2 \,\mathrm{d}s + \log p_0({Y}_T) - \int_0^T \nabla\cdot\mu \,\mathrm{d}s
    \, \Bigg\vert \, {Y}_0=x
    \right]  =: {\mathcal{E}}^\infty,
    \label{eq:continuous:elbo}
\end{align}
where the expectation is taken wrt the Brownian motion $\hat{B}_s$, and ${Y}_s$ solves\footnote{Note that $\mu$ and $\sigma$ run backward in time from $T$, whereas $a$ runs forward.}
\begin{align}
    \mathrm{d} {Y} = (-\mu + \sigma a)  \,\mathrm{d}s + \sigma \mathrm{d} \hat{B}_s.
    \label{eq:fk-fp-dynamic-reparameterized}
\end{align}
\end{restatable}

We call ${Y}_s$ solving (\ref{eq:fk-fp-dynamic-reparameterized}) the inference SDE, and ${\mathcal{E}}^\infty$ the continuous-time ELBO (CT-ELBO).

\begin{restatable}[\bf Computation]{rmk}{}
\label{rm:computation}
This lower bound can be numerically estimated by using any black box SDE solver, by augmenting the dynamic of $y$ with the accumulation of $||a||^2$ and $\nabla\cdot\mu$.
Computing the divergence term $\nabla\cdot\mu$ directly can be expensive, but it can be efficiently estimated using the Hutchinson trace estimator \citep{hutchinson1989stochastic} along with reverse-mode automatic differentiation, similar to \citet{grathwohl2018ffjord}.
As the parameters of both the generative and inference models are decoupled from the random variable $\hat{B}_s$, their gradients can be estimated via the reparameterization trick \citep{kingma2014auto, rezende2014stochastic}.
Furthermore, backpropagation can be computed using an adjoint method with a constant memory cost \citep{li2020scalable}.
\end{restatable}

\begin{restatable}[\bf Drift $a$]{rmk}{}
(i) In general, the drift term of the approximate posterior to the latent Brownian motion can be amortized, so that it will encode the information of individual datum $x$.
(ii) The regularization $||a||^2$ ensures that $a$ is kept close to $0$, since it represents the deviation of the measure it induces ({\it i.e.}\@\xspace ${\mathbb{Q}}$) from the classical Wiener measure (which is a centered Gaussian measure).
(iii) When the diffusion coefficient $\sigma$ is $0$, the inference SDE reduces to the reverse dynamic of the generative ODE, and if $a\equiv 0$ in this case, the lower bound is tight.
(iv) There is generally no constraint on the form of $a(\omega, s)$, so one can potentially augment it with additional dimensions to have a non-Markovian inference SDE.
For simplicity, we let the inference SDE be a Markovian model, i.e. $a=a(y, s)$. This is justified by the following theorem.
\end{restatable}

\begin{restatable}[\textbf{Variational gap and optimal inference SDE}]{thm}{optinf}
\label{thm:optinf}
The variational gap can be written as
\begin{align}
    \log p(x,T) - {\mathcal{E}}^\infty = \int_0^T\mathbb{E}\left[||a(\omega, s) - \sigma^\top \nabla \log p({Y}_s, T-s)||^2\right] \,\mathrm{d}s.
\end{align}
In particular,
${\mathcal{E}}^\infty=\log p(x,T)$ if and only if
$a(\omega, s)$ can be written as $a(\omega, s) = a({Y}_s(\omega), s)$ for almost every $s\in[0,T]$ and $\omega\in\Omega$, and
$a(y, s)=\sigma^\top \nabla \log p(y, T-s)$ almost everywhere.
\end{restatable}

\begin{restatable}[\bf Variational gap]{rmk}{}
Even though the inference SDE seemingly takes a simple form, it is sufficiently flexible in that
this type of variational problem can be generally solved by taking the supremem over all progressively measurable processes $a(\omega, s)$ \citep{boue1998variational}.
In fact, the above theorem shows that
${\mathcal{E}}^\infty = \log p(x,T)$ if and only if $a(y, s) = \sigma^\top \nabla \log p(y, T-s)$.
This means a non-amortized Markovian inference process is powerful enough.
\end{restatable}

\section{Infinitely deep hierarchical VAE}
\label{sec:infvae}
Before we make the connection to score matching, we formally address the common belief that ``diffusion models can be viewed as the continuous limit of hierarchical VAEs'' \citep{tzen2019neural}, and show that the CT-ELBO consistently extends their discrete-time counterpart.
We do so by inspecting the ELBO of a hierarchical VAE defined as discretized\footnote{We follow the Euler-Maruyama (EM) scheme. Other discretization scheme may also work; we leave that for future work.} generative and inference SDEs.
We assume the generative model ({\it i.e.}\@\xspace the decoder) follows the transitional distributions
\begin{gather}
    p(x_{i+1} | x_{i}) = {\mathcal{N}}(x_{i+1} ; \tilde{\mu}_{i}(x_{i}), \tilde{\sigma}^2_{i}) \\
    \tilde{\mu}_{i}(x) = x + \Delta t\mu(x, i\Delta t) \qquad
    \tilde{\sigma}^2_{i} = \Delta t\sigma^2(i\Delta t),
    \label{eq:discrete-transition-p}
\end{gather}
where $\Delta t = T/L$ is the step size and $L$ is the number of layers.
For the inference model ({\it i.e.}\@\xspace the encoder), we assume
\begin{gather}
    q(x_{i} | x_{i+1}) = {\mathcal{N}}(x_{i} ; \hat{\mu}_{i+1}(x_{i+1}), \hat{\sigma}^2_{i+1}) \\
    \hat{\mu}_{i}(x) = x + \Delta t (- \mu(x, i\Delta t) + \sigma(i\Delta t) a(x,T-i\Delta t)) \qquad
    \hat{\sigma}^2_{i} = \Delta t\sigma^2(i\Delta t).
    \label{eq:discrete-transition-q}
\end{gather}

These transition kernels constitute a hierarchical variational autoencoder of $L$ stochastic layers, whose marginal likelihood can be lower bounded by
\begin{align}
    \log p(x_{L}) \geq \mathbb{E}_q\left[\log p(x_0)+\sum_{i=0}^{L-1}\log \frac{p(x_{i+1} |x_i)}{ q(x_i|x_{i+1})}\right] =: {\mathcal{E}}^L,
    \label{eq:discrete-elbo}
\end{align}
which we refer to as the discrete-time ELBO (DT-ELBO).
The reconstruction error of the stochastic layer can be seen as some form of finite difference approximation to differentiation, which gives rise to $\nabla\cdot\mu$ in the CT-ELBO in the infinitesimal limit (as $\Delta t$ approaches $0$).
The regularization of $||a||^2$ pops up when we compare the difference between $\tilde{\mu}_i$ and $\hat{\mu}_i$ using the Gaussian reparameterization to compute the reconstruction error.
We formalize this idea in the following theorem.

\begin{restatable}[\textbf{Consistency}]{thm}{consistency}
Assume $\mu$, $\sigma$, $\sigma^{-2}$, $a$, $||a||^2$ and their derivatives up to the fourth order are all bounded and continuous, and that $\sigma$ is non-singular. Then ${\mathcal{E}}^L\rightarrow{\mathcal{E}}^\infty$ as $L\rightarrow\infty$.
\end{restatable}
This theorem tells us that the CT-ELBO we derive for continuous-time diffusion models is not that different from the traditional ELBO, and that maximizing the CT-ELBO can be seen as training an infinitely deep hierarchical VAE.
We present the proof in Appendix \ref{app:proofs}, which formalizes the above intuition, using Taylor's theorem to control the polynomial approximation error, which will go to $0$ as the step size $\Delta t$ vanishes when the number of layers $L$ increases to infinity.

\section{Score-based generative modeling}
\label{sec:score}

\begin{figure}
    \centering
    \includegraphics[width=0.50\textwidth]{figures/equivalent-plug-in-reverse-sdes-main.png}
    \hfill
    \includegraphics[width=0.44\textwidth]{figures/lambda_lower_bound.pdf}
    \caption{
    \emph{Left}: Samples from plug-in reverse SDEs with different $\lambda$ values (rows).
    We use the same score function $\mathsfit{s}_\theta$ trained on the Swiss roll dataset, and plug it into (\ref{eq:lambda-sde-pair}).
    For generation, we use the Euler Maruyama method with a step size of $\Delta t = 1/1000$.
    We visualize the samples for the $i$-th iterates (columns), which approximately represent the same marginal distribution when the score function is well trained.
    \emph{Right}:
    Lower bound on the marginal likelihood of a continuum of plug-in reverse SDEs.
    The lower bound is optimized when the score matching loss is minimized, which will push up the entire dark blue curve.
    }
    \label{fig:equiv-plug-in-lambda-elbo}
\end{figure}
Recall that our goal is to analyze the plug-in reverse SDE and draw connection to score matching.
To this end, we reparameterize the generative (\ref{eq:gen-sde}) and inference (\ref{eq:fk-fp-dynamic-reparameterized}) SDEs as
\begin{align}
    \mathrm{d}{X} = (gg^\top \mathsfit{s}_\theta -f)\,\mathrm{d}t + g\,\mathrm{d}B_t \,\,\text{ and }\,\, \mathrm{d}{Y} = f\,\mathrm{d}s + g\,\mathrm{d}\hat{B}_s,
    \label{eq:plug-in-reverse-forward}
\end{align}
by letting $a=g^\top\mathsfit{s}_\theta$, where the time variable is reversed ($T-t$) for the generative process, and forward in time ($s$) for inference.
The ELBO (\ref{eq:continuous:elbo}) can be rewritten as
\begin{align}
    {\mathcal{E}}^\infty
    = \mathbb{E}_{{Y}_T}[\log p_0({Y}_T)\,|\,{Y}_0=x] - \int_0^T \mathbb{E}_{{Y}_s}\left[ \frac{1}{2} ||\mathsfit{s}_\theta||_{gg^\top}^2 + \nabla \cdot (gg^\top \mathsfit{s}_\theta - f) \,\bigg|\, {Y}_0=x \right] \,\mathrm{d}s.
    \label{eq:continuous-elbo-ism}
\end{align}

Comparing the integrand to the implicit score matching loss in Table \ref{tab:score-matching-losses}, we immediately see that the network $\mathsfit{s}_\theta$ approximates $\nabla \log q(y, s)$, the score function of the marginal density of ${Y}_s$.
That is, \emph{matching the score of $q(y,t)$ amounts to maximizing the lower bound on the marginal likelihood of the plug-in reverse SDE}. %

Recently, \citet{durkan2021maximum}\footnote{This refers to the \href{https://arxiv.org/abs/2101.09258v1}{v1} of the paper on arXiv. This version was later on replaced with a new version where they derived a similar bound as ours.} also attempt to establish the equivalency between maximum likelihood and score matching, by showing the following relationship between the forward KL divergence and a weighted sum of score matching loss (aka the Fisher divergence):
\begin{align}
    D_{\mathrm{KL}}(q(y,0) ||r(y, 0)) = \frac{1}{2}\int_0^T \mathbb{E}_{q(\cdot, s)}\left[||\nabla \log r({Y}_s, s) - \nabla \log q({Y}_s, s)||_{gg^\top}^2\right]\,\mathrm{d}s,
\end{align}

where $r(y, s)$ is the density of ${Y}_s$ solving the same inference SDE with the initial condition $y_0\sim r(\cdot, 0)$, assuming $q(y,T)=r(y,T)$.
However, it is inaccurate to claim that score matching is equivalent to maximum likelihood.
This is because if we simply let $r(y, 0)= p(y, T)$, {\it i.e.}\@\xspace the density of the generative SDE evaluated at $y$, $r(y, s)$ will not necessarily be the same as either $p(y, T-s)$ or $\mathsfit{s}_\theta(y, s)$.
This means the KL divergence is not equal to the integral of the weighted score matching loss $\mathbb{E}[\frac{1}{2}||\mathsfit{s}_\theta - \nabla \log q||_{gg^\top}^2]$.
In fact, the latter corresponds to a lower bound on the likelihood (the cross-entropy term of the KL) up to some constant, as equation (\ref{eq:continuous-elbo-ism}) suggests.

More generally, we can apply our analysis to a family of plug-in reverse SDEs indexed by some parameter $\lambda\leq1$:
\begin{align}
    \resizebox{0.99\textwidth}{!}{%
    $\mathrm{d} {X} =
    \left(\left(1-\frac{\lambda}{2}\right)g^2 \mathsfit{s}_\theta - f\right) \,\mathrm{d}t + \sqrt{1-\lambda} g \,\mathrm{d}B_t
    \,\,\text{ and }\,\,
    \mathrm{d} {Y} = \left(f - \frac{\lambda}{2} g^2 \nabla \log q\right) \,\mathrm{d}s + \sqrt{1-\lambda} g \,\mathrm{d}\hat{B}_s,
    $}
    \label{eq:lambda-sde-pair}
\end{align}
where we assume $g$ is diagonal for simplicity.
We defer the formal discussion to Appendix \ref{app:equiv}, but the essence is that this inference SDE induces the same marginal distribution as (\ref{eq:fixed-inference-sde}), and the generative SDE is its corresponding plug-in reverse.
Equation (\ref{eq:lambda-sde-pair}) includes the original plug-in reverse SDE (\ref{eq:plug-in-reverse-forward}) and an equivalent ODE as special cases with $\lambda=0$ and $\lambda=1$.
Denote its corresponding CT-ELBO by ${\mathcal{E}}^\infty_\lambda$.
Specifically, (\ref{eq:continuous-elbo-ism}) becomes ${\mathcal{E}}^\infty_0$.
Then we have the following relationship.

\begin{restatable}[\bf Plug-in reverse SDE ELBO, abridged]{thm}{elboscoreshort}
For $\lambda < 1$,
\begin{align}
    \mathbb{E}_{{Y}_0}[{\mathcal{E}}^\infty_\lambda] = \mathbb{E}_{{Y}_0}[{\mathcal{E}}^\infty_0] - \left(\frac{\lambda^2}{4(1-\lambda)}\right) \int_{0}^T \mathbb{E}_{{Y}_s}\left[\frac{1}{2}||\mathsfit{s}_\theta({Y}_s, s) - \nabla \log q({Y}_s, s)||_{g^2}^2\right] \,\mathrm{d}s
\end{align}
\end{restatable}

We state the full theorem in Appendix \ref{app:score-plug-in-likelihood}, where we rearrange the terms to show that the average CT-ELBO of the $\lambda$-plug-in reverse SDE is also equivalent to the ISM loss, similarly to (\ref{eq:continuous-elbo-ism}) but up to some multiplying and additive constants.
The implication is that \emph{while minimizing the score matching loss, we implicitly maximize the likelihood of a continuum of plug-in reverse SDEs which include the ODE as a limiting case ($\lambda\rightarrow1$)}.
See Figure \ref{fig:equiv-plug-in-lambda-elbo} (right) for illustration.
This suggests the likelihood of the equivalent ODE can be improved by minimizing the score matching loss, as the ODE's likelihood will be close to plug-in reverse SDEs with $\lambda\approx1$,
which explains the good likelihood of the equivalent ODE reported in \citet{song2021score}.
In practice, we can only estimate the ELBO of the case $\lambda=0$ since otherwise there will be some constant we do not have access to, but their gradients can all be estimated via score matching.

\subsection{Computational trade-off}
\label{subsec:computation}
Having a general framework for estimating the likelihood of diffusion processes allows us to compare a wide family of models, including continuous-time flows and plug-in reverse SDEs trained by score matching.
We compare the two by measuring the negative ELBO throughout training to highlight their computation-estimation trade-off.
We train the models on the Swiss roll toy data.
For continuous-time flow, we set $\sigma=0$, using the Hutchinson trace estimator following \citet{grathwohl2018ffjord}.
The ELBO in this case is tight since $a$ will be penalized to be 0.
We use the \texttt{torchdiffeq} library \citep{chen2018neural} for numerical integration for fairer comparison\footnote{Black-box SDE solvers such as \texttt{torchsde} \citep{li2020scalable} might not be optimized for the deterministic case, since their stochastic adjoint method scales ${\mathcal{O}}(L\log L)$ in time whereas deterministic numerical solvers are usually faster. This matters for our runtime comparison.}.
For plug-in reverse SDE, we train the drift network $a$ using SSM and DSM (for DSM the loss is weighted to reduce variance, which introduces some bias; see the next subsection).
We use the variance-preserving inference SDE from \citet{song2021score}, which allows us to sample ${Y}_s$ using a closed form formula, for $s$ sampled uniformly between $[0,T]$.
The trained models are visualized in Figure \ref{fig:generative_sde}, the learning curves presented in Figure \ref{fig:ode-sde}.

From the learning curve figures, we see that neg-likelihood decreases rapidly for the continuous-time flow in terms of the number of parameter updates.
But once the x-axis is normalized by runtime, the convergence speed becomes almost indistinguishable.
This is because for continuous-time flows, numerical integration takes time, whereas for plug-in reverse SDEs, we train on a random time step $s$; that is, within a fixed amount of time the latter can make more parameter updates at the cost of noisier gradients.
Note that both models have constant memory cost (wrt $T$ or $L$, the number of integration steps), so a large batch size can be used to reduce variance for training.

\subsection{Bias and variance trade-off}
\label{subsec:bias-variance}
The integral in equation (\ref{eq:continuous-elbo-ism}) can be estimated by sampling $({Y}_s, s)$, and using the Hutchinson trace estimator to estimate the divergence, which corresponds to implicit score matching.
However, in practice the variance of this estimator is very high when the norm of the Jacobian $\nabla \mathsfit{s}_\theta$ is large.
Another popular approach is to use the denoising estimator (recall the identity from (\ref{eq:sm-identity})),
\begin{align}
    \mathbb{E}_{{Y}_s}\left[\frac{1}{2} ||\mathsfit{s}_\theta({Y}_s, s) - \nabla \log q({Y}_s|{Y}_0)||_{gg^\top}^2 \,\bigg|\, {Y}_0=x\right].
    \label{eq:dsm}
\end{align}
The inference SDE is typically chosen so that ${Y}_s$ can be easily sampled, e.g. following ${\mathcal{N}}(\mu_s, \sigma_s^2)$, where $\mu_s$ and $\sigma_s$ are functions of ${Y}_0$ and $s$.
In this case, if we reparameterize ${Y}_s=\mu_s + \sigma_s \epsilon$ where $\epsilon\sim{\mathcal{N}}(0,\mathbf{I})$, then the score becomes
$\nabla \log q = - \frac{\epsilon}{\sigma_s}$.
Since $\sigma_s\rightarrow0$ as $s\rightarrow0$, this estimator normally has unbounded variance.
\citet{song2019generative, song2021score} propose to remedy
this by multiplying the DSM loss by $\sigma_s^2/g^2$ (assuming $g$ is a scalar for simplicity), so that the target has constant magnitude on average $\mathbb{E}[\frac{1}{2}||\sigma_s\mathsfit{s}_\theta + \epsilon||^2]$, which would result in a biased gradient estimate with much smaller variance.
We can debias this estimator by sampling $s\sim q(s)\propto g^2/\sigma_s^2$.
This ratio, however, is usally not normalizable in practice (as it integrates to $\infty$).
As an alternative, we consider the following unnormalized density
$\tilde{q}_\epsilon(s) = g^2(s_\epsilon)/\sigma_{s_\epsilon}^2$ for $s\in[0, s_\epsilon]$, and $\tilde{q}_\epsilon(s) = g^2(s)/\sigma_s^2$ for $s\in[s_\epsilon, T]$.
We experiment with this debiased procedure by sampling $s\sim q_\epsilon\propto\tilde{q}_\epsilon$, for $f$ and $g$ chosen to be the variance-preserving SDE.
$s_\epsilon$ is small so that the bias is negligible.

We train the model on MNIST \citep{lecun1998gradient} and CIFAR10 \citep{krizhevsky2009learning}.
We present the learning curves and the standard error of the estimate of the ELBO in Figure \ref{fig:debias}.
The lower bound is estimated using the Hutchinson trace estimator with $s$ sampled uniformly from $[0,T]$, with the same batch size, so the only thing that will affect the dispersion is the magnitude of $\nabla \mathsfit{s}_\theta$.
Since smaller values of $s$ are more likely to be sampled under $q_\epsilon$, the debiased model will see samples with less perturbation more often.
On the contrary, sampling $s$ uniformly will bias the model to learn from noisier data, causing the learned score to be smoother.
We also experiment with parameterizing $\mathsfit{s}_\theta$ vs parameterizing $a$.
We find the latter parameterization to be helpful since the relationship $\mathsfit{s}_\theta=g^{-1}a$ has the effect of negating the multiplier $\sigma_s$ in the reweighted loss, {\it i.e.}\@\xspace $\mathbb{E}[\frac{1}{2}||\frac{\sigma_s}{g}a + \epsilon||^2]$.
This is similar to the noise conditioning technique introduced in \citet{song2020improved}.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.49\linewidth]{figures/ODE-SDE-iterations.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/ODE-SDE-seconds.pdf}
    \caption{Neural ODE vs plug-in reverse SDE (denoising or slice score matching). The learning curves are presented as a function of iterations (left) and runtime (right) to emphasize the computational distinction between the two families of models.}
    \label{fig:ode-sde}
\end{figure}
\begin{figure}
    \centering
    \includegraphics[width=0.49\linewidth]{figures/mnist_bpd_aggregated.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/mnist_bpd_std_err_aggregated.pdf}
    \includegraphics[width=0.49\linewidth]{figures/cifar_bpd_aggregated.pdf}
    \hfill
    \includegraphics[width=0.49\linewidth]{figures/cifar_bpd_std_err_aggregated.pdf}
    \caption{Likelihood estimation on MNIST (first row) and CIFAR10 (second row). $\mathsfit{s}_\theta$ and $a$ denote which model we parameterize. Y-axes are bits-per-dim and the standard error of BPD of the test set.
    The debiased curves improve upon the original biased gradient estimator \citep{song2021score} since it maximizes a proper ELBO.
    Shaded area reflects the uncertainty estimated by 3 random seeds.}
    \label{fig:debias}
\end{figure}

\section{Related work}
\label{sec:related}

\paragraph{Diffusion-based generative models}
Our work lays a theoretical foundation for \citet{song2021score}, which recognizes that conditional denoising score matching \citep{song2019generative, song2020improved} and discrete-time diffusion-based generative models \citep{sohl2015deep, goyal2017variational, ho2020denoising} can be viewed as learning to revert an inference process (using the plug-in reverse SDE).
Different from \citet{ho2020denoising}, which shows the ELBO of discrete time diffusion process can be likened to DSM (Section 3.2 of the paper), we show that ISM loss naturally arises from the Fokker-Planck equation of the marginal density, via the Fenman-Kac representation and the Girsanov change of measure.
This line of work has been successfully applied to modeling high dimensional natural images \citep{dhariwal2021diffusion, saharia2021image}, audio \citep{kong2020diffwave}, 3D point cloud \citep{cai2020learning, zhou20213d}, and discrete data \citep{hoogeboom2021argmax}.

\paragraph{Time-reversal of diffusion processes}
Plenty of works have studied the reverse-time diffusion processes (\ref{eq:reverse-sde}), including \citet{anderson1982reverse, follmer1985entropy, elliott1985reverse, haussmann1986time}.
These are different from our marginal-equivalent (reverse) processes (\ref{eq:lambda-sde-pair}) when $\mathsfit{s}_\theta=\nabla \log q$, since the latter is related by the marginals only.

\paragraph{Score matching for energy-based models}
Besides the connection to diffusion models, score matching is also often used as a method for learning energy based models (EBM)---~see \citet{song2021train} for a comprehensive review on useful techniques---.
When used as an EBM, sampling from the conditional score model can be achieved by running the annealed Langevin diffusion \citep{neal2001annealed}, which is connected to free-energy estimaton in physics \citep{jarzynski1997equilibrium}, wherein the \emph{path integral} is essentially a Feynman Kac representation.

\paragraph{De Bruijn's identity}
To connect maximum likelihood and score matching, \citet{durkan2021maximum} shows that KL divergence can be represented as an integral of weighted Fisher divergence, generalizing the case of \citet{lyu2009interpretation} where the inference perturbation is a simple Brownian motion.
This type of formulas fall into the category of de Bruijn's identity \citep{cover1999elements} for relative entropy.
A similar differential form result can be found in \citet{wibisono2017information}.

\paragraph{Learning SDEs}
\citet{tzen2019neural, li2020scalable} also propose to learn a neural SDE by applying Girsanov's theorem.
The key difference is that they treat the SDE entirely as a latent variable, with an additional emission probability, whereas we use the Feynman-Kac formula to directly express the marginal density as an expectation, side-stepping the need to smooth out the density using the emission probability (which will be a Dirac point mass in our case).
In their case, the inference direction is the same as the generative direction, since they infer the latent SDE directly, whereas we apply Girsanov to the Feynman-Kac diffusion (opposite the generative direction).
\citet{xu2021infinitely} further apply neural SDE as an infinitely deep Bayesian neural network.

\section{Conclusion and Discussion}
In this work, we derive a general variational framework for estimating the marginal likelihood of continuous-time diffusion models.
This framework allows us to study a wide spectrum of models, including continuous-time normalizing flows and score-based generative models.
Using our framework, we show that performing score matching with a particular choice of mixture weighting is equivalent to maximizing a lower bound on the marginal likelihood of a family of plug-in reverse SDEs, of which the one used in \citet{song2021score} and the equivalent ODE are special cases.
Empirically, we validate our theory by monitoring the ELBO while performing score matching, and discuss the implication of the choice of mixture weighting and the potential of debiasing via non-uniform sampling.
We emphasize that our theory does not explain the impressive sample quality of this family of models, which is still an open research problem and we leave it for future work.

This work introduces a general framework to estimate the likelihood of diffusion-based models, which allows the parameters of both the generative and inference SDEs to be learned, using a numerical solver with constant memory cost (as per Remark \ref{rm:computation}).
The training time can be reduced via the connection to score matching and the reverse-time parameterization (\ref{eq:plug-in-reverse-forward}) as long as $f$ and $g$ take a simple form (so that ${Y}_t$ can be sampled without numerical integration).
For example, one can generalize the Ornstein-Uhlenbeck process to have non-linear (in time) $f$ and $g$, similar to the variance-presering SDE, by parameterizing the integral of $f$ using a monotone network \citep{sill1998monotonic,kay2000estimating,daniels2010monotone,huang2018neural}.
This has been explored in a concurrent work by \citet{kingma2021variational} in a different framework.

\section*{Acknowledgements}
We would like to thank
David Kanaa, Ricky Chen, Simon Verret, Rémi Piché-Taillefer, Alexia Jolicoeur-Martineau,  and Faruk Ahmed for giving their feedback on this manuscript.
We would also like to thank the INNF+ 2021 reviewers and NeurIPS 2021 reviewers for their constructive suggestions, which help us improve the clarity of the paper.
Chin-Wei is supported by the Google PhD fellowship.

We also acknowledge the Python community \citep{van1995python, oliphant2007python} for developing the tools that enabled this work, including
numpy \citep{oliphant2006guide,van2011numpy, walt2011numpy, harris2020array},
PyTorch \citep{paszke2019pytorch},
Matplotlib \citep{hunter2007matplotlib},
seaborn \citep{seaborn}, and
SciPy \citep{jones2014scipy}.