\section{Introduction}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Let $X$ be an irreducible projective variety, over an algebraically closed field $\mathbf{k}$ with $\operatorname{char}(\mathbf{k}) = 0$, and fix a big line bundle $L$ on $X$ with section ring
$$
R = R(X,L) := \bigoplus_{m\geq 0} \H^0(X,mL) \text{.}
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
The aim of this note is to study the expectations of the Duistermaat-Heckman measures, which are determined by \emph{linearly bounded above} filtrations of $R$.  In particular, we discuss topics that surround \cite[Theorem 1.11]{Boucksom:Chen:2011}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  The theory of the Duistermaat-Heckman measures and their moments appear in several
contexts.  On the one hand, there are the traditional applications within the theory of test configurations and questions that surround the K-stability of polarized projective varieties.  In more recent times, the overall structure of the distributions that surround these measures has been clarified a good deal.  As some works that are relevant to our viewpoint here, we mention \cite{Donaldson:2002} and \cite{Boucksom-Hisamoto-Jonsson:2016}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  At the same time, these concepts from toric geometry and Geometric Invariant Theory naturally arise in Diophantine arithmetic questions for projective varieties.  Such connections arise in the work of Faltings and W\"{u}stholz \cite{Faltings:Wustholz}.  More recently, these directions have been expanded upon in \cite{McKinnon-Roth}, \cite{Grieve:chow:approx}, \cite{Grieve:2018:autissier} and \cite{Grieve:Divisorial:Instab:Vojta}.  Aspects of those works build on earlier ideas from \cite{Evertse:Ferretti:2002} and \cite{Ferretti:2003}.  In a parallel direction, these invariants have applications to questions of arithmetic hyperbolicity and related topics.  Two such examples include \cite{Autissier:2011} and \cite{Ru:Vojta:2016}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  A key result that underlies each of these viewpoints points is the fact that the expectation of the Duistermaat-Heckman measure may be understood, via the theory of concave transforms, in terms of the limit expectations which are closely related to the theory of Chow weights for polarized projective varieties.  We state that result here as Theorem \ref{DH:Expect}.  It is of independent interest and has been applied on a number of different occasions (including \cite{Grieve:chow:approx}, \cite{Grieve:2018:autissier}, \cite{Grieve:Divisorial:Instab:Vojta} and \cite{Blum:Jonnson:2017}).  In Section \ref{restricted:volume:functions}, we make a related remark about the nature of the restricted volume functions.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Because of these applications, it is worthwhile to revisit \cite{Boucksom:Chen:2011}.  That work has been developed and pursued further in a related context \cite{Boucksom:Kuronya:Maclean:Szemberg:2015}.  In fact, Theorem \ref{DH:Expect} below, is indeed well-known but our treatment of that result here still complements the many existing more recent treatments of related concepts such as \cite{Boucksom-Hisamoto-Jonsson:2016}, \cite{Szek:2015} and \cite{Blum:Jonnson:2017}.  Similar considerations apply to Theorem \ref{DH:restricted:volume:thm}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  For more precise statements, fix $\mathcal{F}^\bullet = \mathcal{F}^\bullet R$
a \emph{linearly bounded above filtration} of $R$ (see Definition \ref{linearly:bounded:above}).  The \emph{vanishing numbers} of $\mathcal{F}^\bullet $, for $m \in \ZZ_{\geq 0}$, are the sequence of real numbers
$$
a_{\min}(mL) = a_0(mL) \leq \dots \leq a_{n_m}(mL) = a_{\max}(mL)
$$
which are defined by the condition that
$$
a_j(mL) = a_j(mL,\mathcal{F}^\bullet) :=
\inf
\{
t \in \RR : \operatorname{codim} \mathcal{F}^t \H^0(X, mL) \geq j+1
\} \text{.}
$$
In particular, we may consider the discrete measures that they determine
$$
\nu_m = \nu_m(t) := \frac{1}{h^0(X,mL)} \sum_{j = 0}^{n_m} \delta_{m^{-1} a_j(mL)}(t)\text{.}
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  In this note, our main goal is to give a reasonably self contained proof of Theorem \ref{DH:Expect} below.  This result is due to Boucksom and Chen \cite[Theorem 1.11]{Boucksom:Chen:2011} (see also \cite{Boucksom:Kuronya:Maclean:Szemberg:2015}).  A key tool is the theory of Okounkov bodies \cite{Lazarsfeld:Mustata:2009}.

\begin{theorem
\label{DH:Expect}
Let $\mathcal{F}^\bullet = \mathcal{F}^\bullet R$ be a linearly bounded above filtration of
$$
R(X,L) = \bigoplus_{m \geq 0} \H^0(X,mL)
$$
the section ring of $L$ a big line bundle on an irreducible projective variety $X$.  The discrete measures
$$
\nu_m = \nu_m(t) := \frac{1}{h^0(X,mL)} \sum_{j = 0}^{n_m} \delta_{m^{-1} a_j(mL)}(t) \text{, }
$$
which are determined by $\mathcal{F}^\bullet$,
converge weakly to  a limit measure
$$
\nu := \lim_{m \to \infty} \nu_m \text{.}
$$
Furthermore
$$
\mathbb{E}(\nu) = \lim_{m \to \infty} \mathbb{E}(\nu_m) \text{.}
$$
In other words, the limit of the expectations of the measures $\nu_m$ coincides with the expectations of the limit measure $\nu$.
\end{theorem}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  We prove Theorem \ref{DH:Expect} in Section \ref{DH:Proof}.  In doing so, we establish a handful of related observations that augment this circle of ideas (see Theorem \ref{DH:Expect:2} and Corollaries \ref{DH:filtered:OK:body:cor} and \ref{DH:Expect:Filtered:Volume}).  Finally, in Section \ref{restricted:volume:functions}, we given an interpretation in terms of the restricted volume functions (see Theorem \ref{DH:restricted:volume:thm}).

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. } {\bf Acknowledgements.}  This note was written while I was a postdoctoral fellow at Michigan State University.  I thank colleagues for comments and discussions on related topics.

\section{Filtered algebras}\label{Preliminiaries:Filtered:Algebras}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  In this section, we fix our conventions about filtrations of the section ring
$$
R(X,L) := \bigoplus_{m \geq 0} \H^0(X,mL) = \bigoplus_{m \geq 0} R_m \text{,}
$$
for $L$ a big line bundle on $X$, an irreducible projective variety over $\mathbf{k}$.
For the most part, our conventions and notations are consistent with \cite{Boucksom:Chen:2011} and \cite{Boucksom:Kuronya:Maclean:Szemberg:2015}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  By a \emph{filtration} of $R$, we mean an \emph{$\RR$-filtration}
$$\mathcal{F}^\bullet = \mathcal{F}^\bullet R = \left( \mathcal{F}^t R_m \right)_{t \in \RR, m \in \ZZ_{\geq 0}}$$
that has the following properties
\begin{itemize}
\item{
$\mathcal{F}^t R_m \subseteq R_m$, for all $t$;
}
\item{
$\mathcal{F}^s R_m \subseteq \mathcal{F}^t R_m$, when $s \geq t$;
}
\item{
$\mathcal{F}^t R_m = \bigcap_{s \leq t} \mathcal{F}^s R_m$, for $t \geq 0$;
}
\item{
$\mathcal{F}^0 R_m = R_m$ and $\mathcal{F}^t R_m = 0$, for $t \gg 0$;
}
\item{
$\mathcal{F}^s R_m \cdot \mathcal{F}^t R_n \subseteq \mathcal{F}^{s+t} R_{m+n} $; and
}
\item{
$\mathcal{F}^t R_m = \mathcal{F}^{t-\epsilon} R_m$,
}
\end{itemize}
for all $t \in \RR$, all $0 < \epsilon \ll 1$ and all $m,n \in \ZZ_{\geq 0}$.
Following the terminology of \cite{Boucksom:Chen:2011}, such filtrations are \emph{decreasing}, \emph{left-continuous} and \emph{multiplicative}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Put
$$
a_j(R_m,\mathcal{F}^\bullet) = a_j(m,\mathcal{F}^\bullet) := \inf \{ t \in \RR : \operatorname{codim} \mathcal{F}^t R_m \geq j + 1 \}\text{.}
$$
These are the \emph{vanishing numbers} of $R$, with respect to $\mathcal{F}^\bullet$.  They have the property that
$$
 a_0(m,\mathcal{F}^\bullet) \leq \dots \leq a_{n_m}(m,\mathcal{F}^\bullet)
$$
for
$
n_m := \dim R_m - 1.
$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
Furthermore, the non-increasing left-continuous step functions
$$
t \mapsto \dim_\mathbf{k} \mathcal{F}^t R_m\text{, }
$$
for $m \in \ZZ_{\geq 0}$, satisfy the condition that
$$
\dim_\mathbf{k} \mathcal{F}^t R_m = j
$$
if and only if
$$
t \in \ ]a_{j-1}(m,\mathcal{F}^\bullet), a_j(m,\mathcal{F}^\bullet)].
$$
In particular, it holds true that
$$
 \frac{d}{dt} \dim \mathcal{F}^t R_m = - \sum_{j=0}^{n_m} \delta_{a_j(m,\mathcal{F}^\bullet)}
$$
(in the sense of distributions).

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
For later use, we set
$$
a_{\min}(m,\mathcal{F}^\bullet) :=  a_0(m,\mathcal{F}^\bullet)\text{, }
$$
$$
a_{\max}(m,\mathcal{F}^\bullet) := a_{n_m}(m,\mathcal{F}^\bullet)
$$
and
$$
\operatorname{mass}_+(R_m,\mathcal{F}^\bullet) := \sum_{a_j(R_m,\mathcal{F}^\bullet) > 0} a_j(R_m,\mathcal{F}^\bullet) \text{.}
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
Our main interest here is to study those filtrations which are linearly bounded above.  We make that concept precise in the following way.

\begin{defn}[\cite{Boucksom:Chen:2011}]\label{linearly:bounded:above}  An $\RR$-filtration $\mathcal{F}^\bullet$ of $R$ is \emph{linearly bounded above} if there exists a constant $C > 0$ such that
$
a_{\max}(m,\mathcal{F}^\bullet) \leq C m
$
for all $m \in \ZZ_{\geq 0}$.
\end{defn}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  For the applications that we have in mind, the most natural class of examples of linearly bounded above filtrations are those which arise by considering orders of vanishing along divisors.  Indeed, we discuss these topics in further detail in Section \ref{restricted:volume:functions}.

\begin{example}
Let $L$ be a big line bundle on a normal (irreducible) projective variety $X$.  Let \emph{$E$ be a Cartier divisor over $X$} (compare with \cite[Definition 2.24]{Kollar:Mori:1998}).  By this we mean that $E$ is a nonzero effective Cartier divisor on some normal proper model of $X$.

Each such Cartier divisor $E$ over $X$ determines a linearly bounded above $\RR$-filtration
$\mathcal{F}^\bullet = \mathcal{F}^\bullet R$
of the section ring
$$
R(X,L) = \bigoplus_{ m \geq 0} \H^0(X,mL)\text{.}
$$
Indeed, such filtrations have the form
$$
\mathcal{F}^t R_m := \H^0(Y, m\pi^* L - t E)\text{, }
$$
for
$\pi \colon Y \rightarrow X$
some normal proper model with
$E \subseteq Y$.

That such filtrations are linearly bounded from above follows from the fact that
$$
\mathcal{F}^t R_m = 0
$$
for all $t > \gamma_{\mathrm{eff}} m$, where
$$\gamma_{\mathrm{eff}} := \inf \{ t : \pi^*L - t E  \text{ is pseudoeffective} \} < \infty \text{.}$$
Similar considerations apply to divisorial valuations over $X$ and, more generally, to Abhyankar valuations \cite[Proposition 2.12]{Boucksom:Kuronya:Maclean:Szemberg:2015}.
\end{example}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Finally, given a such a filtration
$\mathcal{F}^\bullet = \mathcal{F}^\bullet R \text{,}$
we set
$$
R^t_m := \mathcal{F}^{tm} R_m.
$$
In this way, we obtain graded subalgebras
$$
R^t_\bullet := \bigoplus_{m \geq 0} R^t_m \subseteq R
$$
for each $t \in \RR$.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
We also make explicit note of the following special case of \cite[Lemma 1.4]{Boucksom:Chen:2011}.  We include a proof for the sake of completeness.

\begin{lemma}[{\cite[Lemma 1.4]{Boucksom:Chen:2011}}]\label{a:max:lemma}
Let $\mathcal{F}^\bullet$ be a linearly bounded above filtration of the section ring
$$R(X,L) = \bigoplus_{m \geq 0} \H^0(X,mL) = \bigoplus_{m \geq 0} R_m \text{.}$$
Then
$$
a_{\max}(R,\mathcal{F}^\bullet) = \lim_{m \to \infty} \frac{ a_{\max}(R_m,\mathcal{F}^\bullet) }{ m } = \sup_{m \geq 1} \frac{a_{\max}(R_m,\mathcal{F}^\bullet)}{m}\text{.}
$$
\end{lemma}

\begin{proof}
Similar to \cite{Boucksom:Chen:2011}, our present assumptions on the filtration $\mathcal{F}^\bullet$ imply that the quantities $a_{\max}(R_m,\mathcal{F}^\bullet)$ are super-additive in $m$, for $m \gg 1$.  So, the desired results follow because of the fact that, for super-additive sequences, the equality
$$
a_{\max} = \lim_{m \to \infty} \frac{a_m}{m} = \sup_{m \geq 1} \frac{a_m}{m}
$$
holds true.
\end{proof}

\section{Okounkov bodies and concave transforms}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Let
$\mathcal{F}^\bullet = \mathcal{F}^\bullet R$
be a linearly bounded above filtration of $R(X,L)$, the
the section ring of $L$ a big line bundle on an irreducible projective variety $X$.  We assume the basic theory of Okounkov bodies from \cite{Lazarsfeld:Mustata:2009} and adopt similar notation.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Let $V_\bullet$ be a graded linear series of $L$.  Assume that $V_\bullet$ contains an ample series.  In particular, $V_\bullet$ is a graded $\mathbf{k}$-subalgebra of $R(X,L)$ and Condition (C), point (i), from \cite[Definition 2.9]{Lazarsfeld:Mustata:2009} holds true.  By fixing an \emph{admissible flag} of irreducible subvarieties
$$
Y_\bullet \colon Y_0 = X \supsetneq Y_1 \supsetneq \hdots \supsetneq Y_d = \{\mathrm{pt} \}
$$
we may associate to  $V_\bullet$, via the valuation like functions that are determined by $Y_\bullet$, the semigroup
$$
\Gamma(V_\bullet) \subseteq \ZZ_{\geq 0}^{d+1} \text{.}
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  In this context, the closed convex cone
$$
\Sigma(V_\bullet) \subseteq \RR^{d+1}\text{, }
$$
which is generated by $\Gamma(V_\bullet)$, has a compact convex basis
$$
\Delta(V_\bullet) := \Sigma(V_\bullet) \bigcap \left( \{1 \} \times \RR^d \right)\text{.}
$$
This is the \emph{Okounkov body} of $V_\bullet$.  It depends on the fixed choice of admissible flag $Y_\bullet$.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  On the other hand, following \cite{Boucksom:Chen:2011}, the \emph{concave transform} of $\mathcal{F}^\bullet$ is the concave function
$$
G_{\mathcal{F}^\bullet} \colon \Delta(L) \rightarrow [-\infty, \infty[
$$
which is defined by
$$
x \mapsto \sup \left\{t \in \RR : x \in \Delta \left(R^t_\bullet \right)\right\}\text{.}
$$
By way of the concave transform, we may define the \emph{filtered Okounkov body} of $\mathcal{F}^\bullet $.  This is the compact convex subset
$$
\Delta(\mathcal{F}^\bullet ) := \left\{ (x,t) \in \Delta(L) \times \RR \text{, } 0 \leq t \leq G_{\mathcal{F}^\bullet}(x) \right\} \subseteq \RR^{d+1} \text{.}
$$

\section{Proof of Theorem \ref{DH:Expect}}\label{DH:Proof}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }   The goal of this section is to establish Theorem \ref{DH:Expect}.  In doing so, we establish Theorem \ref{DH:Expect:2}, Corollaries \ref{DH:filtered:OK:body:cor} and \ref{DH:Expect:Filtered:Volume}.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  First, we establish Theorem \ref{DH:Expect:2} below.  The argument that we give follows closely those of \cite[Proof of Theorem 1.11]{Boucksom:Chen:2011} and \cite[Proof of Theorem 2.20]{Boucksom:Kuronya:Maclean:Szemberg:2015}.

\begin{theorem
\label{DH:Expect:2
Let $\mathcal{F}^\bullet = \mathcal{F}^\bullet R$ be a linearly bounded above filtration of $R(X,L)$,
the section ring of $L$ a big line bundle on an irreducible projective variety $X$.  The discrete measures
$$
\nu_m = \nu_m(t) := \frac{1}{h^0(X,mL)} \sum_{j = 0}^{n_m} \delta_{m^{-1} a_j(mL)}(t)\text{, }
$$
which are determined by $\mathcal{F}^\bullet$, converge weakly to  a limit measure $\nu$.
\end{theorem}

\begin{proof
Let
$d := \dim X\text{, }$
put
$$
\mu_m := \frac{h^0(X,mL)}{m^d} \nu_m
$$
and
$$
a_{\max}(R, \mathcal{F}^\bullet) = \sup_{m \geq 1} \frac{a_{\max}(m,\mathcal{F^\bullet})}{m}.
$$

Observe that $- \mu_m$ is the distributional derivative of the non-increasing left continuous step function
$$
g_m(t) := m^{-d} \dim \mathcal{F}^{mt} R_m = m^{-d} \dim R^t_m.
$$
Set
$$
g(t) := \operatorname{Vol} \left( \Delta(R_\bullet^t) \right)
$$
and recall that the graded subalgebras
$$
R^t_\bullet := \bigoplus_{m \geq 0} R^t_m = \bigoplus_{m \geq 0} \mathcal{F}^{tm} R_m\text{,}
$$
for
$$t < a_{\max}(R,\mathcal{F}^\bullet)\text{,}$$
contain an ample series \cite[Lemma 1.6]{Boucksom:Chen:2011}.
Thus, by the theory of Okounkov bodies, \cite[Lemma 2.12 and Theorem 2.13]{Lazarsfeld:Mustata:2009}, we obtain that
$$
\lim_{m\to \infty} g_m(t) = g(t).
$$
Furthermore, observe that the inequality
$$
0 \leq g_m(t) \leq m^{-d} h^0(X,mL)
$$
is uniformly bounded.

Thus, by dominated convergence, we obtain
$$g_m \to g$$
in $L^1_{\mathrm{loc}}(\RR)$.  In particular
$$
- \mu_m = \frac{d}{dt}(g_m)  \to \frac{dg}{dt}
$$
in the sense of distributions.

Set
$$
\mu := \left( G_{\mathcal{F}^\bullet} \right)_* \lambda \text{.}
$$
It remains to show that
$$
\frac{dg}{dt}  = - \mu.
$$
To this end, first observe that
$$
\lim_{s \to t^-} g(s) = \lambda \left( \{G_{\mathcal{F}^\bullet} \geq t \} \right) \text{.}
$$

On the other hand
$$
h(t) := \lambda \left( \{ G_{\mathcal{F}_\bullet} \geq t \} \right)
$$
and the discontinuity locus of $g(t)$ is at most countable.  In particular, we obtain the equality of distributions
$$
g = h \text{.}
$$

We now interpret these observations in terms of the concave transform $G_{\mathcal{F}^\bullet}$.  To this end, let $\lambda$ denote the restriction of the Lebesgue measure to the interior of the Okounkov body $\Delta(L)$ and consider its pushforward with respect to the concave transform
$$
\mu := \left( G_{\mathcal{F}^\bullet} \right)_* \lambda \text{.}
$$

Then note that,
on $\RR$, we have
$$
h(t) = \mu \left( \{x \in \RR : x \geq t \} \right)\text{;}
$$
it then follows from integration theory that
$$
\frac{dh}{dt} = - \mu\text{.}
$$
The proof is now complete.
\end{proof}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. } {\bf Remark.}  As noted in the proof of Theorem \ref{DH:Expect:2}, the measures $\mu_m$ and $\nu_m$ are related as
$$
\mu_m = \frac{h^0(X,mL)}{m^d} \nu_m.
$$
Furthermore,
$$
\mu = \lim_{m \to \infty} \mu_m  = \frac{\operatorname{Vol}_X(L)}{g!} \lim_{m \to \infty} \nu_m = \frac{\operatorname{Vol}_X(L)}{g!} \nu \text{.}
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Next, similar to \cite[Corollary 1.13]{Boucksom:Chen:2011}, we observe how the Euclidean volume of the filtered Okounkov bodies relate to the Duistermaat-Heckman measures.

\begin{corollary
\label{DH:filtered:OK:body:cor}  With the same assumptions and hypothesis of Theorem \ref{DH:Expect:2}, the expectation $\mathbb{E}(\nu)$ of the Duistermaat-Heckman measure $\nu$, determined by $\mathcal{F}^\bullet$, may be described as
$$
\mathbb{E}(\nu) = \frac{d!}{\operatorname{Vol}_X(L)} \int_0^\infty t \cdot \mathrm{d} \mu(t) \text{.}
$$
\end{corollary}
\begin{proof}
In our present notation
$$
\frac{ \operatorname{mass}_+(R_m,\mathcal{F}^\bullet)}{m^{d+1}} = \int_0^\infty t \cdot \mathrm{d}(\mu_m) \text{.}
$$
Furthermore,
$$
\operatorname{Vol}(\Delta(\mathcal{F}^\bullet R)) = \int_0^\infty t \cdot \left( G_{\mathcal{F}^\bullet} \right)_* \lambda
$$
and, by Lemma \ref{a:max:lemma}, we have
$$
a_{\max} := a_{\max}(R,\mathcal{F}^\bullet)  = \lim_{m \to \infty} \frac{a_{\max}(R_m,\mathcal{F}^\bullet)}{m}  = \sup_{m \geq 1} \frac{ a_{\max}(R_m, \mathcal{F}^\bullet)}{m} \text{.}
$$
Finally, both of the measures $\mu_m$ and $(G_{\mathcal{F}^\bullet})_* \lambda$ are supported on $]-\infty, a_{\max}]$ since
$$G_{\mathcal{F}^\bullet} \leq a_{\max} \text{.}$$

Because of these considerations, Corollary \ref{DH:filtered:OK:body:cor} follows by applying Theorem \ref{DH:Expect:2}.  Indeed, since
$$
\mu = \lim_{m \to \infty} \mu_m = \frac{\operatorname{Vol}_X(L)}{g!} \lim_{m \to \infty} \nu_m = \frac{\operatorname{Vol}_X(L)}{g!} \nu \text{, }
$$
we may apply Theorem \ref{DH:Expect:2}
 to any continuous function which has compact support and which coincides with $\max \{t, 0 \}$ on $]-\infty , a]$.
\end{proof}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Before establishing Theorem \ref{DH:Expect}, we mention one other interpretation of Theorem \ref{DH:Expect:2}.  This form of Theorem \ref{DH:Expect}, namely Corollary \ref{DH:Expect:Filtered:Volume} below, is used on a number of different occasions, for example, in \cite{Grieve:chow:approx}, \cite{Grieve:2018:autissier}, \cite{Grieve:Divisorial:Instab:Vojta} and \cite{Blum:Jonnson:2017}.  It is also mentioned in \cite{Boucksom:Chen:2011}.

\begin{corollary
\label{DH:Expect:Filtered:Volume}  Continuing within the context of Theorem \ref{DH:Expect}, the expectation of the measure $\nu$ coincides with the limit of the expectations of the measures $\nu_m$.  More explicitly
$$
\mathbb{E}(\nu) = \lim_{m \to \infty} \mathbb{E}(\nu_m) \text{.}
$$
\end{corollary}

\begin{proof}
To begin with
$$
\mathbb{E}(\nu_m)  = \frac{m^d}{h^0(X,mL)} \mathbb{E}(\mu_m)
 = \frac{\operatorname{mass}_+(R_m,\mathcal{F}^\bullet) }{ m h^0(X, mL) } \text{.}
$$
On the other hand, by Corollary \ref{DH:Expect:Filtered:Volume}, it then follows that
$$
\mathbb{E}(\nu)  = \frac{d!}{\operatorname{Vol}_X(L)} \mathbb{E}(\mu)
 = \frac{d!}{\operatorname{Vol}_X(L)} \lim_{m \to \infty} \mathbb{E}(\mu_m)
 = \lim_{m \to \infty } \mathbb{E}(\nu_m) \text{.}
$$
\end{proof}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  Finally, collecting the results of this section, we obtain Theorem \ref{DH:Expect}.

\begin{proof}[Proof of Theorem \ref{DH:Expect}]
The first assertion is a reformulation of Theorem \ref{DH:Expect:2}.  The second assertion is the conclusion of Corollary \ref{DH:Expect:Filtered:Volume}.
\end{proof}

\section{Interpretation in terms of restricted volume functions}\label{restricted:volume:functions}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  The goal of this section is to establish a slight variant of \cite[Theorem 2.24]{Boucksom:Kuronya:Maclean:Szemberg:2015} which is adequate for our purposes here.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  To begin with, let $X$ be an (irreducible) normal projective variety, of dimension $d$, and let $E$ be a irreducible and reduced divisor over $X$.  Fix a normal proper model
$$
\pi \colon Y \rightarrow X
$$
with the property that $E \subseteq Y$.  Given a big line bundle $L$ on $X$, we will denote by
$$h^0(X|E, L) = h^0(Y|E, \pi^* L) $$
the rank of the restriction map
$$
\H^0(Y,\pi^* L) \rightarrow \H^0(E, \pi^*L|E).
$$

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
In particular, we define the \emph{restricted volume} of $L$ along $E$ by
\begin{align*}
\operatorname{Vol}_{X|E} (L) & := \operatorname{Vol}_{Y|E}(\pi^* L) \\
 & = \limsup_{m \to \infty} \frac{(d-1)! }{m^{d-1 }} h^0(X|E,L)
\\&  = \limsup_{m \to \infty} \frac{(d-1)! }{m^{d-1 }} h^0(Y|E,\pi^*L)\text{.}
\end{align*}

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }
For later use, similar to \cite[Section 4.2]{Boucksom:Favre:Jonsson:2009}, we remark that in order to determine the nature of such restricted volume functions, we may replace $Y$ with some non-singular model
$$
\pi' \colon Y' \rightarrow Y
$$
and $E$ with its strict transform $E'$ with respect to $\pi'$.  Indeed, this follows in light of the relation
$$
\operatorname{Vol}_{Y|E}(\pi^*L) = \operatorname{Vol}_{Y'|E'}(L') \text{, }
$$
for $L'$ the pullback of $L$ to $Y'$ under the composition
$$
Y' \xrightarrow{\pi'} Y \xrightarrow{\pi} X \text{.}
$$
A key point here is that, by normality of $X$, the fibres of such models over $X$ are connected.

\vspace{3mm}\refstepcounter{subsection}\noindent{\bf \thesubsection. }  From this point of view, we wish to use the differentiability property of the volume function to establish the following variant of \cite[Theorem 2.24]{Boucksom:Kuronya:Maclean:Szemberg:2015}.  This result is of course well-known amongst experts.

\begin{theorem
\label{DH:restricted:volume:thm}
Let $L$ be a big line bundle on a normal irreducible projective variety $X$ and $E$ an irreducible and reduced Cartier divisor over $X$.  Choose a normal model
$$
\pi \colon Y \rightarrow X
$$
with the property that
$E \subseteq Y $
and let
$\mathcal{F}^\bullet = \mathcal{F}^\bullet R$
be the corresponding filtration of the section ring
$$
R = \bigoplus_{m \geq 0} R_m = \bigoplus_{m \geq 0} \H^0(X, mL)
$$
which is determined by $E$.  In this context, the limit measure $\nu$ may be described as
$$
\nu = \nu_{L,E} = d \frac{\operatorname{Vol}_{X|E}(\pi^*L - t E)}{\operatorname{Vol}(L)} \mathrm{d}t \text{.}
$$
Furthermore,
$$
a_{\max}(R,\mathcal{F}^\bullet) = \sup \{t > 0 : \pi^* L - t E \text{ is big} \} \text{.}
$$
\end{theorem}

\begin{proof}
The proof is similar to that which is given in \cite[Theorem 2.24]{Boucksom:Kuronya:Maclean:Szemberg:2015}.    Recall that the filtration
$
\mathcal{F}^\bullet = \mathcal{F}^\bullet R
$
is described as
$$
\mathcal{F}^t R_m = \H^0(Y, m \pi^* L - t E) \text{.}
$$
The assertion about $a_{\max}(R,\mathcal{F}^\bullet)$ is then clear.

Next, set
$v = E$
and
$$
\operatorname{Vol}(L, v \geq t) := \lim_{m \to \infty} \frac{d!}{m^d} h^0(Y, m \pi^* L - m t E) \text{.}
$$
In this notation
$$
\nu = \frac{\operatorname{Vol}(L , x \geq t) }{ \operatorname{Vol}(L) } \mathrm{d}t\text{.}
$$

In particular, by continuity of the volume function, the limit measure $\nu$ is absolutely continuous with respect to Lebesgue measure on the interval $(-\infty, a_{\max}(||L||, \mathcal{F}^\bullet))$.  Furthermore, $\nu$ is the weak derivative of
$$
- \frac{\operatorname{Vol}(L, x \geq t)}{\operatorname{Vol}(L) } = - \frac{\operatorname{Vol}(L - t E) }{ \operatorname{Vol}(L) } \text{.}
$$

The result then follows by the differentiability property of the volume function \cite{Boucksom:Favre:Jonsson:2009}.  Indeed, recall that the restricted volume functions are related as
$$
\operatorname{Vol}_{X|E}(L) = \operatorname{Vol}_{Y|E}(L) = \operatorname{Vol}_{Y'|E'}(L') \text{.}
$$
\end{proof}

\providecommand{\bysame}{\leavevmode\hbox to3em{\hrulefill}\thinspace}
\providecommand{\MR}{\relax\ifhmode\unskip\space\fi MR }
\providecommand{\MRhref}[2]{%
  \href{http://www.ams.org/mathscinet-getitem?mr=#1}{#2}
}
\providecommand{\href}[2]{#2}


\subsubsection{Learning the model - imagination} \label{sect:modellearning}

One plausible solution to that difficulty could be learning the model of the environment instead. The idea of learning the model of the environment from the sampled experience when its dynamics are not fully known is one of the most important concepts adapted in this thesis. However, this notion is not new. Current investigation in this area concentrates on application of variational autoencoders (VAEs) \citep{kingma2013vae}, recurrent neural networks (RNNs) \citep{medsker1999rnn}, and/or Bayesian methods. \citet{lenz2015deepmpc} introduced a novel robot control algorithm leveraging RNN that predicted the position of robot's parts. \citet{bellemare2013bayes} factors the state space to decompose model learning into smaller, more manageable sub-problems and applies Bayesian inference to predict future states.

A substantial breakthrough in learning the model of the environment in the RL setting and the algorithm that is most widely applicable was proposed by \citet{oh2015ataripred}. The paper presented two novel Encoding-Transformation-Decoding architectures to learn the transition probabilities function $T$. They first encode high-dimensional state $s_t$ using convolutional network \citep{lecun1999cnn}, then the transformation conditioned on $a_t$ is performed to convert a high-level encoding of the of the current state to a high-level encoding of the next state, and finally decoding using deconvolutional network \citep{zeiler2010decnn} is executed to decode high-level next-state features into the full representation of the next state $s_{t+1}$. The first architecture employs a simple feed-forward method and takes a fixed history of states as an input. The other takes advantage of LSTM cells \citep{hochreiter1997lstm} to capture the most relevant features from the past.

This work was later extended through a straightforward modification to support modelling of reward function $R$, and thus, to able to learn the full model of the environment \citep{leibfried2016rewardstatepred}. It was also incrementally improved by \citet{chiappa2017recatari} by the alteration of the original architecture and exploration of a few novel ideas.

Nevertheless, these model generation techniques have two major areas for improvement:
\begin{itemize}
    \item They do not utilise Markov property \eqref{eq:mdp} fully, treating the model learning as a highly sequential and non-stationary problem instead of transforming it into much easier, stationary scenario. It makes capturing all intrinsics of the environment much harder to learn.
    \item They utilise L1 or L2 objective to train the model and predict future states. L1/L2 loss penalises each difference between prediction and ground truth uniformly. Thus, it struggles to prioritise small but significant features over large but meaningless (e.g. slight ball location change over reduced saturation of a black background in Pong Atari game) and often produces blurry results. This is a well-known problem when applying standard classification/regression deep learning models for generative tasks.
\end{itemize}

\subsubsection{Applications of imagination} \label{sect:imagapp}

Historically, there has been a significant amount of work devoted to model-based reinforcement learning methods that took advantage of known environment dynamics. Firstly, as already mentioned in section \ref{sect:rl}, having that knowledge allows to directly solve the RL setting using simple linear algebra. If state/action is too intricate for that, we can apply traditional planning techniques. Additionally, Monte Carlo tree search \citep{browne2012mcts} can be deployed for more reliable evaluation of the future rewards. Several model-based techniques like DQN-cognate methods coupled with prior knowledge about the intrinsic model of the environment have already yielded outstanding effects \citep{silver2016alphago, silver2017alphago}.

Sadly, using these techniques directly on the imagination (approximated real environment) has been shown to perform very poorly \citep{talvitie2014error1, talvitie2015error2}. It is because approximations, by their nature, introduce a certain level of error. This error quickly compounds over time when performing long environment rollouts into the future. Utilising this approximation to estimate policy using planning-like methods is directly translating this compounded error into estimated policy causing it to be, in the best case, sub-optimal.

One of the first algorithms that managed to do that successfully was Dyna-Q \citep{sutton1990dyna}. It used a look-up table for imagination modelling. The imagination was later used to simulate and replace the real environment. Unfortunately, it did not scale to more complex problems due to the use of a finite look-up table. The high-level structure of the algorithm proposed in this thesis is inspired by the traditional Dyna-Q and can be partially viewed as its expansion to bigger and/or continuous state spaces.

Most recently, \citet{racaniere2017i2a} successfully employed model learning techniques described in section \ref{sect:modellearning} by combining them with the standard model-free methods in the decision making process. Their results showed improvements in the effective exploration and in the environments where long-term planning is crucial, even, when the imagination of the agent was far from perfect. Nevertheless, it did not significantly reduce the size of the system exposure necessary for training a well-performing agent.

Recent work that most closely resembles this thesis was done by \citet{venkatraman2016efficient1} and \citet{gu2016efficient2}. They also try to improve sample efficiency of RL by using Dyna-Q inspired algorithms. However, both of the studies adopt simple, often linear, methods to train the imagination. Hence, they can only be applied to specific domains where such methods are successful. Additionally, their data efficiency is not specifically better than the current model-free state-of-the-art Rainbow DQN \citep{hessel2017rainbow}.

\subsubsection{Model-free reinforcement learning} \label{sect:rlimplement}
Implementation started with the standard Deep Q-Network code following the original paper \citep{mnih2015dqn}. The initial structure consisted of 2 hidden layers with 24 nodes each to accustom for simpler environments. Although the most popular activation function for hidden layers of deep neural networks is rectifier linear unit (ReLU) ($f(x) = \max(0, x)$) introduced by \citet{nair2010relu}, from my experience it fairly often causes 'dead neurons' problem, i.e. input to the ReLU is negative (thus output is a constant $0$) causing lack of gradient flowing through the node during the backpropagation. Therefore, from the beginning, DQN was employing leaky rectifier linear units ($f(x) = \max(\alpha x, x)$) with default parameter $\alpha_{lrelu} = 0.2$ \citep{maas2013lrelu}. Starting configuration was using random minibatches of 32 experience samples from the replay buffer and Adam optimisation technique \citep{kingma2014adam} to derive an optimal set of network weights.

It was then debugged and hypertuned until it was able to solve both, previously mentioned, environments easily. Most interesting changes performed during the hypertuning regarded the batch size and the optimisation technique. Initially, the agent was not able to even reach close-to-optimal policies. A single change from Adam optimiser to the standard stochastic gradient descent without any additional acceleration (SGD) was enough to fix the issue. Deep reinforcement learning is known to suffer from high instability. Momentum-based optimisation may often exacerbate this problem. Additionally, SGD has been recently shown to have better generalisation properties \citep{wilson2017sgdgood}, what may be another reason for its superiority in this setting. Following this change, the agent became able to reach optimal-like behaviour; however, it was quickly forgetting what it has learned and collapsing back to sub-optimal policies. The solution was to increase the batch size, from 32 to 256 strongly. A batch made out of only 32 samples was often not diverse enough to represent the actual signal coming from the environment, thus again intensifying instabilities of deep RL.

Afterwards, the state-of-the-art Rainbow DQN was written inheriting the DQN structure and following improvements described by \citet{hessel2017rainbow}. It followed the same debugging and hypertuning procedure. The final Rainbow DQN hyperparameters are presented in \tablename{ \ref{table:rainbow}}. The same Rainbow DQN was then used both as a baseline and within the model-free module of the GAIRL framework.

\begin{table}[ht]
\centering
    \begin{tabular}{|l|l|} \hline
    Hyperparameter                               & Value              \\ \hline
    Hidden layers                                & $[24, 24]$         \\
    Hidden layers activation                     & Leaky ReLU         \\
    Leakiness parameter ($\alpha_{lrelu}$)       & $0.2$              \\
    Dropout probability                          & $0$                \\
    Final layer activation                       & Linear             \\
    Optimiser                                    & Gradient descent   \\
    Learning rate ($\alpha_{lr}$)                & $5 \times 10^{-3}$ \\
    Gradient clipping                            & $1$                \\
    Discount factor ($\gamma$)                   & $0.99$             \\
    Exploration strategy                         & $\epsilon\text{-greedy}$ \\
    Exploration $\epsilon$ decay                 & $1 \to 0.05 \text{ (linear)}$ \\
    Exploration decay start                      & $\num{1000} \text{ steps}$ \\
    Exploration decay length                     & $\num{10000} \text{ steps}$ \\
    Replay buffer size                           & $\num{10000}$      \\
    Replay batch size                            & $256$              \\
    Prioritisation $\epsilon$                    & $1 \times 10^{-5}$ \\
    Prioritisation $\alpha$                      & $0.6$              \\
    Prioritisation $\beta$ decay                 & $0.4 \to 1 \text{ (linear)}$ \\
    Prioritisation decay length                  & $\num{50000}$      \\
    Noisy networks $\delta_0$                    & $0.5$              \\
    Multi-step returns $n$                       & $3$                \\
    Online network update frequency              & $4$                \\
    Target network update frequency              & $500$              \\
    \hline
    \end{tabular}
\caption{Final parameters of the Rainbow DQN agent.}
\label{table:rainbow}
\end{table}

\subsubsection{Generative models} \label{sect:gansimpl}
Following the successful implementation, debugging, and hypertuning reinforcement learning agents, implementation of the second group of necessary algorithms~--~generative models~--~has started. All of the neural networks for the generative models were also initially configured to use Leaky ReLU activation for the hidden layers and Adam optimisation for training.

In the first step, I implemented standard GAN based on the original work by \citet{goodfellow2014gan}. Similarly to the DQN, it was then debugged and hypertuned; however, this time using standard MNIST dataset \citep{lecun1999cnn}. At first, both generator and discriminator seemed like they entirely lack gradient to progress, yet none of the well known GAN issues occurred. Interestingly, the problem did not lay purely in standard hyperparameters of the networks but in the weights initialisation. Originally, GAN's weights were initialised according to the same distribution as weights of RL algorithms. In DQN, weights have to be initialised to values with a mean $\mu = 0$ and a very low standard distribution to avoid divergence and exploding gradients that can be caused by recursive updates. For GANs, however, these values were too low to produce any meaningful signal.  Increasing the standard deviation of initial weights distribution solved the difficulty. Unfortunately, networks imbalance problem that was described in detail in section \ref{sect:gans} followed. Discriminator started to completely overwhelm generator causing the gradient of both networks to vanish. The solution to that was decreasing the number of discriminator training steps for each generator training step from $k = 5$ to $k = 1$.

After the original GAN was implemented, both Wasserstein GAN (WGAN) and Wasserstein GAN with gradient penalty (WGANGP) were built on top. Because tuning weight clipping constant $c$ in WGAN can be extremely hard and mundane, after making sure there are no bugs in the code, focus moved to the WGANGP implementation, without fully hypertuning the standard WGAN. Sadly, the WGANGP did not perfectly work on the dataset with the same hyperparameters as the standard GAN. The reason behind it was that, unlike the discriminator in the standard GAN, the critic in the WGANGP has to be much more powerful than the generator so it can fully converge to the real value of the Wasserstein distance between the real and the generated data. Slightly increasing the network size of the critic and the number of critic steps per single generator step from $k = 1$ to $k = 10$ caused the algorithm to perform much better. Nevertheless, it was still suffering from high variance and problematic convergence. As hypothesised and proved by the next runs of the model, this was caused by too high learning rate in both of the networks.

Furthermore, after implementation of the GAN, WGAN, and the WGANGP based on the original papers, they were expanded to allow for a conditional generation as proposed by \citet{mirza2014cgan}. Conditional versions of these algorithms did not need any additional hypertuning to learn probabilistic distribution underlying the MNIST dataset adequately.

As described in section \ref{sect:imagination}, an L1/L2 loss based model is also necessary for the expected reward generation. Therefore, in addition to GANs, a standard multilayer perceptron (MLP), optimising for the mean absolute error (L1 loss), has been implemented. L1 objective, instead of the most popular mean squared error (L2), has been chosen because it had been shown to work better for generative tasks \citep{zhao2017l1better}. The MLP was also initially debugged and hypertuned to fit the MNIST dataset. The comparison between implemented WGANGP and MLP on the MNIST dataset can be seen in figure \figurename{ \ref{fig:ganvsmlp}}. It is a great example of the superiority of GANs over traditional models in high-dimensional generative tasks.

What is also crucial, the property that all generative models followed equally, both for the MNIST dataset and as a part of the imaginative framework, was normalisation. Namely, all values from the data, both inputs and outputs, were scaled to fit the range $[0; 1]$. It further optimised their performance but also made it easier to interpret results. The fact that the output space is a range $[0; 1]$ is leveraged by employed evaluation metrics as described in section \ref{sect:metrics}.

Final hyperparameters of models that were optimised for the MNIST can be found in
\subsubsection{MountainCar}
\begin{figure}[htb]
    \centering
    \includegraphics[width=0.75\textwidth]{img/mountaincar.png}
    \caption{MountainCar environment.}
    \label{fig:mountaincar}
\end{figure}
MountainCar is based on a simple, low-dimensional setting. The agent controls a car that must drive up the steep slope. The car is not able to climb the hill directly. The agent has to learn to leverage the fact that it is situated in a valley and can use potential energy of the opposite slopes.

The state space is represented by two continuous variables: horizontal position of the car $x_t$, and velocity of the car across the car's axis $v_t$ ($s_t = (x_t, v_t) \in \mathbb{R}^2$). Action space is defined by a single choice: drive left, or drive right ($a_t \in \{-1, 1\}$). The agent gets a reward $r = -1$ for every time step until the episode is terminated (car drives up the right hill). An optimal policy should minimise the time it takes to reach the top of the hill, and thus maximise the cumulative reward obtained during the episode.

To solve the MountainCar environment, the algorithm has to be capable of handling continuous state space. Additionally, what makes it harder to learn is a very sparse reward signal. The agent has no information about the goal until it is reached. A meaningful signal occurs once per from 200 (close-to-optimal policy) to over 5000 (random policy) actions.

\subsubsection{Acrobot}
\begin{figure}[htb]
    \centering
    \includegraphics[width=0.75\textwidth]{img/acrobot.png}
    \caption{Acrobot environment.}
    \label{fig:acrobot}
\end{figure}

Acrobot, although still fairly low-dimensional, is harder and much more difficult task. The agent controls a two-link, under-actuated robot arm. The first (upper) joint, attached to the background, is out of control of the agent. The only controllable part is the torque of the lower joint of the robot. The goal is to balance the whole arm, so the tip of the second link swings above the episode termination line.

Six continuous variables represent the state space: sinus and cosine of the angle $\alpha_t$ between the first link and a horizontal line; sinus and cosine of the angle $\beta_t$ between the first and the second link; an angular velocity $\omega_t$ of the first joint; and an angular velocity $\theta_t$ of the second joint.
$$s_t = (\sin{\alpha_t}, \cos{\alpha_t}, \sin{\beta_t}, \cos{\beta_t}, \omega_t, \theta_t) \in [-1;1]^4 \times R^2$$
Action space is again defined by a single choice, although slightly modified: apply positive torque, no torque, or negative torque to the second joint ($a_t \in \{-1, 0, 1\}$). Similarly to the MountainCar, the agent gets a reward $r = -1$ for every time step until the episode finishes (tip of the second link swings above the termination line) to encourage policies that take the least amount of time to complete the episode.
\subsection{Model-free module}

The MFM consists of a standard model-free reinforcement learning algorithm; in this study, it is the state-of-the-art Rainbow DQN. It is the core decision-making module that models policy mapping $\pi : S \to A$ of the agent. The advantage of the imaginative framework is that it can leverage any other existing model-free algorithm if it is more suited for a given task (e.g. D4PG in environments with continuous action space) or if a model-free algorithm that overperforms Rainbow DQN is introduced in the future.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.75\textwidth]{img/mfm.png}
    \caption{Model-free module. It is in the essence a standard RL algorithm working exactly as described in sections \ref{sect:rl} and \ref{sect:rl}. It represents policy $\pi(s_t) = a_t$ that is incrementally updated based on the consequences of performed actions ($r_{t+1}, s_{t+1}$).}
    \label{fig:mfm}
\end{figure}

\subsection{Imagination module}

The IM is a crucial part of the whole framework. It is a trainable system that can simulate the behaviour of the environment dynamics, i.e. accurately approximate the transition probabilities function $T$ and the reward function $R$, similarly to the approach presented by \citet{leibfried2016rewardstatepred}. Similarly to the MFM, any generative model can be employed in place of the IM. Nonetheless, this thesis focuses on a specially designed IM to optimise its data efficiency. In depth design of the IM to create the GAIRL algorithm can be found in section \ref{sect:imagination}.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.75\textwidth]{img/im.png}
    \caption{Imagination module. Its goal is to replace the real environment by modelling its dynamics (functions $T$ and $R$). Thus, its input/output exactly resembles input/output of the environment shown in \figurename{~\ref{fig:rl}}.}
    \label{fig:im}
\end{figure}

\subsection{Model-free phase}

The MFP mostly follows the standard RL procedure taking advantage of the model-free RL algorithm employed within the MFM. Similarly to the standard RL, it is based entirely on the real environment. The only difference is that, in addition to the use of experience samples returned by the environment to improve the policy $\pi$ encapsulated in the MFM, it also stores these samples within the agent's memory $M$.

MFP is the only one out of three phases that requires the real environment to sample experience. Therefore, to limit the amount of the real experience that is required, it is also recommended to keep it as short as possible.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.85\textwidth]{img/mfp.png}
    \caption{Model-free phase. Excluding the memory that stores transition tuples ($s_t, a_t, r_{t+1}, s_{t+1}$), the whole concept was already visualised in \figurename{~\ref{fig:rl}}. MFM given a state $s_t$ produces an action $a_t$. The pair ($s_t, a_t$) is fed to the real environment (technically $s_t$ is not produced by the agent nor fed into the environment, environment simply \textbf{is} in the state $s_t$). Environment moves into the next state $s_{t+1}$ and produces the reward $r_{t+1}$. The pair ($r_{t+1}, s_{t+1}$) is then given as a feedback to the agent and together with the pair ($s_t, a_t$) it is saved in the memory. Finally, the next state $s_{t+1}$ becames the new current state ($s_t$) and the whole process starts from the beginning.}
    \label{fig:mfp}
\end{figure}

\subsection{Imagination training phase}

Imagination training phase is focused on using transitions samples ($s_t, a_t, r_{t+1}, s_{t+1}$) stored in the memory $M$ to train the IM to accurately follow the dynamics of the real environment (approximate functions $T$ and $R$) in a purely supervised learning manner.

Length of the ITP does not affect the real environment at all as it exploits past, memorised, experience instead. Hence, it can, and it should to produce more optimal results, run until the IM fully converges to the representation of the data that is stored in the memory.

\subsection{Imagination based phase}
Having fully trained imagination, we can start leveraging it. The IBP is focused on improving the agent's policy by letting MFM train and rollout experimental scenarios on the IM instead of making potentially expensive trial and error in the real world. In fact, the MFM is not even aware of the fact that its learning process has been moved onto the 'fake' environment instead of the real one.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.85\textwidth]{img/ibp.png}
    \caption{Imagination based phase. The concept works exactly like the one visualised in \figurename{ \ref{fig:rl}} or in \figurename{ \ref{fig:mfp}} (excluding memory). The only difference is that the real environment was replaced by the imagination module.}
    \label{fig:ibp}
\end{figure}

IBP, similarly to the ITP, should last until the agent's policy fully converges to the imaginative environment to extract as much signal, from experience gathered so far, as possible. Nevertheless, in practice, experiments focused on performing only three times more steps in the imagination than in the real environment, even, when the policy did not always fully converged during the imagination based phase.

\subsection{Summary}
The whole premise of the framework is to extract meaningful signal from the obtained experience as efficiently as possible. Imagination module serves as a way to generalise possible distribution of the world to create a safe, artificial, imaginative environment where the agent can learn and experiment without any risks associated with the actions in the real world.

The three phases work in a loop following Algorithm \ref{alg:gairl}. The loop part is crucial. In the first MFP, the agent usually performs completely random and experimental actions and is very likely to not reach more advanced environment states such as the next level in a video game. Going through the ITP and the IBP can allow it to master the first level of the game. However, it needs to go back to the real environment to explore newly reachable states to be able to imagine them in the second iteration accurately. This process should continue until the agent's policy fully converges to the given environment.

Convenient characteristic of the whole framework is that it can work for any model-free RL algorithm in place of the MFM and any generative module in place of the IM. Nevertheless, only the framework that specifically leverages GANs and Markov property for the data efficient IM is defined as GAIRL as explained in the next section.

\begin{algorithm}
\caption{Imagination for sample efficient reinforcement learning} \label{alg:gairl}
\begin{algorithmic}[1]
\Procedure{GAIRL}{$MFM, IM, env$}
    \State Initialise $MFM$
    \State Initialise $IM$
    \State Create and initialise $M$
    \While{true}
        \State Train $MFM$ on $env$ while collecting experience samples in $M$ (MFP)
        \If{$MFM$ converged on $env$}\\
~~~~~~~~~~~~~~\Return $MFM$ agent
        \EndIf
        \State Train $IM$ on the data from $M$ (ITP)
        \State Train $MFM$ on $IM$ (IBP)
    \EndWhile
\EndProcedure
\end{algorithmic}
\end{algorithm}
\subsection{Imagination performance} \label{sect:imagresults}

\subsubsection{Reward generation}

\figurename{ \ref{fig:reward}} shows the performance of the reward imagination submodule in terms of precision and recall in both environments. Both metrics, besides recall of Acrobot reward that requires twice as much experience, converge after $\num{120000}$ steps. For MountainCar, as it is a simple environment, the MLP can easily reach over $99\%$ recall and precision in a few GAIRL iterations. Acrobot is a bit more challenging and even converged imagination's precision can drop below $99\%$. Although results of this magnitude are sufficiently accurate, they could potentially be improved if more time was spent on hypertuning the machine learning model.

\begin{figure}[htb]
\centering
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/reward_precision.png}
    \caption{Recall on the test memory.}
\end{subfigure}%
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/reward_recall.png}
    \caption{Precision on the test memory.}
\end{subfigure}
\caption{Performance of the reward-generating MLP imagination submodule on experimental environments. The x-axis represents training steps performed solely in the ITP (in other phases the performance stays constant). Y-axis shows the value of an appropriate metric.}
\label{fig:reward}
\end{figure}

What can be intriguing, are big jumps about every $\num{60000}$ steps in the performance of the network as the training progresses. They are caused by the the main GAIRL algorithm loop. For every GAIRL iteration, agent gathers $\num{20000}$ samples of experience from the real environment ($\rho_1 = \num{20000})$ to then train the imagination for $\num{60000}$ gradient descent steps ($\rho_2 = \num{60000}$). Therefore, each multiple of $\num{60000}$ marks a point after which more real data enters the process helping the model to better generalise to unseen samples. I.e. in the first $\num{60000}$ steps imagination learns only based on $\num{16000}$ samples ($80\%$ of $\num{20000}$ because another $20\%$ belongs to the test memory), in the $\num{60000} - \num{120000}$ period based on $\num{32000}$, in the $\num{12000} - \num{180000}$ based on $\num{48000}$, and so on.

\subsubsection{State generation}

\begin{figure}[H]
\centering
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/wasserstein.png}
    \caption{Wasserstein loss for both Acrobot and MountainCar (WGANGP only).} \label{fig:wass}
\end{subfigure} \\%
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/state_loss_mountaincar.png}
    \caption{State generation MAE for MountainCar.} \label{fig:l1mountaincar}
\end{subfigure}%
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/state_loss_acrobot.png}
    \caption{State generation MAE for Acrobot.} \label{fig:l1acrobot}
\end{subfigure}
\caption{Performance of the state-generating imagination submodule on experimental environments. The x-axis represents training steps performed solely in the ITP (in other phases the performance stays constant). The y-axis shows the value of an appropriate metric.}
\label{fig:state}
\end{figure}

The effectiveness of the state imagination submodule is presented in \figurename{ \ref{fig:state}}. Both \figurename{ \ref{fig:l1mountaincar}} and \ref{fig:l1acrobot} show MAE for both WGANGP and MLP state generation. \figurename{ \ref{fig:wass}}, on the other hand, shows Wasserstein distance as approximated by the critic for WGANGP imagination only.

We can see that WGANGP performs worse than the MLP in terms of MAE. This is, however, expected behaviour. MLP optimises for the L1 loss, that is in essence MAE multiplied by a constant, directly. GANs superiority lays in the fact that they do not optimise towards minimising mean error on the individual level, but towards minimising the difference between data distributions. These plots only showcase that indeed both of the modules seem to model dynamics of the real distribution accurately. Using them for direct comparison of the models would be unfair for the WGANGP.

MLP reaches over $99.9\%$ accuracy on MountainCar and $99.5\%$ accuracy on Acrobot, even without much of hypertuning. Unlike with the reward, very high accuracy of state generation is crucial because the correctness of future states highly depends on the correctness of previous states. When making rollouts into the future using pure imagination, errors may compound. Given accuracy $a$ of the state generation model and rollout of length $\tau$, final state's accuracy may drop, in the worst case, to $a^{\tau}$ (nevertheless, this is also unlikely).

Even WGANGP that does not optimise for MAE reaches good results of over $97\%$ accuracy for both environments proving its convergence properties. This is also shown by the estimated Wasserstein distance between the generator distribution and the environment distribution that reaches values lower than $0.015$ for both environments (even below $0.005$ for MountainCar).

These results show a high promise of replacing the real environment with the imagination, especially considering the fact that not much time and resources have been spent on optimising the architecture and parameters used in the experiments.

\subsection{Data efficiency}
Data efficiency is the main problem targeted in this study. Results in section \ref{sect:imagresults} show that deep learning models can efficiently learn the dynamics of the real environment. The remaining question is whether these models are accurate enough to replace the real environment in the RL process.

\figurename{ \ref{fig:gairlresults}} presents results of both MLP-based and WGANGP-based imaginative framework in comparison to the imagination-free state-of-the-art algorithm. Both imaginative algorithms highly outperform imagination-free Rainbow DQN being more than twice as data efficient on both environments.

GAIRL requires even as much as over three times fewer experience samples than the framework with MLP-based imagination, and over 6 times less than the current state-of-the-art on the more complex Acrobot environment. Surprisingly, despite the simplicity of MountainCar setting, GAIRL also solves it most optimally.

What is also important, one of the concerns was that imagination-aided agent would be much less stable than the standard model-free algorithm. It is indeed the case on the simple MountainCar, especially for WGANGP-based framework. Remarkably, once the complexity of the environment increases (Acrobot), it is no longer the case.

\begin{figure}[htb]
\centering
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/mountaincar_results.png}
    \caption{Results on MountainCar}
\end{subfigure}%
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \includegraphics[width=1 \textwidth]{img/acrobot_results.png}
    \caption{Results on Acrobot}
\end{subfigure}
\caption{Performance of the MLP-based and WGANGP-based framework compared to the state-of-the-art baseline. The x-axis represents a size of experience from the real environment. The y-axis represents a mean reward from past 100 episodes. Black dashed line represents the top performance achieved by the state-of-the-art.} \label{fig:gairlresults}
\end{figure}

In addition, two-tailed Wilcoxon signed rank tests were performed to statistically compare obtained results. There were $N=15$ non-zero difference samples in every comparison. The critical value to make sure that the results are statistically significant  ($p \leq 0.05$) for $N=15$ samples is $\omega_c = 25$

Firstly, the comparison between the imagination-free state of the art, and the mlp-based imaginative algorithm was made. For the MountainCar environment, Wilcoxon test produced rank sums $T^+_{m_1} = 114$ and $T^-_{m_1} = 6$. Clearly, $\omega_{m_1} = T^-_{m_1} < \omega_c$. Similarly, in the Acrobot environment rank sums $T^+_{a_1} = 111$ and $T^-_{a_1} = 9$ were collected. Again, $\omega_{a_1} = T^-_{a_1} < \omega_c$. Therefore, the null hypothesis stating that the imaginative framework does not improve sample efficiency of the state of the art can be safely rejected.

Then, the focus moved to the analysis of GAIRL benefits over the more primitive MLP-based imaginative framework. Rank sums for the MountainCar environment in this scenario are $T^+_{m_2} = 82$ and $T^-_{m_2} = 38$. This time, $\omega_{m_2} > \omega_c$. Nevertheless, for the Acrobot, $T^+_{a_2} = 102$ and $T^-_{a_2} = 18$, so $\omega_{a_2} < \omega_c$. Thus, the superiority of GANs in the generative framework is statistically significant only for the more complicated environment. This result was somewhat expected; the central promise of GANs is to work much better on complex and high dimensional domains. The MLP should not be much worse in an elementary environment like MountainCar.

Full table with the data that was used to calculate test statistics is available in

\section*{Abstract}
\input{1_header/abstract.tex}
\pagebreak

\section*{Acknowledgments}
\input{1_header/acknowledgments.tex}
\pagebreak

\setcounter{tocdepth}{5}
\tableofcontents
\pagebreak

\section{Introduction} \label{sect:intro}
\input{1_header/introduction.tex}

\section{Background} \label{sect:background}

\subsection{Reinforcement learning} \label{sect:rl}
\input{2_body/background/rl.tex}

\subsection{Deep reinforcement learning} \label{sect:deeprl}
\input{2_body/background/deeprl.tex}

\subsection{Generative adversarial networks} \label{sect:gans}
\input{2_body/background/gans.tex}

\section{Related work} \label{sect:relatedwork}

\subsection{Model-based reinforcement learning} \label{sect:model}
\input{2_body/relatedwork/imaginationrl.tex}

\subsection{GANs in reinforcement learning} \label{sect:gansrl}
\input{2_body/relatedwork/gansrl.tex}

\section{Imaginative framework for data efficiency} \label{sect:framework}
\input{2_body/framework.tex}

\section{Imagination module structure for GAIRL} \label{sect:imagination}
\input{2_body/imagination.tex}

\section{Experimental setup} \label{sect:setup}

\subsection{Environments}
\input{2_body/setup/environments.tex}

\subsection{Implementation} \label{sect:implementation}
\input{2_body/setup/implementation.tex}

\subsection{Evaluation metrics} \label{sect:metrics}
\input{2_body/setup/metrics.tex}

\section{Results} \label{sect:results}
\input{2_body/results.tex}

\section{Discussion} \label{sect:discussion}
\input{2_body/discussion.tex}

\section{Conclusion} \label{sect:conclusion}
\input{2_body/conclusion.tex}

\bibliographystyle{agsm}