Minimax Distribution Estimation in Wasserstein Distance
The Wasserstein metric is an important measure of distance between probability distributions, with applications in machine learning, statistics, probability theory, and data analysis. This paper provides upper and lower bounds on statistical minimax rates for the problem of estimating a probability distribution under Wasserstein loss, using only metric properties, such as covering and packing numbers, of the sample space, and weak moment assumptions on the probability distributions.
\section{Preliminary Lemmas and Proof Sketch of Theorem~\ref{thm:expectation_bound_appendix}}
\label{sec:lemmas}

In this section, we outline the proof of Theorem~\ref{thm:expectation_bound}, our upper bound for the case of totally bounded metric spaces. The proof of the more general Theorem~\ref{thm:unbounded_upper_bound} for unbounded metric spaces, which is given in the next section, builds on this.

We begin by providing a few basic lemmas; these lemmas are not fundamentally novel, but they will be used in the subsequent proofs of our main upper and lower bounds, and also help provide intuition for the behavior of the Wasserstein metric and its connections to other metrics between probability distributions. The proofs of these lemmas are given later, in Appendix~\ref{app:proofs}. Our first lemma relates Wasserstein distance to the notion of resolution of a partition.

\begin{lemma}
Suppose $\S \in \SS$ is a countable Borel partition of $\Omega$. Let $P$ and $Q$ be Borel probability measures such that, for every $S \in \S$, $P(S) = Q(S)$. Then, for any $r \geq 1$, $W_r(P, Q) \leq \operatorname{Res}(\S)$.
\label{lemma:measures_agree_on_partition}
\end{lemma}

Our next lemma gives simple lower and upper bounds on the Wasserstein distance between distributions supported on a countable subset $\X \subseteq \Omega$, in terms of $\Diam(\X)$ and $\Sep(\X)$. Since our main results will utilize coverings and packings to approximate $\Omega$ by finite sets, this lemma will provide a first step towards approximating (in Wasserstein distance) distributions on $\Omega$ by distributions on these finite sets. Indeed, the lower bound in Inequality~\eqref{ineq:countable_support_bound} will suffice to prove our lower bounds, although a tighter upper bound, based on the upper bound in~\eqref{ineq:countable_support_bound}, will be necessary to obtain tight upper bounds.

\begin{lemma}
Suppose $(\Omega, \rho)$ is a metric space, and suppose $P$ and $Q$ are Borel probability distributions on $\Omega$ with countable support; i.e., there exists a countable set $\X \subseteq \Omega$ with $P(\X) = Q(\X) = 1$. Then, for any $r \geq 1$,
\begin{equation}
(\Sep(\X))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|
  \leq W_r^r(P,Q)
  \leq (\Diam(\X))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|.
\label{ineq:countable_support_bound}
\end{equation}
\label{lemma:countable_support_bound}
\end{lemma}

\begin{remark}
Recall that the term $\sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|$
in Inequality~\eqref{ineq:countable_support_bound} is the $\L_1$ distance
\[\|p - q\|_1 := \sum_{x \in \X} \left| p(x) - q(x) \right|\]
between the densities $p$ and $q$ of $P$ and $Q$ with respect to the counting measure on $\X$, and that this same quantity is twice the total variation distance
\[TV(P,Q) := \sup_{A \subseteq \Omega} \left| P(A) - Q(A) \right|.\]
Hence, Lemma~\ref{lemma:countable_support_bound} can be equivalently written as
\[\Sep(\Omega) \left( \|p - q\|_1 \right)^{1/r} \leq W_r(P,Q) \leq \Diam(\Omega) \left( \|p - q\|_1 \right)^{1/r}\]
and as
\[\Sep(\Omega) \left( 2 TV(P,Q) \right)^{1/r} \leq W_r(P,Q) \leq \Diam(\Omega) \left( 2 TV(P,Q) \right)^{1/r},\]
bounding the $r$-Wasserstein distance in terms of the $\L_1$ and total variation distance.
As noted in Example~\ref{ex:discrete_bound}, equality holds in \eqref{ineq:countable_support_bound} precisely when $\rho$ is the unit discrete metric given by $\rho(x,y) = 1_{\{x \neq y\}}$ for all $x,y \in \Omega$.

On metric spaces that are discrete (i.e., when $\Sep(\Omega) > 0$), the Wasserstein metric is (topologically) at least as strong as the total variation metric (and the $\L_1$ metric, when it is well-defined), in that convergence in Wasserstein metric implies convergence in total variation (and $\L_1$, respectively). On the other hand, on bounded metric spaces, the converse is true. In either of these cases, \emph{rates} of convergence may differ between metrics, although, in metric spaces that are both discrete \textit{and} bounded (e.g., any finite space), we have $W_r \asymp TV^{1/r}$.
\label{remark:Wasserstein_L1_TV}
\end{remark}

To obtain tight bounds as discussed below, we will require not only a partition of the sample space $\Omega$, but a nested sequence of partitions, defined as follows.

\begin{definition}[Refinement of a Partition, Nested Partitions]
Suppose $\S, \T \in \SS$ are partitions of $\Omega$. $\T$ is said to be a \emph{refinement of $\S$} if, for every $T \in \T$, there exists $S \in \S$ with $T \subseteq S$. A sequence $\{\S_k\}_{k \in \N}$ of partitions is called \emph{nested} if, for each $k \in \N$, $\S_k$ is a refinement of $\S_{k + 1}$,
\end{definition}

While Lemma~\ref{lemma:countable_support_bound} gave a simple upper bound on the Wasserstein distance, the factor of $\Diam(\Omega)$ turns out to be too large to obtain tight rates for a number of cases of interest (such as the $D$-dimensional unit cube $\Omega = [0,1]^D$, discussed in Example~\ref{ex:unit_cube_lower_bound}). The following lemma gives a tighter upper bound, based on a hierarchy of nested partitions of $\Omega$; this allows us to obtain tighter bounds (than $\Diam(\Omega)$) on the distance that mass must be transported between $P$ and $Q$.
Note that, when $K = 1$, Lemma~\ref{lemma:nested_partitions_Wasserstein_bound} reduces to a trivial combination of Lemmas~\ref{lemma:measures_agree_on_partition} and \ref{lemma:countable_support_bound}; indeed, these lemmas are the starting point for proving Lemma~\ref{lemma:nested_partitions_Wasserstein_bound} by induction on $K$.

Note that the idea of such a ``multi-resolution'' upper bound has been utilized extensively before, and numerous versions have been proven before (see, e.g., Fact 6 of \citet{do2011sublinearTimeEMD}, Lemma 6 of \citet{fournier2015rate}, or Proposition 1 of \citet{weed2017sharp}). Most of these versions have been specific to Euclidean space; to the best of our knowledge, only Proposition 1 of \citet{weed2017sharp} applies to general metric spaces. However, that result also requires that $(\Omega,\rho)$ is totally bounded (more precisely, that $m_x^\infty(P) < \infty$, for some $x \in \Omega$).

\begin{lemma}
Let $K$ be a positive integer. Suppose $\{\S_k\}_{k \in \N}$ is a nested sequence of countable Borel $\delta$-partitions of $(\Omega,\rho)$. Then, for any $r \geq 1$ and Borel probability measures $P$ and $Q$ on $\Omega$,
\begin{equation}
W_r^r(P, Q)
  \leq (\operatorname{Res}(\S_0))^r + \sum_{k = 1}^\infty \left( \operatorname{Res}(\S_k) \right)^r
                                       \left( \sum_{S \in \S_{k + 1}} \left| P(S) - Q(S) \right| \right).
\label{ineq:multiresolution_bound}
\end{equation}
\label{lemma:nested_partitions_Wasserstein_bound}
\end{lemma}

Lemma~\ref{lemma:nested_partitions_Wasserstein_bound} requires a sequence of partitions of $\Omega$ that is not only multi-resolution but also nested. While the $\epsilon$-covering number implies the existence of small partitions with small resolution, these partitions need not be nested as $\epsilon$ becomes small. For this reason, we now give a technical lemma that, given any sequence of partitions, constructs a \textit{nested} sequence of partitions of the same cardinality, with only a small increase in resolution.

\begin{lemma}
Suppose $\S$ and $\T$ are partitions of $(\Omega,\rho)$, and suppose $\S$ is countable. Then, there exists a partition $\S'$ of $(\Omega,\rho)$ such that:
\begin{enumerate}[label=\alph*)]
\item
$|\S'| \leq |\S|$.
\item
$\operatorname{Res}(\S') \leq \operatorname{Res}(\S) + 2\operatorname{Res}(\T)$.
\item
$\T$ is a refinement of $\S'$.
\end{enumerate}
\label{lemma:fine_refinement}
\end{lemma}

Lemmas~\ref{lemma:nested_partitions_Wasserstein_bound} and \ref{lemma:fine_refinement} are the main tools needed to bound the expected Wasserstein distance $\E[W_r^r(P, \hat P)]$ of the empirical distribution from the true distribution into a sum of its expected errors on each element of a nested partition of $\Omega$. Then, we will need to control the total expected error across these partition elements, which we will show behaves similarly to the $\L_1$ error of the standard maximum likelihood (mean) estimator a multinomial distribution from its true mean. Thus, the following result of \citet{han2015minimax} will be useful.

\begin{lemma}[Theorem 1 of \citep{han2015minimax}]
Suppose $(X_1,...,X_K) \sim \operatorname{Multinomial}(n,p_1,...,p_K)$. Let
\[Z := \|X - n p\|_1 = \sum_{k = 1}^K \left| X_k - n p_k \right|.\] Then,
$\E \left[ Z/n \right] \leq \sqrt{(K - 1)/n}$.
\label{lemma:multinomial_expectation}
\end{lemma}

Finally, we are ready to prove Theorem~\ref{thm:expectation_bound_appendix}.

\begin{customthm}{\ref{thm:expectation_bound}}
Let $(\Omega,\rho)$ be a metric space on which $P$ is a Borel probability measure. Let $\hat P$ denote the empirical distribution of $n$ IID samples $X_1,...,X_n \stackrel{IID}{\sim} P$, give by
\[\hat P(S) := \frac{1}{n} \sum_{i = 1}^n 1_{\{X_i \in S\}}, \quad \forall S \in \Sigma.\]
Then, for any sequence $\{\epsilon_k\}_{k \in [K]} \in (0,\infty)^K$ with $\epsilon_0 = \Diam(\Omega)$,
\[\E \left[ W_r^r(P, \hat P) \right] \leq \epsilon_K^r + \frac{1}{\sqrt{n}} \sum_{k = 1}^K \left( \sum_{j = k - 1}^K 2^{j - k} \epsilon_j \right)^r \sqrt{N(\epsilon_k) - 1}.\]
\label{thm:expectation_bound_appendix}
\end{customthm}

\begin{proof}
By recursively applying Lemma~\ref{lemma:fine_refinement}, there exists a sequence $\{\S_k\}_{k \in [K]}$ of partitions of $(\Omega,\rho)$ satisfying the following conditions:
\begin{enumerate}
\item
for each $k \in [K]$, $|\S_k| = N(\epsilon_k)$.
\item
for each $k \in [K]$,
$\displaystyle \operatorname{Res}(\S_k) \leq \sum_{j = k}^K 2^{j - k} \epsilon_j$.
\item
$\{S_k\}_{k \in [K]}$ is nested.
\end{enumerate}
Note that, for any $k \in [K]$, the vector $n\hat P(S)$ (indexed by $S \in \S_k$) follows an $n$-multinomial distribution over $|\S_k|$ categories, with means given by $P(S)$; i.e., \[(n\hat P(S_1),...,n\hat P(S_k)) \sim \operatorname{Multinomial}(n,P(S_1),...,P(S_k)).\]
Thus, by Lemma~\ref{lemma:multinomial_expectation}, for each $k \in [K]$,
\[\E \left[ \sum_{S \in \S_k} \left| P(S) - \hat P(S) \right| \right]
  \leq \sqrt{\frac{|\S_k| - 1}{n}}
  = \sqrt{\frac{N(\epsilon_k) - 1}{n}}.\]
Thus, by Lemma~\ref{lemma:nested_partitions_Wasserstein_bound},
\begin{align*}
\E \left[ W_r^r(P, Q) \right]
& \leq \E \left[ \epsilon_K^r + \sum_{k = 1}^K \left( \sum_{j = k}^K 2^{j - k} \epsilon_j \right)^r \left( \sum_{S \in \S_k} \left| P(S) - Q(S) \right| \right) \right] \\
& \leq \epsilon_K^r + \sum_{k = 1}^K \left( \sum_{j = k}^K 2^{j - k} \epsilon_j \right)^r \E \left[ \sum_{S \in \S_k} \left| P(S) - Q(S) \right| \right] \\
& \leq \epsilon_K^r + \frac{1}{\sqrt{n}} \sum_{k = 1}^K \left( \sum_{j = k}^K 2^{j - k} \epsilon_j \right)^r \sqrt{N(\epsilon_k) - 1}
\end{align*}
\end{proof}

\section{Proof Sketch of Theorem~\ref{thm:unbounded_upper_bound_appendix}}

In this section, we prove our more general upper bound, Theorem~\ref{thm:unbounded_upper_bound_appendix}, which applies to potentially unbounded metric spaces $(\Omega,\rho)$, assuming that $P$ is sufficiently concentrated (i.e., has at least $\ell > 0$ finite moments).

The basic idea is to partition the potentially unbounded metric space $(\Omega,\rho)$ into countably many totally bounded subsets $B_1,B_2,...$, and to decompose the Wasserstein error into its error on each $B_i$, weighted by the probability $P(B_i)$. Specifically, fixing an arbitrary base point $x_0$, $B_1,B_2,...$ will be spherical shells, such that $x_0 \in B_1$, and both the distance between $B_i$ and $x_0$, as well as the size (covering number) of $B_i$, increase with $i$. For large $i$, the assumption that $P$ has $\ell$ bounded moments implies (by Markov's inequality) that $P(B_i)$ is small, whereas, for small $i$, we adapt our previous result Theorem~\ref{thm:expectation_bound_appendix} in terms of the covering number.

To carry out this approach, we will need two new lemmas. The first decomposes Wasserstein distance into the sum of its distances on each $B_i$, and can be considered an adaptation of Lemma 2.2 of \citet{lei2018convergence} (for Banach spaces) to general metric spaces.

\begin{lemma}
Fix a reference point $x_0 \in \Omega$ and a non-decreasing real-valued sequence $\{w_k\}_{k \in \N}$ with $w_0 = 0$ and $\lim_{k \to \infty} w_k = \infty$. For each $k \in \N$, define
\[B_k := \left\{x \in \Omega : w_k \leq \rho(x_0, x) < w_{k + 1} \right\}.\]
Then, there exists a constant $C_r$ depending only on $r$ such that, for any Borel probability measures $P$ and $Q$ on $\Omega$,
\[W_r^r(P,Q)
  \leq C_r \sum_{k = 0}^\infty
    w_k^r
    \min \left\{ P(B_k), Q(B_k) \right\}
    W_r^r(P_{B_k},Q_{B_k})
  + \left| P(B_k) - Q(B_k) \right|.\]
where, for any sets $A, B \subseteq \Omega$,
\[P_A(B) = \frac{P(A \cap B)}{P(B)}\]
(under the convention that $\frac{0}{0} = 0$) denotes the conditional probability of $B$ given $A$, under $P$.
\label{lemma:sigma_finite_partition}
\end{lemma}

The second lemma is more nuanced variant of Lemma~\ref{lemma:multinomial_expectation} (albeit, leading to slightly looser constants). When $i$ is large the covering number of $B_i$ can become quite large, but the total probability $P(B_i)$ is quite small. Whereas Lemma~\ref{lemma:multinomial_expectation} depends only on the size of the partition, the following result will allow us to control the total error using both of these factors.

\begin{lemma}[Theorem 1 of \citet{berend2013binomialMAD}]
Suppose $X \sim \operatorname{Binomial}(n,p)$. Then, we have the bound
\begin{equation}
\E \left[ \left| X - n p \right| \right] \leq n \min \left\{ 2P(A), \sqrt{P(A)/n} \right\}.
\label{ineq:binomial_MAD}
\end{equation}
on the mean absolute deviation of $X$.
\label{lemma:binomial_MAD}
\end{lemma}

Finally, we are ready to prove our main upper bound result for unbounded metric spaces.

\begin{customthm}{\ref{thm:unbounded_upper_bound}}[General Upper Bound for Unbounded Metric Spaces]
Let $x_0 \in \Omega$ and suppose $m_{\ell,x_0}(P) \in [1, \infty)$. Let $J$ be a positive integer. Fix two non-decreasing real-valued sequences $\{w_k\}_{k \in \N}$ and $\{\epsilon_j\}_{j \in \N}$, of which $\{w_k\}_{k \in \N}$ is non-decreasing with $w_0 = 0$ and $\lim_{k \to \infty} w_k = \infty$ and $\{\epsilon_j\}_{j \in [J]}$ is non-increasing.
For each $k \in \N$, define
\[B_k(x_0) := \left\{ y \in \Omega : w_k \leq \rho(x_0, x) < w_{k + 1} \right\}.\]
Then,
\begin{align*}
\E \left[ W_r^r(P, \hat P) \right]
& \leq m_{\ell,x_0}^\ell \sum_{k \in \N} w_k^{-\ell} \left( \epsilon_J \right)^r + 2^r w_k^{r - \ell/2} \min \left\{ 2w_k^{-\ell/2}, \sqrt{\frac{1}{n}} \right\} \\
& \hspace{2cm} + \sum_{j = 1}^J \left( \sum_{t = j}^J 2^{J - t} \epsilon_t \right)^r \min \left\{ 2w_k^{-\ell}, \sqrt{\frac{w_k^{-\ell}}{n} N(B_k,\rho,\epsilon_j)} \right\}.
\end{align*}
\label{thm:unbounded_upper_bound_appendix}
\end{customthm}

\begin{proof}
As in the proof of Theorem~\ref{thm:expectation_bound_appendix}, by recursively applying Lemma~\ref{lemma:fine_refinement}, for each $k \in \N$, we can construct a nested sequence $\{\S_{k,j}\}_{j \in [J_k]}$ of partitions of $B_k$ such that, for each $j \in [J_k]$,
\begin{equation}
|\S_{k,j}| = N(B_k,\rho,\epsilon_{k,j})
  \quad \text{ and } \quad
  \operatorname{Res}(\S_{k,j}) \leq \sum_{t = 0}^j 2^t \epsilon_{k,j}.
  \label{eq:recursive_fine_refinement}
\end{equation}
Since each $P_{B_k}$ and $\hat P_{B_k}$ are supported only on $B_k$, plugging the bound Lemma~\ref{lemma:nested_partitions_Wasserstein_bound} into the bound in Lemma~\ref{lemma:sigma_finite_partition} gives
\begin{align*}
& W_r^r(P, \hat P) \\
& \leq \sum_{k \in \N} \min \left\{ P(B_k), \hat P(B_k) \right\} \left( \left( \operatorname{Res}(\S_{k,0}) \right)^r + \sum_{j = 1}^{J_k} \left( \operatorname{Res}(\S_{k,j}) \right)^r \sum_{S \in \S_{k,j + 1}} \left| P_{B_k}(S) - \hat P_{B_k}(S) \right| \right) \\
& \hspace{1cm} + 2^r w_k^r \left| P(B_k) - \hat P(B_k) \right| \\
& \leq \sum_{k \in \N}  2^r w_k^r \left| P(B_k) - \hat P(B_k) \right| + P(B_k) \left( \operatorname{Res}(\S_{k,0}) \right)^r + \sum_{j = 1}^J \left( \operatorname{Res}(\S_{k,j}) \right)^r \sum_{S \in \S_{k,j + 1}} \left| P(S) - \hat P(S) \right|.
\end{align*}
Since each $\hat P(S) \sim \operatorname{Binomial}(n, P(S))$, for each $k \in \N$ and $j \in [J_k]$, Lemma~\ref{lemma:binomial_MAD} followed by Cauchy-Schwarz gives
\begin{align*}
\E \left[ \sum_{S \in \S_{k,j}} \left| P(S) - \hat P(S) \right| \right]
& \leq \sum_{S \in \S_{k,j + 1}} \min \left\{ 2P(S), \sqrt{P(S)/n} \right\} \\
& \leq \min \left\{ 2P(B_k), \sqrt{\frac{P(B_k)}{n} |\S_{k,j}|} \right\}.
\end{align*}
Therefore, taking expectations (over $X_1,...,X_n$), applying Inequality~\ref{eq:recursive_fine_refinement}, and applying Lemma~\ref{lemma:binomial_MAD} once more gives
\begin{align*}
\E \left[ W_r^r(P, \hat P) \right]
& \leq \sum_{k \in \N} P(B_k) \left( \epsilon_{k,0} \right)^r + 2^r w_k^r \min \left\{ 2P(B_k), \sqrt{P(B_k)/n} \right\} \\
& \hspace{1cm} + \sum_{j = 1}^{J_k} \left( \sum_{t = 0}^j 2^t \epsilon_{k,j} \right)^r \min \left\{ 2P(B_k), \sqrt{\frac{P(B_k)}{n} N(B_k,\rho,\epsilon_{k,j + 1})} \right\}.
\end{align*}
Now note that, by Markov's inequality,
\begin{equation}
P(B_k)
  \leq \pr_{X \sim P} \left[ \rho(x_0, X) \geq w_k \right]
  = \pr_{X \sim P} \left[ \rho^\ell (x_0, X) \geq w_k^\ell \right]
  \leq \frac{m_{\ell,x_0}^\ell(P)}{w_k^\ell}.
\end{equation}
Therefore, assuming that each $m_{\ell,x_0}^\ell \geq 1$, so that $m_{\ell,x_0}^\ell \geq m_{\ell,x_0}^{\ell/2}$,
\begin{align*}
\E \left[ W_r^r(P, \hat P) \right]
& \leq m_{\ell,x_0}^\ell \sum_{k \in \N} w_k^{-\ell} \left( \epsilon_{k,0} \right)^r + 2^r w_k^r \min \left\{ 2w_k^{-\ell}, \sqrt{w_k^{-\ell}/n} \right\} \\
& \hspace{1cm} + \sum_{j = 1}^{J_k} \left( \sum_{t = 0}^j 2^t \epsilon_{k,j} \right)^r \min \left\{ 2w_k^{-\ell}, \sqrt{\frac{w_k^{-\ell}}{n} N(B_k,\rho,\epsilon_{k,j + 1})} \right\},
\end{align*}
proving the theorem.
\end{proof}

\section{Proofs of Lemmas}
\label{app:proofs}

\begin{customlemma}{\ref{lemma:measures_agree_on_partition}}
Suppose $\S \in \SS$ is a countable Borel partition of $\Omega$. Let $P$ and $Q$ be Borel probability measures such that, for every $S \in \S$, $P(S) = Q(S)$. Then, for any $r \geq 1$, $W_r(P, Q) \leq \operatorname{Res}(\S)$.
\end{customlemma}

\begin{proof}
This fact is intuitively obvious; clearly, there exists a transportation map $\mu$ from $P$ to $Q$ that moves mass only within each $S \in \S$ and therefore without moving any mass further than $\delta$. For completeness, we give a formal construction.

Let $\mu : \Sigma^2 \to [0,1]$ denote the coupling that is conditionally independent given any set $S \in \S$ with $P(S) = Q(S) > 0$ (that is, for any $A, B \in \Sigma$, $\mu(A \times B \cap S \times S) P(S) = P(A \cap S) Q(B \cap S)$).\footnote{The existence of such a measure can be verified by the Hahn-Kolmogorov theorem, similarly to that of the usual product measure (see, e.g., Section IV.4 of \citet{doob2012measure}).} It is easy to verify that $\mu \in \mathcal{C}(P,Q)$. Since $\S$ is a countable partition and $\mu$ is only supported on $\bigcup_{S \in \S} S \times S$,
\begin{align*}
W_r(P, Q)
& \leq \left( \int_{\Omega \times \Omega} \rho^r(x,y) \, d\mu(x,y) \right)^{1/r} \\
& = \left( \sum_{S \in \S} \int_{S \times S} \rho^r(x,y) \, d\mu(x,y) \right)^{1/r} \\
& \leq \left( \sum_{S \in \S} \int_{S \times S} \delta^r \, d\mu(x,y) \right)^{1/r} \\
& = \delta \left( \sum_{S \in \S} \mu(S \times S) \right)^{1/r}
  = \delta \left( \sum_{S \in \S} \frac{P(S) Q(S)}{P(S)} \right)^{1/r}
  = \delta \left( \sum_{S \in \S} Q(S) \right)^{1/r}
  = \delta.
\end{align*}
\end{proof}

\begin{customlemma}{\ref{lemma:countable_support_bound}}
Suppose $(\Omega, \rho)$ is a metric space, and suppose $P$ and $Q$ are Borel probability distributions on $\Omega$ with countable support; i.e., there exists a countable set $\X \subseteq \Omega$ with $P(\X) = Q(\X) = 1$. Then, for any $r \geq 1$,
\[(\Sep(\X))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|
  \leq W_r^r(P,Q)
  \leq (\Diam(\X))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|.\]
\end{customlemma}

\begin{proof}
The term $\sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right| = TV(P, Q)$ is precisely the (unweighted) amount of mass that must be transported to transform between $P$ and $Q$. Hence, the result is intuitively fairly obvious; all mass moved has a cost of at least $\Sep(\Omega)$ and at most $\Diam(\Omega)$. However, for completeness, we give a more formal proof below.

To prove the lower bound, suppose $\mu \in \Pi(P, Q)$ is any coupling between $P$ and $Q$. For $x \in \X$,
\[\mu(\{x\} \times \{x\}) + \mu(\{x\} \times (\Omega \sminus \{x\}))
  = \mu(\{x\} \times \Omega)
  = P(\{x\})\]
and, similarly,
\[\mu(\{x\} \times \{x\}) + \mu((\Omega \sminus \{x\}) \times \{x\})
  = \mu(\Omega \times \{x\})
  = Q(\{x\}).\]
Since $P(\{x\}), Q(\{x\}) \in [0,1]$, it follows that
\[\mu(\{x\} \times (\Omega \sminus \{x\})) + \mu(\mu((\Omega \sminus \{x\}) \times \{x\}))
  \geq \left| P(\{x\} - Q(\{x\}) \right|.\]
Therefore, since $\rho(x,y) = 0$ whenever $x = y$ and $\rho(x, y) \geq \Sep(\Omega)$ whenever $x \neq y$,
\begin{align*}
\int_{\Omega \times \Omega} \rho^r(x, y) \, d\mu(x,y)
& = \int_{\X \times \X} \rho^r(x, y) \, d\mu(x,y) \\
& = \sum_{x \in \X} \int_{\{x\} \times (\Omega \sminus \{x\})} \rho^r(x, y) \, d\mu(x,y)
  + \int_{(\Omega \sminus \{x\}) \times \{x\}} \rho^r(x, y) \, d\mu(x,y) \\
& \geq (\Sep(\Omega))^r \sum_{x \in \X} \mu(\{x\} \times (\Omega \sminus \{x\}))
  + \mu((\Omega \sminus \{x\}) \times \{x\}) \\
& \geq (\Sep(\Omega))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|.
\end{align*}
Taking the infimum over $\mu$ on both sides gives
\[(\Sep(\Omega))^r \sum_{x \in \X} \left| P(\{x\}) - Q(\{x\}) \right|
  \leq W_r^r(P, Q).\]
To prove the upper bound, since $\rho$ is upper bounded by $\Diam(\Omega)$, it suffices to construct a coupling $\mu$ that only moves mass into or out of each given point, but not both; that is, for each $x \in \X$,
\[\min\{\mu(\{x\} \times (\Omega \sminus \{x\})), \mu((\Omega \sminus \{x\}) \times \{x\})\}
  = 0.\]
One way of doing this is as follows. Fix an ordering $x_1,x_2,...$ of the elements of $\X$.
For each $i \in \N$, define
\[X_i := \sum_{\ell = 1}^i (P(x_\ell) - Q(x_\ell))_+
  \quad \text{ and } \quad
  Y_i := \sum_{\ell = 1}^i (Q(x_\ell) - P(x_\ell))_+,\]
and further define
\[j_i := \min \{ j \in \N : X_i \leq Y_j \}
  \quad \text{ and } \quad
  k_i := \min \{ k \in \N : X_j \geq Y_i \}.\]
Then, for each $i \in \N$, move $X_i$ mass from $\{x_1,...,x_i\}$ to $\{y_1,...,y_{j_i}\}$ and move $Y_i$ mass from $\{y_1,...,y_i\}$ to $\{x_1,...,x_{k_i}\}$. As $i \to \infty$, by construction of $X_i$ and $Y_i$, the total mass moved in this way is
\[\mu((\X \times \X) \sminus \{(x,x) : x \in \X\})
  = \lim_{i \to \infty} X_i + Y_i = \sum_{x \in \X} \left| P(x) - Q(x) \right|.\]
\end{proof}

\begin{customlemma}{\ref{lemma:nested_partitions_Wasserstein_bound}}
Let $K$ be a positive integer.
Suppose $\{\S_k\}_{k \in [K]}$ is a sequence of nested countable Borel partitions of $(\Omega,\rho)$, with $\S_0 = \Omega$. Then, for any $r \geq 1$ and any Borel probability distributions $P$ and $Q$ on $\Omega$,
\[W_r^r(P, Q)
  \leq (\operatorname{Res}(\S_K))^r + \sum_{k = 1}^K \left( \operatorname{Res}(\S_{k - 1}) \right)^r
                                       \left( \sum_{S \in \S_k} \left| P(S) - Q(S) \right| \right).\]
\end{customlemma}

\begin{proof}
Our proof follows the same ideas as and slightly generalizes of the proof of Proposition 1 in \citet{weed2017sharp}.
Intuitively, to prove Lemma~\ref{lemma:nested_partitions_Wasserstein_bound} it suffices to find a transportation map such that
For each $k \in [K]$, recursively define
\[P_k := P - \sum_{j = 0}^{k - 1} \mu_k
  \quad \text{ and } \quad
  Q_k := Q - \sum_{j = 0}^{k - 1} \nu_k,\]
where, for each $k \in [K]$, $\mu_k$ and $\nu_k$ are Borel measures on $\Omega$ defined for any $E \in \Sigma$ by
\[\mu_k(E) := \sum_{S \in \S_k : P_k(S) > 0} \left( P_k(S) - Q_k(S) \right)_+ \frac{P_k(E \cap S)}{P_k(S)}\]
and
\[\nu_k(E) := \sum_{S \in \S_k : Q_k(S) > 0} \left( Q_k(S) - P_k(S) \right)_+ \frac{Q_k(E \cap S)}{Q_k(S)}.\]

By construction of $\mu_k$ and $\nu_k$, each $\mu_k$ and $\nu_k$ is a non-negative measure and $\sum_{k = 1}^K \mu_k \leq P$ and $\sum_{k = 1}^K \nu_k \leq Q$. Furthermore, for each $k \in [K - 1]$, for each $S \in \S_k$, $\mu_{k + 1}(S) = \nu_{k + 1}(S)$, and
\[\mu_k(\Omega) = \nu_k(\Omega) \leq \sum_{S \in \S_k} \left| P(S) - Q(S) \right|.\]
Consequently, although $\mu$ and $\nu$ are not probability measures, we can slightly generalize the definition of Wasserstein distance by writing
\[W_r^r \left( \mu_k, \nu_k \right)
  := \mu(\Omega) \inf_{\tau \in \Pi \left( \frac{\mu_k}{\mu_k(\Omega)}, \frac{\nu_k}{\nu_k(\Omega)}\right)} \E_{(X,Y) \sim \mu} \left[ \rho^r \left( X, Y \right) \right]\]
(or $W_r^r(\mu_k, \nu_k) = 0$ if $\mu_k = \nu_k = 0$).
In particular, this is convenient because we one can easily show that, by construction of the sequences $\{P_k\}_{k \in [K]}$ and $\{Q_k\}_{k \in [K]}$,
\begin{equation}
W_r^r(P, Q)
  \leq W_r^r \left( P_K, Q_K \right) + \sum_{k = 1}^K W_r^r \left(\mu_k, \nu_k \right).
  \label{ineq:decomposition}
\end{equation}
For each $k \in [K]$, Lemma~\ref{lemma:countable_support_bound} implies that
\begin{align*}
W_r^r(\mu_k,\nu_k)
& \leq \sum_{S \in \S_{k - 1}} \left( \Diam(S) \right)^r \sum_{T \in \S_k : T \subseteq S} \left| P(T) - Q(T) \right| \\
& \leq \left( \operatorname{Res}(\S_{k - 1}) \right)^r \sum_{S \in \S_{k - 1}} \sum_{T \in \S_k : T \subseteq S} \left| P(T) - Q(T) \right| \\
& = \left( \operatorname{Res}(\S_{k - 1}) \right)^r \sum_{T \in \S_k} \left| P(T) - Q(T) \right|.
\end{align*}
Furthermore, for each $S \in \S_K$, $P_K = Q_K$, Lemma~\ref{lemma:measures_agree_on_partition} gives that
\[W_r^r \left( P_K, Q_K \right)
  \leq \left( \operatorname{Res}(\S_K) \right)^r\]
Plugging these last two inequalities into Inequality~\eqref{ineq:decomposition} gives the desired result:
\[W_r^r(P, Q)
  \leq \left( \operatorname{Res}(\S_K) \right)^r + \sum_{k = 1}^K \left( \operatorname{Res}(\S_{k - 1}) \right)^r \sum_{S \in \S_k} \left| P(S) - Q(S) \right|.\]
\end{proof}


\begin{customlemma}{\ref{lemma:fine_refinement}}
Suppose $\S$ and $\T$ are partitions of $(\Omega,\rho)$, and suppose $\S$ is countable. Then, there exists a partition $\S'$ of $(\Omega,\rho)$ such that:
\begin{enumerate}[label=\alph*)]
\item
$|\S'| \leq |\S|$.
\item
$\operatorname{Res}(\S') \leq \operatorname{Res}(\S) + 2\operatorname{Res}(\T)$.
\item
$\T$ is a refinement of $\S'$.
\end{enumerate}
\end{customlemma}
\begin{proof}
Enumerate the elements of $\S$ as $S_1,S_2,...$. Define $S_0' := \emptyset$, and then, for each $i \in \{1,2,...\}$, recursively define
\[S_i' := \left. \left( \bigcup_{T \in \T : T \cap S_i \neq \emptyset} T \right) \middle \sminus \left( \bigcup_{j = 1}^{i - 1} S_j' \right) \right.,\]
and set $\S' = \{S_1',S_2',...\}$. Clearly, $|\S'| \leq |\S|$ (equality need not hold, as we may have some $S_i' = \emptyset$).
By the triangle inequality, each
\[\Diam(S_i')
  \leq \Diam \left( \bigcup_{T \in \T : T \cap S_i \neq \emptyset} T \right)
  \leq \delta_\S + 2 \delta_T.\]
Finally, since $\T$ is a partition and we can write
\[S_i' = \left. \left( \bigcup_{T \in \T : T \cap S_i \neq \emptyset} T \right) \middle \sminus \left( \bigcup_{j = 1}^{i - 1} \bigcup_{T \in \T : T \cap S_j' \neq \emptyset} T \right) \right.,\]
$\T$ is a refinement of $\S'$.
\end{proof}

\section{Proof of Lower Bound}
In this section, we provide a proof of our main lower bound, Theorem~\ref{thm:Wasserstein_distribution_estimation_lower_bound} in the main text. The proof consists of two main steps. First, we show that the minimax error of estimation in Wasserstein distance can be lower bounded by a product of two terms, one depending on the packing radius $R$ and the other depending on the minimax risk of estimating a particular discrete (i.e., multinomial) distribution under $\L_1$ loss. The second step is then to apply a minimax lower bound on the risk of estimating a multinomial distribution under $\L_1$ loss. These two steps respectively rely on two lemmas, Lemma~\ref{lemma:wasserstein_projections} and Lemma~\ref{lemma:multinomial_minimax_lower_bound} given below.

The first lemma implies that, when a distribution $P$ is supported on a finite subset $\D$ of the sample space, then there exists an estimator $\hat P_\D$ of $\hat P$ that is supported on $\D$ is minimax optimal, up to a small constant factor. While this fact is relatively obvious for measure-theoretic metrics such as $\L_p$ distances, it is somewhat less obvious for Wasserstein distances, which also depend on metric properties of the space. This observation is key to lower bounding the minimax rate in terms of the minimax rate for estimating a discrete distribution.
\begin{lemma}[Wasserstein Projections]
Let $(\X,\rho)$ be a metric space and let $\D \subseteq \X$ be finite. Let $\P$ denote the family of all Borel probability distributions on $\X$, and let
\[\P_\D := \{P \in \P : P(\D) = 1\}\]
denote the set of distributions supported only on $\D$. Suppose $P \in \P_\D$ and $Q \in \P$. Then,
\[\mathop{\arg\!\min}_{\tilde Q \in \P_\D} W_r(Q, \tilde Q) \neq \emptyset
  \quad \text{ and, for any } \quad
  Q' \in \mathop{\arg\!\min}_{\tilde Q \in \P'} W_r(Q, \tilde Q),\]
we have $W_r(P, Q') \leq 2W_r(P, Q)$.
  \label{lemma:wasserstein_projections}
\end{lemma}

\begin{proof}
Let $\{\S_x\}_{x \in \D}$ denote the Voronoi diagram of $\X$ with respect to $\D$; that is, for each $x \in \D$, let
\[\S_x := \{y \in \X : x \in \mathop{\arg\!\min}_{z \in \D} \rho(x,y) \}.\]
Since $\{S_x\}_{x \in \D}$ is a finite cover of $\X$, we can disjointify it (see Remark~\ref{remark:disjointification}) while retaining the property that, for every $x \in \D$ and every $y \in \S_x$, $\rho(x,y) = \min_{z \in \D} \rho(z,y)$; hence, we assume without loss of generality that $\{\S_x\}_{x \in \D}$ is a partition of $\X$. Then, there is a unique distribution $Q' \in \P_D$ such that, for each $x \in \D$, $Q'(\{x\}) = Q(\S_x)$. It is easy to see by definition of the Voronoi diagram that $Q' \in \mathop{\arg\!\min}_{\tilde Q \in \P_\D} W_r(Q, \tilde Q)$; the unique transportation map $\mu_* \in \Pi(Q,Q')$ such that each $\mu(\S_x,\{x\}) = Q(\S_x)$ clearly minimizes
\[\E_{(X,Y) \sim \mu} \left[ \rho^r(X,Y) \right]\]
over all $\mu \in \bigcup_{\tilde Q \in \P_\D} \Pi(Q, \tilde Q)$. Moreover, since $P \in \P_D$, by the triangle inequality and the definition of $Q'$, $W_r(P, Q') \leq W_r(P, Q) + W_r(Q, Q') \leq 2 W_r(P, Q)$.
\end{proof}

The second lemma is a simple minimax lower bound for the problem of estimating the mean vector of a multinomial distribution, under $\L_1$ loss.

\begin{lemma}[Minimax Lower Bound for Mean of Multinomial Distribution]
Suppose $k \leq 32 n$. Let $p \in \Delta^k$, and suppose $X_1,...,X_n \stackrel{IID}{\sim} \operatorname{Categorical}(p_1,...,p_k)$ are distributed IID according to a categorical distribution on $[k]$, with mean vector $p$. Then, we have the following minimax lower bound for estimating $p$ under $\L^1$-loss:
\[\inf_{\hat p} \sup_{p \in \Delta^k} \E \left[ \|p - \hat p\|_1 \right] \geq \frac{3\log 2}{4096} \sqrt{\frac{k - 1}{n}},\]
where the infimum is taken over all estimators (i.e., all (potentially randomized) functions $\hat p : [k]^n \to \Delta^k$ of the data).
\label{lemma:multinomial_minimax_lower_bound}
\end{lemma}

Note that, while the above result is phrased for categorical distributions to simplify notation in the proof, the result is equivalent to a statement for multinomial distributions, since $\sum_{i = 1}^n X_i \sim \text{Multinomial}(n,p_1,...,p_k)$ and $X_1,...,X_n$ are assumed to be IID and therefore exchangeable.

\begin{proof}
We follow a standard procedure for proving minimax lower bounds based on Fano's inequality, as outlined in Section 2.6 of \citet{tsybakov2009introduction}.

Let $p_0 = \left( 1/k, ...., 1/k \right) \in \Delta^K$ denote the uniform vector in $\Delta^k$.
Let $\I := \left[ \lfloor \frac{k}{2} \rfloor \right]$. For each $j \in \I$, define $\phi_j : [k] \to \R^k$ by
\[\phi_j := 1_{\{2j - 1\}} - 1_{\{2j\}},\]
and, for each $\tau \in \{-1,1\}^\I$, define
\[p_\tau := p_0 + \frac{c}{k} \sum_{j \in \I} \tau_j \phi_j,\]
where
\[c = \frac{1}{16} \sqrt{\frac{k - 1}{n}\log 2} \leq \frac{1}{2}.\]
Note that, since $|c| \leq 1$ and, for each $j \in \I$, $\sum_{\ell \in [k]} \phi_j(\ell) = 0$, each $p_\tau \in \Delta^k$.
Observe that, for any $\tau,\tau' \in \{-1,1\}^\I$, we have
\[\|p_\tau - p_{\tau'}\|_1
  = \frac{4 c \omega(\tau,\tau')}{k},
  \quad \text{ where } \quad
  \omega(\tau,\tau') = \sum_{i \in \I} 1_{\{\tau_i \neq \tau_i'\}}\]
denotes the Hamming distance between $\tau$ and $\tau'$. By the Varshamov-Gilbert bound (see, e.g., Lemma 2.9 of \citet{tsybakov2009introduction}), there exists a subset $T \subseteq \{-1,1\}^\I$ such that $\log |T| \geq \frac{\lfloor k/2 \rfloor \log 2}{8}$ and, for every $\tau, \tau' \in T$,
\[\omega(\tau,\tau') \geq \frac{|\I|}{8}
  = \frac{\lfloor k/2 \rfloor}{8},
  \quad \text{ and hence } \quad
  \|p_\tau - p_{\tau'}\|_1 \geq c \frac{\lfloor k/2 \rfloor}{2k}.\]
Also, for any $\tau \in T$,
\begin{align*}
D_{KL}(p_\tau^n,p_0^n)
& = n D_{KL}(p_\tau,p_0) \\
& = n \sum_{j \in [k]} p_{\tau,j} \log \left( \frac{p_{\tau,j}}{p_{0,j}}\right) \\
& = n \sum_{j \in \I} p_{\tau,2j - 1} \log \left( \frac{p_{\tau,2j - 1}}{1/k} \right) + p_{\tau,2j} \log \left( \frac{p_{\tau,2j}}{1/k} \right) \\
& = \frac{n |\I|}{k} \left( (1 - c) \log \left( 1 - c \right) + (1 + c) \log \left( 1 + c \right) \right)
\end{align*}
One can check (e.g., by Taylor expansion) that, for any $c \in (0,1/2)$,
\[(1 - c) \log \left( 1 - c \right) + (1 + c) \log \left( 1 + c \right)
  < 2c^2.\]
Thus, since $|\I| \leq k/2$,
\[D_{KL}(p_\tau^n,p_0^n)
  \leq \frac{2 n |\I| c^2}{k}
  \leq n c^2.\]
It follows that from the choice of $c$ (and noting that, by the assumptions that $k \leq 32n$, $c \in (0,1/2)$) that
\[\frac{1}{|T|} \sum_{\tau \in T} D_{KL}(p_\tau^n, p_0^n)
  \leq nc^2
  \leq \frac{\lfloor k/2 \rfloor \log 2}{128}
  \leq \frac{1}{16} \log |T|.\]
Therefore, by Fano's method for lower bounds (see, e.g., Theorem 2.5 of \citet{tsybakov2009introduction}, with $\alpha = 1/16$ and
\[s := \frac{c}{16}
    \leq c \frac{\lfloor k/2 \rfloor}{4k}
    \leq \frac{1}{2} \|p_\tau - p_{\tau'}\|_1,\]
we have
\begin{align*}
\inf_{\hat p} \sup_{p \in \Delta^k} \E \left[ \|p - \hat p\|_1 \right]
& \geq \inf_{\hat p} \sup_{p \in \Delta^k} c \frac{\lfloor k/2 \rfloor}{4k} \pr \left[ \|p - \hat p\|_1 \geq c \frac{\lfloor k/2 \rfloor}{4k} \right] \\
& \geq c \frac{\lfloor k/2 \rfloor}{4k} \frac{3}{16} \\
& \geq \frac{3\log 2}{4096} \sqrt{\frac{k - 1}{n}}.
\end{align*}
\end{proof}

\begin{customthm}{\ref{thm:Wasserstein_distribution_estimation_lower_bound}}
Let $(\Omega,\rho)$ be a metric space, and let $\P$ denote the set of Borel probability measures on $(\Omega,\rho)$.
\[\inf_{\hat P : \X^n \to \P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r(P, \hat P(X_1,...,X_n)) \right]
  \geq c_r \sup_{k \in [32n]} R^r(k) \sqrt{\frac{k - 1}{n}},\]
where
\[c_r = \frac{3\log 2}{4096\cdot 2^r}.\]
is independent of $n$ and the infimum is taken over all estimators (i.e., all (potentially randomized) functions $\hat P : \X^n \to \P$ of the data).
\label{thm:Wasserstein_distribution_estimation_lower_bound_appendix}
\end{customthm}

\begin{proof}

Let $k \leq 32n$, and let $\D$ be an $R(k)$-packing $\D$ of $(\Omega,\rho)$ with $|\D| = k$. Let $\P_\D$ denote the class of (discrete) distributions over $\D$.
By Lemma~\ref{lemma:countable_support_bound}, Lemma~\ref{lemma:wasserstein_projections}, Lemma~\ref{lemma:multinomial_minimax_lower_bound}, and the definition of the packing radius (in that order)
\begin{align*}
\inf_{\hat P : \X^n \to \P} \sup_{P \in \P} \E \left[ W_r^r(\hat P, P) \right]
& \geq \left( \Sep(\D) \right)^r \inf_{\hat P : \X^n \to \P} \sup_{P \in \P} \E \left[ \|\hat P - P\|_1 \right] \\
& \geq \left( \Sep(\D) \right)^r \inf_{\hat P : \X^n \to \P} \sup_{P \in \P_\D} \E \left[ \|\hat P - P\|_1 \right] \\
& \geq \left( \frac{\Sep(\D)}{2} \right)^r \inf_{\hat P : \X^n \to \P_\D} \sup_{P \in \P_\D} \E \left[ \|\hat P - P\|_1 \right] \\
& \geq \frac{3\log 2}{4096 \cdot 2^r} \left( \Sep(\D) \right)^r \sqrt{\frac{|\D| - 1}{n}} \\
& \geq \frac{3\log 2}{4096\cdot 2^r} R^r(k) \sqrt{\frac{k - 1}{n}}.
\end{align*}
The theorem follows by taking the supremum over $k \leq 32n$ on both sides.
\end{proof}










\section{Introduction}

The Wasserstein metric is an important measure of distance between probability distributions, based on the cost of transforming either distribution into the other through mass transport, under a base metric on the sample space. Originating in the optimal transport literature,\footnote{The Wasserstein metric has been variously attributed to Monge, Kantorovich, Rubinstein, Gini, Mallows, and others; see Chapter 3 of \citep{villani2008optimalTransport} for detailed history.}
the Wasserstein metric has, owing to its intuitive and general nature, been utilized in such diverse areas as probability theory and statistics, economics, image processing, text mining, robust optimization, and physics~\citep{villani2008optimalTransport,fournier2015rate,esfahani2015robustOptimization,gao2016distributionallyRobust}.

In the analysis of image data, the Wasserstein metric has been used for various tasks such as texture classification and face recognition~\citep{sandler2011NMFImageAnalysis}, reflectance interpolation, color transfer, and geometry processing~\citep{solomon2015imageOptimalTrans}, image retrieval~\citep{rubner2000imageRetrieval},
and image segmentation~\citep{ni2009imageSegmentation}, and, in the analysis of text data, for tasks such as document classification~\citep{kusner2015documentDistances} and machine translation~\citep{zhang2016machineTranslation}.

In contrast to a number of other popular notions of dissimilarity
between probability distributions, such as $\L_p$ distances or Kullback-Leibler and other $f$-divergences~\citep{morimoto1963divergences,csiszar1964divergences,ali1966divergences}, which require distributions to be absolutely continuous with respect to each other or to a base measure, Wasserstein distance is well-defined between \emph{any} pair of probability distributions over a sample space equipped with a metric.\footnote{Hence, we use ``distribution estimation'' in this paper, rather than the more common ``density estimation''.}
As a particularly important consequence, Wasserstein distances between discrete (e.g., empirical) distributions and continuous distributions are well-defined, finite, and informative (e.g., can decay to $0$ as the distributions become more similar).

Partly for this reason, many central limit theorems and related approximation results~\citep{ruschendorf1985wasserstein,johnson2005central,chatterjee2008normalApproximation,rio2009upper,rio2011asymptotic,chen10SteinsMethod,reitzner2013central} are expressed using Wasserstein distances.
Within machine learning and statistics, this same property motivates a class of so-called \emph{minimum Wasserstein distance estimates}~\citep{del1999CLT,del2003correction,bassetti2006minimum,bernton2017inferenceUsingWasserstein} of distributions, ranging from exponential distributions~\citep{baillo2016exponentialWasserstein} to more exotic models such as restricted Boltzmann machines (RBMs)~\citep{montavon2016wassersteinRBMs} and generative adversarial networks (GANs)~\citep{arjovsky2017wassersteinGAN}.
This class of estimators also includes $k$-means and $k$-medians, where the hypothesis class is taken to be discrete distributions supported on at most $k$ points~\citep{pollard1982quantization}; more flexible algorithms such as hierarchical $k$-means~\citep{ho2017multilevel} and $k$-flats~\citep{tseng2000kFlats} can also be expressed in this way, using a more elaborate hypothesis classes. PCA can also be expressed and generalized to manifolds using Wasserstein distance minimization~\citep{boissard2015template}.
These estimators are conceptually equivalent to empirical risk minimization, leveraging the fact that Wasserstein distances between the empirical distribution and distributions in the relevant hypothesis class are well-behaved.
Moreover, these estimates often perform well in practice because they are free of both tuning parameters and strong distributional assumptions.

For many of the above applications, it is important to understand how quickly the empirical distribution converges to the true distribution in Wasserstein distance, and whether there exist distribution estimators that converge more quickly. For example, \citet{canas2012learning} use bounds on Wasserstein convergence to prove learning bounds for $k$-means, while \citet{arora2017generalization} used the slow rate of convergence in Wasserstein distance in certain cases to argue that GANs based on Wasserstein distances fail to generalize with fewer than exponentially many samples in the dimension.

To this end, the {\bf main contribution} of this paper is to identify, in a wide variety of settings, the minimax convergence rate for the problem of estimating a distribution using Wasserstein distance as a loss function. Our setting is very general, relying only on metric properties of the support of the distribution and the number of finite moments the distribution has; some diverse examples to which our results apply are given in Section~\ref{sec:examples}.
Specifically, we assume only that the distribution is has some number of finite moments in a given metric. We then prove bounds on the minimax convergence rates of distribution estimation, utilizing covering numbers of the sample space for upper bounds and packing numbers for lower bounds. It may at first be surprising that positive results can be obtained under such mild assumptions; this highlights that the Wasserstein metric is quite a weak metric (see our Lemma 11
and the subsequent remark for discussion of this). Moreover, our results imply that, without further assumptions on the population distribution, the empirical distribution is typically minimax rate-optimal. Note that, while there has been previous work on upper bounds (discussed in Section~\ref{sec:related_work}), this paper is the first to study minimax lower bounds for this problem.

\textbf{Organization: }
The remainder of this paper is organized as follows.
Section~\ref{sec:notation} provides notation required to formally state both the problem of interest and our results, while Section~\ref{sec:related_work} reviews previous work studying convergence of distributions in Wasserstein distance.
Sections~\ref{sec:upper_bounds} and \ref{sec:lower_bounds} respectively contain our main upper and lower bound results. Since the proofs of the upper bounds, are fairly long, Appendices A and B provide high-level sketches of the proofs, followed by detailed proofs in Appendix C. The lower bound is proven in Appendix D
Finally, in Section~\ref{sec:examples}, we apply our upper and lower bounds to identify minimax convergence rates in a number of concrete examples.
Section~\ref{sec:conclusion} concludes with a summary of our contributions and suggested avenues for future work.

\section{Notation and Problem Setting}
\label{sec:notation}

For any positive integer $n \in \N$, $[n] = \{1,2,...,n\}$ denotes the set of the first $n$ positive integers. For sequences $\{a_n\}_{n \in \N}$ and $\{b_n\}_{n \in \N}$ of non-negative reals, $a_n \lesssim b_n$ and, equivalently $b_n \gtrsim a_n$, indicate the existence of a constant $C > 0$ such that $\limsup_{n \to \infty} \frac{a_n}{b_n} \leq C$. $a_n \asymp b_n$ indicates $a_n \lesssim b_n \lesssim a_n$.

\subsection{Problem Setting}

For the remainder of this paper, fix a metric space $(\Omega,\rho)$, over which $\Sigma$ denotes the Borel $\sigma$-algebra, and let $\P$ denote the family of all Borel probability distributions on $\Omega$. The main object of study in this paper is the Wasserstein distance on $\P$, defined as follows:
\begin{definition}[$r$-Wasserstein Distance]
Given two Borel probability distributions $P$ and $Q$ over $\Omega$ and $r \in [1,\infty)$, the $r$-\emph{Wasserstein distance} $W_r(P,Q) \in [0,\infty]$ between $P$ and $Q$ is defined by
\[W_r(P,Q)
  := \inf_{\mu \in \Pi(P,Q)} \left( \E_{(X,Y) \sim \mu} \left[ \rho^r \left( X, Y \right) \right] \right)^{1/r},\]
where $\Pi(P,Q)$ denotes all couplings between $X \sim P$ and $Y \sim Q$; that is,
\[\Pi(P,Q)
  := \left\{ \mu : \Sigma^2 \to [0,1] \middle| \text{ for all } A \in \Sigma,
             \mu(A \times \Omega) = P(A) \text{ and } \mu(\Omega \times A) = Q(A) \right\},\]
is the set of joint probability measures over $\Omega \times \Omega$ with marginals $P$ and $Q$.
\end{definition}
Intuitively, $W_r(P,Q)$ quantifies the $r$-weighted total cost of transforming mass distributed according to $P$ to be distributed according to $Q$, where the cost of moving a unit mass from $x \in \Omega$ to $y \in \Omega$ is $\rho(x,y)$. $W_r(P,Q)$ is sometimes defined in terms of equivalent (e.g., dual) formulations; these formulations will not be needed in this paper.
$W_r$ it is symmetric in its arguments and satisfies the triangle inequality, and, for all $P \in \P$, $W_r(P, P) = 0$. Thus, $W_r$ is always a pseudometric. Moreover, it is a proper metric (i.e., $W_r(P,Q) = 0 \Rightarrow P = Q$) if and only if $\rho$ is as well.

This paper studies the following problem:

\textbf{Formal Problem Statement:} Suppose $(\Omega,\rho)$ is a known metric space. Suppose $P$ is an unknown Borel probability distribution on $\Omega$, from which we observe $n$ IID samples $X_1,...,X_n \stackrel{IID}{\sim} P$. We are interested in studying the minimax rates at which $P$ can be estimated from $X_1,...,X_n$, in terms of the ($r^{th}$ power of the) $r$-Wasserstein loss. Specifically, we are interested in deriving finite-sample upper and lower bounds, in terms of only properties of the space $(\Omega,\rho)$, on the quantity
\begin{equation}
\inf_{\hat P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r \left( P, \hat P(X_1,...,X_n) \right) \right],
\label{exp:minimax_error}
\end{equation}
where the infimum is taken over all estimators $\hat P$ (i.e., (potentially randomized) functions $\hat P : \Omega^n \to \P$ of the data). In the sequel, we suppress the dependence of $\hat P = \hat P(X_1,...,X_n)$ in the notation.

\subsection{Definitions for Stating our Results}
Here, we give notation and definitions needed to state our main results in Sections~\ref{sec:upper_bounds} and \ref{sec:lower_bounds}.

Let $2^\Omega$ denote the power set of $\Omega$. Let $\SS \subseteq 2^{2^{\Omega}}$ denote the family of all Borel partitions of $\Omega$:
\[\SS := \left\{ \S \subseteq \Sigma \quad : \quad \Omega \subseteq \bigcup_{S \in \S} S \quad \text{ and } \quad \forall S,T \in \S, S \cap T = \emptyset \right\}.\]

We now define some metric notions that will later be useful for bounding Wasserstein distances:

\begin{definition}[Diameter and Separation of a Set, Resolution of a Partition]
For any set $S \subseteq \Omega$, the \emph{diameter $\Diam(S)$ of $S$} is defined by
$\Diam(S) := \sup_{x,y \in S} \rho(x,y)$,
and the \emph{separation $\Sep(S)$ of $S$} is defined by
$\Sep(S) := \inf_{x \neq y \in S} \rho(x,y)$.
If $\S \in \SS$ is a partition of $\Omega$, then the \emph{resolution $\operatorname{Res}(\S)$ of $\S$} defined by
$\operatorname{Res}(\S) := \sup_{S \in \S} \Diam(S)$ is the largest diameter of any set in $\S$.
\end{definition}

We now define the covering and packing number of a metric space, which are classic and widely used measures of the size or complexity of a metric space \citep{dudley1967coveringNumbers,haussler1995sphere,zhou2002covering,zhang2002covering}. Our main convergence results will be stated in terms of these quantities, as well as the packing radius, which acts, approximately, as the inverse of the packing number.

\begin{definition}[Covering Number, Packing Number, and Packing Radius of a Metric Space]
The \emph{covering number $N : (0,\infty) \to \N$ of $(\Omega,\rho)$} is defined for all $\epsilon > 0$ by
\[N(\epsilon) := \min \left\{ |\S| : \S \in \SS \quad \text{ and } \quad \operatorname{Res}(\S) \leq \epsilon \right\}.\]
The \emph{packing number $M : (0,\infty) \to \N$ of $(\Omega,\rho)$} is defined for all $\epsilon > 0$ by
\[M(\epsilon) := \max \left\{ |S| : S \subseteq \Omega \quad \text{ and } \quad \Sep(S) \geq \epsilon \right\}.\]
Finally, the \emph{packing radius $R : \N \to [0,\infty]$} is defined for all $n \in \N$ by
\[R(n) := \sup\{ \Sep(S) : S \subseteq \Omega \quad \text{ and } \quad |S| \geq n\}.\]
Sometimes, we use the covering or packing number of a metric space, say $(\Theta, \tau)$, other than $(\Omega,\rho)$; in such cases, we write $N(\Theta;\tau;\epsilon)$ or $M(\Theta;\tau;\epsilon)$ rather than $N(\epsilon)$ or $M(\epsilon)$, respectively.
For specific $\epsilon > 0$, we will also refer to $N(\Theta;\tau;\epsilon)$ as the \emph{$\epsilon$-covering number of $(\Theta,\tau)$}.
\end{definition}

\begin{remark}
The covering and packing numbers of a metric space are closely related. In particular, for any $\epsilon > 0$, we always have
\begin{equation}
M(\epsilon) \leq N(\epsilon) \leq M(\epsilon/2).
\label{ineq:covering_packing_relationship}
\end{equation}
The packing number and packing radius also have a close approximate inverse relationship. In particular, for any $\epsilon > 0$ and $n \in \N$, we always have
\begin{equation}
R(M(\epsilon)) \geq \epsilon
  \quad \text{ and } \quad
  M(R(n)) \geq n.
\label{ineq:packing_number_radius_relationship}
\end{equation}
However, it is possible that $R(M(\epsilon)) > \epsilon$ or $M(R(n)) > n$.
\end{remark}

Finally, when we consider unbounded metric spaces, we will require some sort of concentration conditions on the probability distributions of interest, to obtain useful results. Specifically, we an appropriately generalized version of the moment of the distribution:

\begin{remark}
We defined the covering number slightly differently from usual (using partitions rather than covers). However, the given definition is equivalent to the usual definition, since (a) any partition is itself a cover (i.e., a set $\C \subseteq 2^\Omega$ such that $\Omega \subseteq \bigcup_{C \in \C} C$), and (b), for any countable cover $\C := \{C_1,C_2,...\} \subseteq 2^\Omega$, there exists a partition $\S \in \SS$ with $|\S| \leq |\C|$ and each $S_i \subseteq C_i$, defined recursively by
$S_i := C_i \sminus \bigcup_{j = 1}^{i - 1} S_i$.
$\S$ is often called the \emph{disjointification} of $\C$.
\label{remark:disjointification}
\end{remark}

\begin{definition}[Metric Moments of a Probability Distribution]
For any $\ell \in [0,\infty]$, probability measure $P \in \P$, and $x \in \Omega$, the \emph{$\ell^{th}$ metric moment $m_{\ell,x}(P)$ of $P$ around $x$} is defined by
\[m_{\ell,x}(P)
  := \left( \E_{Y \sim P} \left[ \left( \rho(x,Y) \right)^\ell \right] \right)^{1/\ell} \in [0,\infty],\]
using the appropriate limit if $\ell = \infty$.
The chosen reference point $x$ only affects constant factors since,
\[\text{ for all } x,x' \in \Omega,
  \quad \left| m_{\ell,x}^\ell(P) - m_{\ell,x'}^\ell(P) \right| \leq \left( \rho(x,x') \right)^\ell.\]
Note that, if $\Omega$ has linear structure with respect to which $\rho$ is translation-invariant (e.g., if $(\Omega,\rho)$ is a Fr\'echet space), we can state our results more simply in terms of $m_\ell(P) := \inf_{x \in \Omega} m_{\ell,x}(P)$.
As an example, if $\Omega = \R$ and $\rho(x,y) = |x - y|$, then $m_2(P)$ is precisely the standard deviation of $P$.
\end{definition}

\section{Related Work}
\label{sec:related_work}

A long line of work~\citep{dudley1969speed,ajtai1984optimalMatchings,canas2012learning,dereich2013constructive,boissard2014mean,fournier2015rate,weed2017sharp,lei2018convergence} has studied the rate of convergence of the empirical distribution to the population distribution in Wasserstein distance. In terms of upper bounds, the most general and tight upper bounds are the recent works of \citep{weed2017sharp}
and \citep{lei2018convergence}.
As we describe below, while these two papers overlap significantly, neither supersedes the other, and our upper bound combines the key strengths of those in \citep{weed2017sharp} and \citep{lei2018convergence}.

The results of \citep{weed2017sharp} are expressed in terms of a particular notion of dimension, which they call the \textit{Wasserstein dimension} $s$, since they derive convergence rates of order $n^{-r/s}$ (matching the $n^{-r/D}$ rate achieved on the unit cube $[0,1]^D$). The definition of $s$ is complex (e.g., it depends on the sample size $n$), but \citep{weed2017sharp} show that, in many cases, $s$ converges to certain common definitions of the intrinsic dimension of the support of the distribution.
This paper overcomes three main limitations of \citep{weed2017sharp}:
\begin{enumerate}[nolistsep,leftmargin=2em]
\item
The upper bounds of \citep{weed2017sharp} apply only to totally bounded metric spaces. In contrast, our upper bounds permit unbounded metric spaces under the assumption that the distribution $P$ has some finite moment $m_\ell(P) < \infty$. The results of \citep{weed2017sharp} correspond to the special case $\ell = \infty$.
\item
Their main upper bound (their Proposition 10) only holds when $s > 2r$, with constant factors diverging to infinity as $s \downarrow 2r$. Hence, their rates are loose when $r$ is large or when the data have low intrinsic dimension.
In contrast, our upper bound is tight even when $s \leq 2r$.
\item
As we discuss in our Example~\ref{ex:Lipschitz_Ball}, the upper bound of \citep{weed2017sharp} becomes loose as the Wasserstein dimension $s$ approaches $\infty$, limiting its utility in infinite-dimensional function spaces. In contrast, we show that our upper and lower bounds match for several standard function spaces.
\end{enumerate}
Intuitively, we find that the finite-sample bounds of \citep{weed2017sharp} are tight when the intrinsic dimension of the data lies in an interval $[a, b]$ with $2r < a < b < \infty$, but they can be loose outside this range. In contrast, we find our results give tight rates for a larger class of problems.

On the other hand, \citep{lei2018convergence} focuses on the case where $\Omega$ is a (potentially unbounded and infinite-dimensional) Banach space, under moment assumptions on the distributions.
Thus, while the results of \citep{lei2018convergence} cover interesting cases such as infinite-dimensional Gaussian processes, they do not demonstrate that convergence rates improve when the intrinsic dimension of the support of $P$ is smaller than that of $\Omega$ (unless this support lies within a \textit{linear} subspace of $\Omega$). As a simple example, if the distribution is in fact supported on a finite set of $k$ linearly independent points, the bound of \citep{lei2018convergence} implies only a convergence rate, whereas we give a bound of order $O(\sqrt{k/n})$. Although we do not delve into this here, our results (unlike those of \citep{lei2018convergence}) should also benefit from the multi-scale behavior discussed in Section 5 of \citep{weed2017sharp}; namely, much faster convergence rates are often observed for small $n$ than for large $n$. This may help explain why algorithms such as functional $k$-means~\citep{garcia2015functionalKMeans} work in practice, even though the results of \citep{lei2018convergence} imply only a slow convergence rate of $O\left( (\log n)^{-p} \right)$, for some constant $p > 0$, in this case.


Under similarly general conditions, \citep{sriperumbudur2010integralProbabilityMetrics,sriperumbudur2012empirical} have studied the related problem of estimating the Wasserstein distance between two unknown distributions given samples from those two distributions.
Since one can estimate Wasserstein distances by plugging in empirical distributions,
our upper bounds imply upper bounds for Wasserstein distance estimation. These bounds are tighter, in several cases, than those of \citep{sriperumbudur2010integralProbabilityMetrics,sriperumbudur2012empirical}; for example, when $\X = [0,1]^D$ is the Euclidean unit cube, we give a rate of $n^{-1/D}$, whereas they give a rate of $n^{-\frac{1}{D + 1}}$. Minimax rates for this problem are currently unknown, and it is presently unclear to us under what conditions recent results on estimation of $\L_1$ distances between discrete distributions~\citep{jiao2017minimaxL1} might imply an improved rate as fast as $\left( n \log n \right)^{-1/D}$ for estimation of Wasserstein distance.

To the best of our knowledge, minimax lower bounds for distribution estimation under Wasserstein loss remain unstudied, except in the very specific case when $\Omega = [0,1]^D$ is the Euclidean unit cube and $r = 1$~\citep{liang2017well}. As noted above, most previous works have focused on studying convergence rate of the empirical distribution to the true distribution in Wasserstein distance. For this rate, several lower bounds have been established, matching known upper bounds in many cases. However, many distribution estimators besides the empirical distribution can be considered. For example, it is tempting (especially given the infinite dimensionality of the distribution to be estimated) to try to reduce variance by techniques such as smoothing or importance sampling~\citep{bucklew2013introduction}. Our lower bound results, given in Section~\ref{sec:lower_bounds}, imply that the empirical distribution is already minimax optimal, up to constant factors, in many cases.

\section{Upper Bounds}
\label{sec:upper_bounds}

In this section, we present our main upper bounds on the convergence rate of the empirical distribution to the true distribution in Wasserstein distance. We begin by presenting a simpler result for the case of totally bounded metric spaces, followed by a more complex but general result for arbitrary metric spaces under finite-moment assumptions on the distribution.


\begin{theorem}
Let $(\Omega,\rho)$ be a metric space on which $P$ is a Borel probability measure. Let $\hat P$ denote the empirical distribution of $n$ IID samples $X_1,...,X_n \stackrel{IID}{\sim} P$, give by
\begin{equation}
\hat P(S) := \frac{1}{n} \sum_{i = 1}^n 1_{\{X_i \in S\}}, \quad \forall S \in \Sigma.
\label{eq:empirical_distribution}
\end{equation}
Then, for any non-increasing sequence $\{\epsilon_k\}_{k \in [K]} \in (0,\infty)^K$ with $\epsilon_0 = \Diam(\Omega)$,
\[\E \left[ W_r^r(P, \hat P) \right] \leq \epsilon_K^r + \frac{1}{\sqrt{n}} \sum_{k = 1}^K \left( \sum_{j = k}^K 2^{K - j} \epsilon_j \right)^r \sqrt{N(\epsilon_k) - 1}.\]
\label{thm:expectation_bound}
\end{theorem}

In the proof of the above theorem, the sequence $\{\epsilon_k\}_{k \in [K]} \in (0,\infty)^K$ gives the resolutions of a sequence of increasingly fine partitions of $\Omega$. The basic idea of the proof is to recursively bound the error over each partition at resolution $\epsilon_j$ in terms of $\epsilon_j$ and the error over the partition of resolution $\epsilon_{j + 1}$. The parameter $K$ restricts us to a particular finite resolution, with optimal value typically increasing with $n$. Note that this ``multi-resolution'' proof approach has been utilized in several special cases, apparently originating in the analysis of
Our Theorem~\ref{thm:expectation_bound} is most comparable to the upper bound (Proposition 10) of \citet{weed2017sharp}.

Theorem~\ref{thm:expectation_bound} requires $(\Omega,\rho)$ to be totally bounded in order for $N(\epsilon)$ to be finite. Next, we present a more complex bound, which, under the additional assumption that $P$ has some number of finite moments, is often finite even when $(\Omega,\rho)$ is \textit{not} totally bounded. The key idea of the proof is to partition $\Omega = \bigcup_{k = 0}^\infty B_k$ into bounded subsets, over each of which we can apply a bound similar to Theorem~\ref{thm:expectation_bound}. Thus, instead of the covering number $N(\epsilon)$ of $(\Omega,\rho)$, this result uses covering numbers $N(B_k,\rho,\epsilon)$ of a partition $\Omega = \bigcup_{k = 0}^\infty B_k$ into totally bounded subsets.

\begin{theorem}[General Upper Bound for Unbounded Metric Spaces]
Let $x_0 \in \Omega$ and suppose $m_{\ell,x_0}(P) \in [1, \infty)$. Let $J \in \N$. Fix two non-decreasing real-valued sequences $\{w_k\}_{k \in \N}$ and $\{\epsilon_j\}_{j \in \N}$, of which $\{w_k\}_{k \in \N}$ is non-decreasing with $w_0 = 0$ and $\lim_{k \to \infty} w_k = \infty$ and $\{\epsilon_j\}_{j \in [J]}$ is non-increasing.
For each $k \in \N$, define $B_k(x_0) := \left\{ y \in \Omega : w_k \leq \rho(x_0, x) < w_{k + 1} \right\}$.
Then,
\begin{align*}
\E \left[ W_r^r(P, \hat P) \right]
& \leq m_{\ell,x_0}^\ell(P) \sum_{k \in \N} w_k^{-\ell} \left( \epsilon_J \right)^r + 2^r w_k^{r - \ell/2} \min \left\{ 2w_k^{-\ell/2}, \sqrt{\frac{1}{n}} \right\} \\
& \hspace{2cm} + \sum_{j = 1}^J \left( \sum_{t = j}^J 2^{J - t} \epsilon_t \right)^r \min 
\left\{ 2w_k^{-\ell}, \sqrt{\frac{w_k^{-\ell}}{n} N(B_k,\rho,\epsilon_j)} \right\}.
\end{align*}
\label{thm:unbounded_upper_bound}
\end{theorem}
In the above, $w_k$ corresponds to radii of the partition of $\Omega = \bigcup_{k = 0}^\infty B_k$ into a sequence of ``spherical shells'', whereas $\epsilon_j$, as in the previous result, corresponds to resolutions of partitions of the $B_k$'s. As with $K$ in the previous result, $J$ is used to ensure that we restrict ourselves to a particular finite resolution. The $\min$ terms appear because, for large $k$, the error is controlled by the fact that $P(B_k)$ is small (due to the moment assumption), rather than using a covering of $B_k$.

\section{Lower Bounds}
\label{sec:lower_bounds}

In this section, we provide a minimax lower bound (over the family $\P$ of all Borel distributions on $\Omega$) for density estimation in Wasserstein distance (that is, the quantity
\begin{equation}
\inf_{\hat P : \X^n \to \P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r \left( P, \hat P \right) \right],
\label{exp:distribution_estimation_minimax}
\end{equation}
where the infimum is over all estimators $\hat P$ of $P$ (i.e., all (potentially randomized) functions $\hat P : \Omega^n \to \P$)). Our bound depends primarily on the packing radius $R$ of $(\Omega,\rho)$, and, presently, we handle only the case without finite-moment assumptions on $P$. However, we show in the next section that this often implies tight lower bounds when enough (roughly, $\ell \geq \max\{D,2r\}$) moments exist.

\begin{theorem}
Let $(\Omega,\rho)$ be a metric space, on which $\P$ is the set of Borel probability measures. Then,
\[\inf_{\hat P : \X^n \to \P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r(P, \hat P(X_1,...,X_n)) \right]
  \geq c_r \sup_{k \in [32n]} R^r(k) \sqrt{\frac{k - 1}{n}},\]
where $c_r = \frac{3\log 2}{2^{r + 12}}$ depends only on $r$.
\label{thm:Wasserstein_distribution_estimation_lower_bound}
\end{theorem}

\section{Example Applications}
\label{sec:examples}

Our theorems in the previous sections are quite abstract and have many tuning parameters. Thus, we conclude by exploring applications of our results to cases of interest. In each of the following examples, $P$ is an unknown Borel probability measure over the specified $\Omega$, from which we observe $n$ IID samples. For upper bounds, $\hat P$ denotes the empirical distribution~\eqref{eq:empirical_distribution} of these samples.

\begin{example}[Finite Space]
Consider the case where $\Omega$ is a finite set, over which $\rho$ is the discrete metric given, for some $\delta > 0$, by
$\rho(x, y) = \delta 1_{\{x = y\}}$, for all $x,y \in \Omega$.
Then, for any $\epsilon \in (0,\delta)$, the covering number is $N(\epsilon) = |\Omega|$. Thus, setting $K = 1$ and sending $\epsilon_1 \to 0$ in Theorem~\ref{thm:expectation_bound} gives
\[\E \left[ W_r^r(P, \hat P) \right] \leq \delta^r \sqrt{\frac{|\Omega| - 1}{n}}.\]
On the other hand, $R(|\Omega|) = \delta$, and so, setting $k = |\Omega|$ in Theorem~\ref{thm:Wasserstein_distribution_estimation_lower_bound} yields
\[\inf_{\hat P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r(P, \hat P) \right]
  \gtrsim \delta^r \sqrt{\frac{|\Omega| - 1}{n}}.\]
\label{ex:discrete_bound}
\end{example}

\begin{example}[Unit Cube, Euclidean Metric]
Consider the case where $\Omega = \R^D$ is the unit cube and $\rho$ is the Euclidean metric. Assuming $\ell > r$, using the fact that $N \left( B_k, \rho, \epsilon \right) \leq \left( \frac{3 w_k}{\epsilon} \right)^D$~\citep{pollard1990empirical} and plugging $\epsilon_j = 2^{-2j}$ and $w_k = 2^k$ into Theorem~\ref{thm:unbounded_upper_bound} gives (after a straightforward but very tedious calculation) a constant $C_{D,r,\ell}$ depending only on $D$, $r$, and $\ell$ such that
\begin{equation}
\E \left[ W_r^r(P, \hat P) \right]
  \leq C_{D,\ell,r} m_\ell^\ell(P) \left( n^{\frac{\ell - r}{\ell}} + 2^{-2Jr} + \sum_{j = 1}^J 2^{(D - 2r)j} \right).
  \label{ineq:general_Euclidean_bound}
\end{equation}
Of these three terms, the first depends only on the number $\ell$ of finite moments $P$ is assumed to have and the order $r$ of the Wasserstein distance, whereas the second and third terms depend on choosing the parameter $J$. The optimal choice of $J$ scales with the sample size $n$ at a rate depending on the quantity $D - 2r$. Specifically, if $D = 2r$, then setting $J \asymp \frac{1}{4r} \log_2 n$ gives a rate of
$\E \left[ W_r^r(P, \hat P) \right]
  \lesssim n^{\frac{\ell - r}{\ell}} + n^{-1/2} \log n$.
If $D \neq 2r$, then~\eqref{ineq:general_Euclidean_bound} reduces to
\[\E \left[ W_r^r(P, \hat P) \right]
  \leq C_{D,\ell,r} m_\ell^\ell(P) \left( n^{\frac{\ell - r}{\ell}} + 2^{-2Jr} + \frac{2^{(D - 2r)J} - 1}{2^{D - 2r} - 1} \right).\]
Then, if $D > 2r$, sending $J \to \infty$ gives $\E \left[ W_r^r(P, \hat P) \right] \lesssim n^{\frac{\ell - r}{\ell}} + n^{-1/2}$. Finally, if $D < 2r$, then setting $J \asymp \frac{1}{2D} \log n$ gives $\E \left[ W_r^r(P, \hat P) \right] \lesssim n^{\frac{\ell - r}{\ell}} + n^{-\frac{r}{D}}$.
To summarize
\[\E \left[ W_r^r(P, \hat P) \right]
  \lesssim n^{\frac{\ell - r}{\ell}} + \left\{
    \begin{array}{ll}
      n^{-1/2} & \text { if } 2r > D \\
      n^{-1/2} \log n & \text { if } 2r = D \\
      n^{-r/D} & \text { if } 2r < D
    \end{array}
  \right.\]
(reproducing Theorem 1 of \citep{fournier2015rate}). On the other hand, it is easy to check that the packing radius $R$ satisfies $R(n) \geq n^{-1/D}$ and $R(2) \geq \sqrt{D}$. Thus, Theorem~\ref{thm:Wasserstein_distribution_estimation_lower_bound} with $k = n$ and $k = 2$ yields
\[\inf_{\hat P} \sup_{P \in \P} \E \left[ W_r^r(\hat P, P) \right]
  \gtrsim \max \left\{ (n + 1)^{-r/D}, D^{r/2} n^{-1/2} \right\}.\]
Together, these bounds give the following minimax rates for density estimation in Wasserstein loss:
\[\inf_{\hat P} \sup_{P \in \P} \E \left[ W_r^r(\hat P, P) \right]
  \asymp \left\{
    \begin{array}{ll}
      n^{-1/2} & \text { if } \ell > 2r > D \\
      n^{-r/D} & \text { if } 2r < D, \ell > \frac{Dr}{D - r}
    \end{array}
  \right.\]
When $2r = D$ and $\ell > 2r$, our upper and lower bounds are separated by a factor of $\log n$. The main result of \citep{ajtai1984optimalMatchings} implies that, for the case $D = 2$ and $r = 1$, the empirical distribution converges as $n^{-1/2} \log n$, suggesting that the $\log n$ factor in our upper bound may be tight. Further generalization of Theorem~\ref{thm:Wasserstein_distribution_estimation_lower_bound} is needed to give lower bounds when both $D, \ell \leq 2r$ or when $D > 2r$ and $\ell \leq \frac{Dr}{D - r}$.
\label{ex:unit_cube_lower_bound}
\end{example}

The next example demonstrates how the rate of convergence in Wasserstein metric depends on properties of the metric space $(\Omega,\rho)$ at both large and small scales. Specifically, if we discretize $\Omega$, then the phase transition at $2r = D$ disappears.
\begin{example}
Suppose $\Omega = \mathbb{Z}^D$ is a $D$-dimensional grid of integers and $\rho$ is $\ell_\infty$-metric (given by $\rho(x,y) = \max_{j \in [D]} |x_j - y_j|$). Since $\Z^D \subseteq \R^D$ and the $\ell_\infty$ and Euclidean metrics are topologically equivalent, the upper bounds from Example~\ref{ex:unit_cube_lower_bound} clearly apply, up to a factor of $\sqrt{D}$. However, we also have the fact that, whenever $\epsilon < 1$, $N(B_k,\rho,\epsilon) = w_k^D$. Therefore, setting $J = 0$, $\epsilon_0 = 0$, and $w_k = 2^k$ in Theorem~\ref{thm:unbounded_upper_bound} gives, for a constant $C_{D,\ell,r}$ depending only on $D$, $\ell$, and $r$,
\begin{align*}
\E \left[ W_r^r(P, \hat P) \right]
& \leq C_{D,\ell,r} m_\ell^\ell(P) \left( n^{\frac{\ell - r}{\ell}} + \sum_{k \in \N} \sqrt{\frac{2^{(D - \ell)k}}{n}} \right).
\end{align*}
When $\ell > D$, this reduces to $\E \left[ W_r^r(P, \hat P) \right] \lesssim n^{\frac{\ell - r}{\ell}} + n^{-1/2}$, giving a tighter rate than in Example~\ref{ex:unit_cube_lower_bound} when $2r \leq D < \ell$.
To the best of our knowledge, no prior results in the literature imply this fact.
\end{example}





Finally, we consider distributions over an infinite dimensional space of smooth functions.
\begin{example}[H\"older Ball, $\L_\infty$ Metric]
Suppose that, for some $\alpha \in (0,1]$,
\[\Omega
  := \left\{ f [0,1]^D \to [-1,1] \quad \middle| \quad \forall x,y \in [0,1]^D, \quad |f(x) - f(y)| \leq \|x - y\|_2^\alpha \right\}\]
is the class of unit $\alpha$-H\"older functions on the unit cube and $\rho$ is the $\L_\infty$-metric given by
\[\rho(f,g) = \sup_{x \in [0,1]^D} |f(x) - g(x)|, \quad \text{ for all } f,g \in \Omega.\]
The covering and packing numbers of $(\Omega,\rho)$ are well-known to be of order $\exp \left( \epsilon^{-D/\alpha} \right)$ \citep{devore1993approximation}; specifically, there exist positive constants $0 < c_1 < c_2$ such that, for all $\epsilon \in (0,1)$,
\[c_1 \exp \left( \epsilon^{-D/\alpha} \right)
  \leq N(\epsilon)
  \leq M(\epsilon)
  \leq c_2 \exp \left( \epsilon^{-D/\alpha} \right).\]
Since $\Diam(\Omega) = 2$, applying Theorem~\ref{thm:expectation_bound} with $K = 1$ and
\[\epsilon_1
  = \left( 2\log n - (\alpha r/D) \log \log n \right)^{-\frac{\alpha r}{D}}
  \quad \text{ gives } \quad
  \E \left[ W_r^r(P, \hat P) \right]
  \lesssim \left( \log n \right)^{\frac{- \alpha r}{D}}.\]
Conversely, Inequality~\eqref{ineq:packing_number_radius_relationship} implies $R(n) \geq \left( \log(n/c_1) \right)^\frac{-\alpha}{D}$, and so setting $k = n$ in Theorem~\ref{thm:Wasserstein_distribution_estimation_lower_bound} gives
\[\inf_{\hat P} \sup_{P \in \P} \E_{X_1,...,X_n \stackrel{IID}{\sim} P} \left[ W_r^r(P, \hat P) \right]
  \gtrsim \left( \frac{1}{\log(n/c_1)} \right)^\frac{\alpha r}{D},\]
showing that distribution estimation over $(\P,W_r^r)$ has the extremely slow minimax rate $\left( \log n \right)^\frac{-\alpha r}{D}$. Although we considered only $\alpha \in (0,1]$ (due to the notational complexity of defining higher-order H\"older spaces), analogous rates hold for all $\alpha > 0$. Also, since our rates depend only on covering and packing numbers of $\Omega$, identical rates can be derived for related Sobolev and Besov classes.
Note that the Wasserstein dimension used in the prior work \citep{weed2017sharp} is of order $\frac{D}{\alpha} \log n$, and so their upper bound (their Proposition 10) gives a rate of $n^{-\frac{\alpha r}{D \log n}} = \exp \left( -\frac{\alpha r}{D} \right)$, which fails to converge as $n \to \infty$.
\label{ex:Lipschitz_Ball}
\end{example}

One might wonder why we are interested in studying Wasserstein convergence of distributions over spaces of smooth functions, as in Example~\ref{ex:Lipschitz_Ball}.
Our motivation comes from the historical use of smooth function spaces have been widely used for modeling images and other complex naturalistic signals \citep{mallat1999wavelet,peyre2011numerical,sadhanala2016totalVariation}.
Empirical breakthroughs have recently been made in generative modeling, particularly of images, based on the principle of minimizing Wasserstein distance between the empirical distribution and a large class of models encoded by a deep neural network~\citep{montavon2016wassersteinRBMs,arjovsky2017wassersteinGAN,gulrajani2017improved}.

However, little is known about theoretical properties of these methods; while there has been some work studying the optimization landscape of such models~\citep{nagarajan2017gradient,liang2018interaction}, we know of far less work exploring their \textit{statistical} properties.
Given the extremely slow minimax convergence rate we derived above, it must be the case that the class of distributions encoded by such models is far smaller or sparser than $\P$. An important avenue for further work is thus to explicitly identify stronger assumptions that can be made on distributions over interesting classes of signals, such as images, to bridge the gap between empirical performance and our theoretical understanding.

\section{Conclusion}
\label{sec:conclusion}

In this paper, we derived upper and lower bounds for distribution estimation under Wasserstein loss. Our upper bounds generalize prior results and are tighter in certain cases, while our lower bounds are, to the best of our knowledge, the first minimax lower bounds for this problem. We also provided several concrete examples in which our bounds imply novel convergence rates.

\subsection{Future Work}

We studied minimax rates over the very large entire class $\P$ of all distributions with some number of finite moments.
It would be useful to understand how minimax rates improve when additional assumptions, such as smoothness, are made (see, e.g., \citep{liang2017well} for somewhat improved upper bounds under smoothness assumptions when $(\Omega,\rho)$ is the Euclidean unit cube).
Given the slow convergence rates we found over $\P$ in many cases, studying minimax rates under stronger assumptions may help to explain the relatively favorable empirical performance of popular distribution estimators based on empirical risk minimization in Wasserstein loss.
Moreover, while rates over all of $\P$ are of interest only for very weak metrics such as the Wasserstein distance (as stronger metrics may be infinite or undefined), studying minimax rates under additional assumptions will allow for a better understanding of the Wasserstein metric in relation to other commonly used metrics.

\newpage
\subsubsection*{Acknowledgments}
This work was partly supported by a NSF Graduate Research Fellowship DGE-1252522 to S.S.

{\small
  \bibliographystyle{plainnat}
  
2018-05-24T02:05:10.000
1802
1802.08855
en
https://arxiv.org/abs/1802.08855
The Wasserstein metric is an important measure of distance between probability distributions, with applications in machine learning, statistics, probability theory, and data analysis. This paper provides upper and lower bounds on statistical minimax rates for the problem of estimating a probability distribution under Wasserstein loss, using only metric properties, such as covering and packing numbers, of the sample space, and weak moment assumptions on the probability distributions.
Statistics Theory (math.ST); Information Theory (cs.IT); Machine Learning (cs.LG); Machine Learning (stat.ML)
Minimax Distribution Estimation in Wasserstein Distance
Qwen/Qwen-72B
1. YES
2. YES