Optimal Testing for Properties of Distributions
Given samples from an unknown distribution $p$, is it possible to distinguish whether $p$ belongs to some class of distributions $\mathcal{C}$ versus $p$ being far from every distribution in $\mathcal{C}$? This fundamental question has received tremendous attention in statistics, focusing primarily on asymptotic analysis, and more recently in information theory and theoretical computer science, where the emphasis has been on small sample size and computational complexity. Nevertheless, even for basic properties of distributions such as monotonicity, log-concavity, unimodality, independence, and monotone-hazard rate, the optimal sample complexity is unknown.We provide a general approach via which we obtain sample-optimal and computationally efficient testers for all these distribution families. At the core of our approach is an algorithm which solves the following problem: Given samples from an unknown distribution $p$, and a known distribution $q$, are $p$ and $q$ close in $\chi^2$-distance, or far in total variation distance?The optimality of our testers is established by providing matching lower bounds with respect to both $n$ and $\varepsilon$. Finally, a necessary building block for our testers and an important byproduct of our work are the first known computationally efficient proper learners for discrete log-concave and monotone hazard rate distributions.

\section{Experimental Evaluation}

In this section, we evaluate the efficacy of our proposed monotonicity tester for $d=1$, and compare it to the estimator proposed in~\cite{BatuKR04}, which we will refer to as the BKR tester henceforth. 
We note that even for $d=1$, our algorithm is seen to outperform the BKR tester in almost all cases. 

Both our algorithm and BKR require a threshold for the test statistic. 
It follows from our results that our estimator's threshold should be at least the standard deviation of the statistic when the distribution is monotone -- this is around $\sqrt{2n}$.
Also, our threshold should be smaller than the expectation of the statistic when the distribution is not in the class.
In particular, we use the value $2m\varepsilon^2+ \sqrt{2n}$ as the threshold for our test statistic. 

The statistic of BKR imitates an $\ell_2$ test by introducing a discrete statistic that counts the number of collisions within nearby elements.
The argument is that monotone distributions will be locally uniform, and therefore the number of such local collisions should be small. 
They also compare their algorithm with a threshold that is proportional to $O(\log^2 n/\varepsilon)$. However, the constant is not specified in their paper. We considered a few Zipf distributions, which are monotone, and perturbed the probability elements to generate distributions that are $\varepsilon$-far from monotone distributions in total variation distance. We then optimized their threshold for the best possible performance for a range of $n$ and $\varepsilon$. In particular, we came up with the constant of 0.6 for their statistic. Note that one might use uniform distributions, or yet another class of distributions to choose the threshold. 

As is expected, when we run experiments on the class that was used to determine the threshold, the performance of BKR is comparable with our approach for moderate values of $\varepsilon$.
For very small $\varepsilon$, once again their performance degrades, since the number of samples required for concentration of their statistic is proportional to $\varepsilon^{-6}$.
However, for experiments on other monotone classes of distributions (i.e., classes which were not used to train the threshold), our algorithm performed significantly better in almost all cases.
In particular, we also consider a test between the uniform distribution and perturbations thereof (which are guaranteed to be $\varepsilon$-far from monotone).

We give two illustrative plots in the appendix. The number of samples are plotted in logarithmic scale. Both these experiments are run for $n=50,000$. In the uniform case, we take the total variation bound to be $0.05$ and for the Zipf distribution we take it to be $0.07$. As we see for the uniform distribution, our algorithm has accuracy 0.9 with 30000 samples, compared to their accuracy of 0.55. For the Zipf plot, their algorithm is comparable, since we chose the threshold to work the best for this class. 





\section{Learning product distributions in $\chi^2$ distance}
\label{sec:independence-appendix}
In this section we prove Lemma~\ref{lem:learning-product}. The proof
is analogous to the proof for learning monotone distributions, and
hinges on the following result of~\cite{KamathOPS15}. 
Given $m$ samples from a distribution $q$ over $n$ elements, the add-1
estimator (Laplace estimator) $q$ satisfies:
\[
\expectation{\chi^2(p,q) } \le \frac{n}{m+1}.
\]

Now, suppose $p$ is a product distribution over
${\cal X}=[n_1]\times\cdots\times[n_d]$. We simply perform the add-1
estimation over each coordinate independently, giving a distribution
$q^1\times\cdots \times q^d$. Since $p$ is a product distribution the
estimates in each coordinate is independent. Therefore, a simple
application of the previous result and independence of the coordinates implies
\begin{align}
\expectation{\chi^2(p,q) } & = 
\prod_{l=1}^{d}\left(1+ \expectation{\chi^2(p^l,q^l) }\right)-1\nonumber\\
& \le \prod_{l=1}^{d}\left(1+ \frac{n_l}{m+1}\right)-1\nonumber\\
&\le \exp\left(\frac{\sum_l {n_l}}{m+1}\right)-1,\label{eqn:haha}
\end{align}
where~\eqref{eqn:haha} follows from $e^x\ge1+x$. 
Using $e^x\le 1+2x$ for $0\le x\le 1$, we have  
\begin{align}
\expectation{\chi^2(p,q) } & \leq 2\frac{\sum_l {n_l}}{m+1},
\end{align}
when $m\geq\sum_l n_l$. Therefore, following an application of Markov's inequality, when $m = \Omega((\sum_l
n_l)/\varepsilon^2)$, Lemma~\ref{lem:learning-product} is proved.

\section{Testing Independence of Random Variables}
\label{sec:independence}
Let 
${\cal X}{\hfill\blksquare}\medskip[n_1]\times\ldots\times[n_d]$, and let $\Pi_d$ be the class
of all product distributions over ${\cal X}$. 
We first bound the $\chi^2$-distance between product distributions in
terms of the individual coordinates. 
\begin{lemma}
\label{lem:prod-chi}
Let $p = p^1\times p^2\ldots\times p^d$, and 
$q = q^1\times q^2\ldots\times q^d$ be two distributions in $\Pi_d$. 
Then 
\[
\chisqpq{p}{q} = \prod_{\gnote{\ell}=1}^d(1+\chisqpq{p\gnote{^\ell}}{q\gnote{^\ell}})-1.
\]
\end{lemma}
\begin{proof}
By the definition of $\chi^2$-distance
\begin{align}
\chisqpq{p}{q} &= \sum_{{\bf i}\in{\cal X}} \frac{{\dP}_{{\bf i}}^2}{{\dQ}_{{\bf i}}}-1\\
& =
\prod_{\gnote{\ell}=1}^d\left[\sum_{i \in[n_\gnote{\ell}]}\frac{\left(p^{\gnote{\ell}}_{i}\right)^2}{q^{\gnote{\ell}}_{i}}\right]-1\\
& =
\prod_{\gnote{\ell = 1}}^\gnote{d}\left(1+\chisqpq{p^\gnote{\ell}}{q^\gnote{\ell}}\right)-1.
\end{align}
\end{proof}

Along the lines of learning monotone distributions in $\chi^2$
distance we obtain the following result, proved in Section~\ref{sec:independence-appendix}.  
\begin{lemma}
\label{lem:learning-product}
There is an algorithm that takes 
\[
O\left(\sum_{\ell=1}^d \frac{n_\ell}{\varepsilon^2}\right)
\]
samples from a distribution $p$ in $\Pi_d$ and outputs a distribution
$q\in\Pi_d$ such that with probability at least $5/6$, 
\[
\chisqpq{p}{q}\le O(\varepsilon^2).
\] 
\end{lemma}
This fits precisely in our framework of robust $\chi^2$-$\ell_1$
testing. In particular, applying Theorem~\ref{thm:chisq-test}, we obtain the following result. 
\begin{theorem}
\label{thm:independence}
For any $d\ge1$, there exists an algorithm for testing independence of
random variables over $[n_1]\times\ldots[n_d]$ with sample and time complexity 
\[
O\left(\frac{(\prod_{\ell = 1}^dn_\ell)^{1/2} + \sum_{\ell = 1}^d n_\ell}{\varepsilon^2}\right).
\] 
\end{theorem}

The following corollaries are immediate. 
\begin{corollary}
Suppose $\prod_{\ell = 1}^d n_\ell^{1/2} \geq \sum_{\ell =1}^d n_\ell$.
Then there exists an algorithm for testing independence over $[n_1] \times \dots \times [n_d]$ with sample complexity $\Theta((\prod_{\ell = 1}^d n_\ell)^{1/2}/\varepsilon^2)$. 
\end{corollary}

In particular, 
\begin{corollary}
There exists an algorithm for testing if two distributions over $[n]$ are independent with sample complexity $\Theta(n/\varepsilon^2)$. 
\end{corollary}


\section{Details of the Lower Bounds}
\label{sec:lb-appendix}
In this section, for the class of distributions $\mathcal{Q}$ described in discussion on lower bounds and a class of interest $\mathcal{C}$, we show that $d_{\mathrm {TV}}(\mathcal{C},\mathcal{Q}) \geq \varepsilon$, thus implying a lower bound of $\Omega(\sqrt{n}/\varepsilon^2)$ for testing $\mathcal{C}$. 
\subsection{Monotone distributions}
We first consider $d=1$ and prove that for appropriately chosen $c$, any monotone distribution over $[n]$ is $\varepsilon$-far from  all distributions in $\mathcal{Q}$.  
Consider any $q \in \mathcal{Q}$.
For this distribution, we say that $i\in[n]$ is a \emph{raise-point} if $\dQ_i<\dQ_{i+1}$. Let $R_{q}$ be the set of raise points of $q$. For $q\in\mathcal{Q}$,~\eqref{eqn:fcl} implies at least one in every four consecutive integers in $[n]$ is a raise point, and therefore, $|R_q|\gen/4$. Moreover, note that if $i$ is a raise-point, then $i+1$ is not a raise point. For any monotone (decreasing) distribution $p$, $\dP_i\ge\dP_{i+1}$. For any raise-point $i\in R_q$, by the triangle inequality,
\begin{align}
\label{eqn:triangle-mon}
|\dP_i-\dQ_i|+|\dP_{i+1}-\dQ_{i+1}|\ge |\dP_i-\dP_{i+1}+\dQ_{i+1}-\dQ_i|\ge \dQ_{i+1}-\dQ_i = \frac{2c\varepsilon}{n}. 
\end{align}
Summing over the set $R_q$, we obtain $d_{\mathrm {TV}}(p,q)\ge \frac12|R_q|\cdot \frac{2c\varepsilon}{n}\ge {c\varepsilon}/{4}$. Therefore, if $c\ge4$, then $d_{\mathrm {TV}}(\mathcal{M}_{n},q)\ge\varepsilon$. This proves the lower bound for $d=1$. 

This argument can be extended to $[n]^d$. Consider the following class of distributions on $[n]^d$. For each point ${\bf i}=(i_1, \ldots, i_d)\in[n]^d$, where $i_1$ is even, generate a random $z\in\{-1,1\}$, and assign 
to ${\bf i}$ a probability of $(1+z c\varepsilon)/n^{d}$. Let ${\bf e}_1{\hfill\blksquare}\medskip(1,0,\ldots,0)$.  Similar to $d=1$, assign a probability $(1-z c\varepsilon)/n^{d}$ to the point ${\bf i}+{\bf e}_1 = (i_1+1,i_2,  \ldots, i_d)$. This class consists of $2^{\frac{n^{d/2}}2}$ distributions, and Paninski's arguments extend to give a lower bound of $\Omega(n^{d/2}/\varepsilon^2)$ samples to distinguish this class from the uniform distribution over $[n]^d$. It remains to show that all these distributions are $\varepsilon$ far from $\mathcal{M}_n^d$. Call a point ${\bf i}$ as a raise point if $p_{\bf i}<p_{{\bf i}+{\bf e}_1}$. For any ${\bf i}$, one of the points ${\bf i}$, ${\bf i}+{\bf e}_1$, ${\bf i}+2{\bf e}_1$, ${\bf i}+3{\bf e}_1$ is a raise point, and the number of raise points is at least $n^{d}/4$. Invoking the triangle inequality (identical to~\eqref{eqn:triangle-mon}) over the raise-points, in the first dimension shows that any monotone distribution over $[n]^d$ is at a distance $\frac{c\varepsilon}4$ from any distribution in this class. Choosing $c=4$ yields a  bound of $\varepsilon$. 

\subsection{Testing Product Distributions}
Our idea for testing independence is similar to the previous section. We sketch the construction of a class of distributions on ${\cal X}=[n_1]\times\cdots\times[n_d]$. Then $|{\cal X}| = n_1\cdot n_2\ldots\cdot n_d$. For each element in ${\cal X}$ assign a value $(1\pm c\varepsilon)$ and then for each such assignment, normalize the values so that they add to 1, giving rise to a distribution. This gives us a class of $2^{|{\cal X}|}$ distributions. The key argument is to show that a \emph{large} fraction of these distributions are far from being a product distribution. This follows since the degrees of freedom of a product distribution is exponentially smaller than the number of possible distributions. The second step is to simply apply Paninski's argument, now over the larger set of distributions, where we show that distinguishing the collection of distributions we constructed from the uniform distribution over ${\cal X}$ (which is a product distribution) requires $\sqrt{|{\cal X}|}/\varepsilon^2$ samples. 


\subsection{Log-concave and Unimodal distributions}
We will show that any log-concave or unimodal distribution is $\varepsilon$-far from all distributions in $\mathcal{Q}$. 
Since $\mathcal{LCD}_n \subset \mathcal{U}_n$, it will suffice to show this for every unimodal distribution. 
Consider any unimodal distribution $p$, with mode $\ell$. Then, $p$ is monotone non-decreasing over the interval $[\ell]$ and non-increasing over $\{\ell+1, \ldots, n\}$.  By the argument for monotone distributions, the total variation distance between $p$ and any distribution $q$ over elements greater than $\ell$ is at least $\frac{n-\ell-1}{n}\frac{c\varepsilon}{4}$, and over elements less than $\ell$ is at least $\frac{\ell-1}{n}\frac{c\varepsilon}{4}$. Summing these two gives the desired bound.

\subsection{Monotone Hazard distributions}
We will show that any monotone hazard rate distribution is $\varepsilon$-far from all distributions in $\mathcal{Q}$. 

Let $p$ be any monotone-hazard distribution. Any distribution $q\in\mathcal{Q}$ has mass at least $1/2$ over the interval $I=[n/4,3n/4]$. Therefore, by Lemma~\ref{lem:mhr-str}, for any $i\in I$, $\dP_{i+1}\left(1+\frac{\dP_i}{1/4}\right)\ge \dP_i$. 
As noted before, at least $n/8$ of the raise-points are in  $I$. 

For any $i\in I\cap R_q$, $\dQ_i=(1+c\varepsilon)/n$, $\dQ_{i+1}=(1-c\varepsilon)/n$
\begin{align}
d_i = |\dP_i-\dQ_i|+|\dP_{i+1}-\dQ_{i+1}|.
\end{align}

If $\dP_i\ge(1+2c\varepsilon)/n$ or $\dP_i\le1/n$, then the first term, and therefore $d_i$ is at least $c\varepsilon/n$. 
If $\dP_i\in(1/n, (1+2c\varepsilon)/n)$, then for $n>5/(c\varepsilon)$
\[
\dP_{i+1}\ge \frac{1}{n}\cdot \frac{1}{1+\frac4{n}}\ge \frac{1-c\varepsilon/2}{n}. 
\]
Therefore the second term of $d_i$ is at $c\varepsilon/2n$. Since there are at least $n/8$ raise points in $I$,   
\begin{align}
d_{\mathrm {TV}}(p,q)\ge \frac12\frac{n}{8}\cdot \frac{c\varepsilon}{2n}\ge \frac{c\varepsilon}{16}. 
\end{align}
Thus any MHR distribution is $\varepsilon$-far from $\mathcal{Q}$ for $c\ge16$. 

\section{Details on Testing Log-Concavity}
\label{sec:LCD}
It will suffice to prove Lemma~\ref{lem:lcd-learn}. 

\begin{prevproof}{Lemma}{lem:lcd-learn}
We first draw samples from $p$ and obtain a $O(1/\varepsilon^{3/2})$-piecewise constant
distribution $f$ by appropriately flattening the empirical
distribution. The proof is now in two parts.
In the first part, we show that if $p \in \mathcal{LCD}_n$ then
$f$ will be close to $p$ in $\chi^2$ distance over its effective
support.  
The second part involves proper learning of $p$. We
will use a linear program on $f$ to find a distribution $q\in\mathcal{LCD}_n$.
This distribution is such that if  $p\in\mathcal{LCD}_n$, then $\chi^2(p,q)$ is
small, and otherwise the algorithm will either output some $q \in \mathcal{LCD}_n$ (with
no other relevant guarantees) or \textsc{Reject}\xspace. 

We first construct $f$. Let $\hat p$ be the empirical distribution
obtained by sampling $O(1/\varepsilon^5)$ samples from $p$.
By Lemma~\ref{lem:dkw}, with probability at least $5/6$, $d_{\mathrm K}(p,\hat p) \leq \varepsilon^{5/2}/10$.
In particular, note that $|p_i - \hat p_i| \leq \varepsilon^{5/2}/10$. 
Condition on this event in the remainder of the proof.

Let $a$ be the minimum $i$ such that $p_i \geq \varepsilon^{3/2}/5$, and let $b$ be
the maximum $i$ satisfying the same condition. 
Let $M = \{a, \dots, b\}$ or $\emptyset$ if $a$ and $b$ are undefined.
By the guarantee provided by the DKW inequality, $p_i \geq \varepsilon^{3/2}/10$ for all $i \in M$. 
Furthermore, $\hat p_i \in p_i \pm \varepsilon^{3/2}/10 \in (1 \pm \varepsilon) \cdot p_i$.
For each $i \in M$, let $f_i = \hat p_i$. 
We note that $|M| = O(1/\varepsilon)$, so this contributes $O(1/\varepsilon)$ constant pieces to $f$.

We now divide the rest of the domain into $t$ intervals, all but constantly many of measure $\Theta(\varepsilon^{3/2})$ (under $p$).
This is done via the following iterative procedure.
As a base case, set $r_0 = 0$.
Define $I_j$ as $[l_j, r_j]$, where $l_j = r_{j-1} + 1$ and $r_j$ is the largest $j \in [n]$ such that $\hat p(I_j) \leq 9\varepsilon^{3/2}/10$.
The exception is if $I_j$ would intersect $M$ -- in this case, we ``skip'' $M$: set $r_j = a-1$ and $l_{j+1} = b+1$.
If such a $j$ exists, denote it by $j^*$. 
We note that $p(I_j) \leq \hat p(I_j) + \varepsilon^{5/2}/10 \leq \varepsilon^{3/2}$.
Furthermore, for all $j$ except $j^*$ and $t$, $r_j + 1 \not \in M$,
so $p(I_j) \geq 9\varepsilon^{3/2}/10 - \varepsilon^{3/2}/5 - \varepsilon^{5/2}/10 \geq 3\varepsilon^{3/2}/5$. 
Observe that this lower bound implies that $t \leq \frac{2}{\varepsilon^{3/2}}$ for $\varepsilon$ sufficiently small. 


\paragraph{Part 1.}
For this part of the algorithm, we only care about the guarantees when
$p \in \mathcal{LCD}_n$, so we assume this is the case. 

For the domain $[n] \setminus M$, we let $f$ be the flattening of
$\hat p$ over the intervals $I_1, \dots I_t$. 
To analyze $f$, we need a structural property of log-concave
distributions due to Chan, Diakonikolas, Servedio, and Sun
\cite{ChanDSS13a}. This essentially states that a log-concave
distribution cannot have a sudden increase in probability. 

\begin{lemma}[Lemma 4.1 in \cite{ChanDSS13a}]
\label{lem:lcd-flat}
Let $p$ be a distribution over $[n]$ that is non-decreasing and
log-concave on $[1,x] \subseteq [n]$. 
Let $I = [x,y]$ be an interval of mass $P(I) = \tau$, and suppose that 
the interval $J = [1,x-1]$ has mass $p(J) = \sigma > 0$. Then 
$$p(y)/p(x) \leq 1 + \tau/\sigma.$$
\end{lemma}

Recall that any log-concave distribution is unimodal, and suppose the
mode of $p$ is at $i_0$.
We will first focus on the intervals $I_1, \dots, I_{t_L}$ which lie entirely to the left of $i_0$ and $M$.
We will refer to $I_j$ as $L_j$ for all $j \leq t_L$.
Note that $p$ is non-decreasing over these intervals.


The next steps to the analysis are as follows.
First we show that the flattening of $p$ over $L_j$ is a multiplicative $(1 + O(1/j))$ estimate for each $p_i \in L_j$.
Then, we show that flattening the empirical distribution $\hat p$ over $L_j$ is a multiplicative $(1 + O(1/j))$ estimate of $p(i)$ for each $i \in L_j$.
Finally, we exclude a small number of intervals (those corresponding to $O(\varepsilon)$ mass at the left and right side of the domain, as well as $j^*$) in order to get the $\chi^2$ approximation we desire on an effective support.
\begin{itemize}
\item First, recall that $p(L_j) \leq \varepsilon^{3/2}$ for all $j$.
Also, letting $J_j = [1, r_{j-1}]$, we have that $p(J_j) \geq (j-1)\cdot 3\varepsilon^{3/2}/5$.
Thus by Lemma \ref{lem:lcd-flat}, $p(r_j) \leq p(l_j) (1 + 2/(j-1))$.
Since the distribution is non-decreasing in $L_j$, the flattening $\bar p$ of $p$ is such that $\bar p(i) \in p(i) (1 \pm \frac{2}{j-1})$ for all $i \in L_j$.
\item We have that $p(L_j) \geq 3\varepsilon^{3/2}/5$, and $\hat p(L_j) \in p(L_j) \pm \varepsilon^{5/2}/10$, so $\hat p(L_j) \in p(L_j) \cdot (1 \pm \frac{\varepsilon}{6})$, and hence
$\hat p(i) \in \bar p(i) \cdot (1 \pm \frac{\varepsilon}{6})$ for all $i \in L_j$. 
Combining with the previous point, we have that 
$$\hat p(i) \in p(i) \cdot \left(1 \pm \left(\frac{2\varepsilon}{3(j-1)} + \frac{\varepsilon}{6} + \frac{2}{j-1}\right)\right) \in 
p(i) \cdot \left(1 \pm \frac{11}{3(j-1)}\right).$$
\end{itemize}

A symmetric statement holds for the intervals that lie entirely to
the right of $i_0$ and $M$. 
We will refer to $I_j$ as $R_{t-j}$ for all $j > t_L$. 

To summarize, we have the following guarantees for the distribution $f$:
\begin{itemize}
\item For all $i \in M$, $f(i) \in p(i) \cdot (1 \pm \varepsilon)$;
\item For all $i \in L_j$ (except $L_1$ and $L_{j^*}$), $f(i) \in p(i) \cdot \left(1 \pm \frac{22}{3j}\right)$;
\item For all $i \in R_j$ (except $R_1$), $f(i) \in p(i) \cdot \left(1 \pm \frac{22}{3j}\right)$;
\end{itemize}
Note that, in particular, we have multiplicative estimates for all intervals, except those in $L_1$, $L_{j^*}$, $R_1$ and the interval containing $i_0$.
Let $S$ be the set of all intervals except $L_{j^*}$, $L_j$ and $R_j$ for $j \leq 1/\sqrt{\varepsilon}$, and the one containing $i_0$ 
Then, since each interval has probability mass at most $O(\varepsilon^{3/2})$ and we are excluding $O(1/\sqrt{\varepsilon})$ intervals,
$p(S)>1-O(\varepsilon)$. 

We now compute the $\chi^2$-distance induced by this approximation for elements in $S$. 
For an element $i \in L_j \cap S$, we have
$$\frac{(f(i) - p(i))^2}{p(i)} \leq \frac{60p(i)}{j^2}.$$
Summing over all $i \in L_j \cap S$ gives
$$\frac{60\varepsilon^{3/2}}{j^2}$$
since the probability mass of $L_j$ is at most $\varepsilon^{3/2}$. 
Summing this over all $L_j$ for $j \geq 1/\sqrt{\varepsilon}$ and $j \neq j^*$ gives
\begin{align*}
  60\varepsilon^{3/2} \sum_{j = 1/\sqrt{\varepsilon}}^{2/\varepsilon^{3/2}} \frac{1}{j^2} &\leq 60\varepsilon^{3/2} \int_{1/\sqrt{\varepsilon}}^{\infty} \frac{1}{x^2} dx \\
                                                                  &= 60\varepsilon^{3/2}(\sqrt{\varepsilon}) \\
                                                                  &= O(\varepsilon^2)
\end{align*}
as desired. 
\paragraph{Part 2.} 
To obtain a distribution $q\in\mathcal{LCD}_n$, we write a linear program. 
We will work in the log domain, so our variables will be $Q_i$,
representing $\log q(i)$ for $i \in [n]$. 
We will use $F_i = \log f(i)$ as parameters in our LP.
There will be no objective function, we simply search for a feasible point.
Our constraints will be
$$Q_{i-1} + Q_{i+1} \leq 2 Q_{i} \ \ \forall i \in [n-1]$$
$$Q_{i} \leq 0 \ \ \forall i \in [n]$$
$$ \log (1 + \varepsilon) \leq |Q_i - F_i| \leq \log (1 + \varepsilon) \ \text{for}\ i \in M$$
$$ \log \left(1 - \frac{22}{3j}\right) \leq |Q_i - F_i| \leq \log \left(1 + \frac{22}{3j}\right) \ \text{for}\ i \in L_j, j \geq 1/\sqrt{\varepsilon} \ \text{and}\ j \neq j^*$$
$$ \log \left(1 - \frac{22}{3j}\right) \leq |Q_i - F_i| \leq \log \left(1 + \frac{22}{3j}\right) \ \text{for}\ i \in R_j, j \geq 1/\sqrt{\varepsilon}$$

If we run the linear program, then after a rescaling and summing the error over all the intervals in the LP gives us that the distance between $p$ and $q$ to be $O(\varepsilon^2)$ $\chi^2$-distance in a set $S$ which has measure $p(S) \geq 1 - 4\varepsilon$, as desired.

If the linear program finds a feasible point, then we obtain a $q \in \mathcal{LCD}_n$.
Furthermore, if $p \in \mathcal{LCD}_n$, this also tells us that (after a rescaling of $\varepsilon$), summing the error over all intervals implies that $\chi^2(p_S,q_S) \leq \frac{\varepsilon^2}{500}$ for a known set $S$ with $p(S) \geq 1 - O(\varepsilon)$, as desired.
If $M \neq \emptyset$, this algorithm works as described.
The issue is if $M = \emptyset$, then we don't know when the $L$ intervals end and the $R$ intervals begin.
In this case, we run $O(1/\varepsilon)$ LPs, using each interval as the one containing $i_0$, and thus acting as the barrier between the $L$ intervals (to its left) and the $R$ intervals (to its right).
If $p$ truly was log-concave, then one of these guesses will be correct and the corresponding LP will find a feasible point.
\end{prevproof}

\section{Lower Bounds}
\label{sec:lower-bounds}
We now prove sharp lower bounds for the classes of distributions we
consider. We show that the example studied by
Paninski~\cite{Paninski08} to prove lower bounds on testing uniformity
can be used to prove lower bounds for the classes we consider.  
They consider a class $\mathcal{Q}$ consisting of $2^{n/2}$ distributions
defined as follows. Without loss of generality assume that $n$ is even. For each of
the $2^{n/2}$ vectors $z_0z_1\ldots
z_{n/2-1}\in\{-1,1\}^{n/2}$, define a distribution $q\in\mathcal{Q}$ over
$[n]$ as follows.  
\begin{align}
\label{eqn:fcl}
q_{i} =\begin{cases}
\frac{(1+z_\ell c\varepsilon)}{n} & \text{ for } i = 2\ell+1\\
 \frac{(1-z_{\ell}c\varepsilon)}{n}& \text{ for } i=2\ell.\\
\end{cases}
\end{align}

Each distribution in $\mathcal{Q}$ has a total variation distance $c\varepsilon/2$
from $U_n$, the uniform distribution over
$[n]$. By choosing $c$ to be an appropriate constant, Paninski~\cite{Paninski08} showed that a distribution picked
uniformly at random from $\mathcal{Q}$ cannot be distinguished from $U_n$ with
fewer than $\sqrt{n}/\varepsilon^2$ samples with probability at least $2/3$. 

Suppose $\mathcal{C}$ is a class of distributions such that 
\begin{itemize}
\item
The uniform distribution $U_n$ is in $\mathcal{C}$,
\item
For appropriately chosen $c$, $d_{\mathrm {TV}}(\mathcal{C}, \mathcal{Q})\ge\varepsilon$,
\end{itemize}
then testing $\mathcal{C}$ is \costasnote{not} easier than distinguishing $U_n$ from $\mathcal{Q}$. 
Invoking~\cite{Paninski08} immediately implies that testing the class
$\mathcal{C}$ requires $\Omega(\sqrt{n}/\varepsilon^2)$ samples. 

The lower bounds for all the one dimensional distributions will follow
directly from this construction, and for testing monotonicity in
higher dimensions, we extend this construction to $d\ge1$,
appropriately. These arguments are proved in
Section~\ref{sec:lb-appendix}, leading to the following lower bounds for testing these classes: 
\begin{theorem}$ $
\label{thm:lbs}
\begin{itemize}
\item
For any $d\ge1$, any algorithm for testing monotonicity over
$[n]^d$ requires $\Omega(n^{d/2}/\varepsilon^2)$ samples.  
\item
For $d\ge1$, any algorithm for testing independence over
$[n_1]\times\cdots\times[n_d]$ requires
$\Omega\left(\frac{(n_1\cdot n_2\ldots\cdot
    n_d)^{1/2}}{\varepsilon^2}\right)$ samples. 
\item
Any algorithm for testing unimodality, log-concavity, or monotone
hazard rate over $[n]$ requires $\Omega(\sqrt{n}/\varepsilon^2)$ samples.
\end{itemize}
\end{theorem}



\section{Introduction} \label{sec:introduction}

The quintessential scientific question is whether an unknown object
has some property, i.e. whether a model from a specific class fits  
the object's observed behavior. If the unknown object is
a probability distribution, $p$, to which we have sample access,
we are typically asked to distinguish whether $p$ belongs to some class 
$\mathcal{C}$ or whether it is sufficiently far from it.

This question has received tremendous attention in \gnote{the field of} statistics (see, e.g.,~\cite{Fisher25,lehmann2006testing}), where
test statistics for important properties such as the ones we
consider here have been proposed. Nevertheless, the 
emphasis has been on asymptotic
analysis, characterizing \gnote{the} rates of convergence of test statistics under null
hypotheses, as the number of samples tends to infinity. In contrast, 
\gnote{we wish to study the following problem in the small sample regime:}

\vspace{2ex}
\begin{center}
\smallskip \framebox{
\begin{minipage}{13.5cm} $\Pi(\mathcal{C},\varepsilon)$: Given a family of distributions $\mathcal{C}$, some
$\varepsilon>0$, and sample access to an unknown distribution $p$ over \gnote{a}
discrete support, how many samples are required to distinguish
between $p \in \mathcal{C}$ versus $d_{\mathrm {TV}}(p,\mathcal{C})>\varepsilon$?
\end{minipage}
}
\end{center}

\vspace{2ex}
The problem has been studied  intensely in the literature on \gnote{property
testing and sublinear algorithms} \cite{Goldreich98, Fischer01, Rubinfeld06, Ron08,canonne2015survey}, where the emphasis has been on
characterizing the optimal tradeoff between $p$'s support size and
the accuracy $\varepsilon$ in the number of samples. Several results have
been obtained, roughly clustering into three groups, where (i) $\mathcal{C}$
is the class of monotone distributions over $[n]$, or more generally a
poset~\cite{BatuKR04,Bhattacharyya11}; (ii) $\mathcal{C}$ is the
class of independent, or $k$-wise independent distributions over a
hypergrid~\cite{batu2001testing,alon2007testing}; and
(iii) $\mathcal{C}$ contains a single-distribution $q$, and the problem
becomes that of testing whether $p$ equals $q$ or is far from
it~\cite{batu2001testing,Paninski08, valiant2014automatic}. 

With respect to (iii), \cite{valiant2014automatic} exactly characterizes the number of samples required to test identity to each distribution $q$, providing a single tester matching this bound simultaneously for all $q$. Nevertheless, this tester and its precursors are not applicable to the composite identity testing problem that we consider. If our class $\mathcal{C}$ were finite, we could test against each element in the class, albeit this would not necessarily be sample optimal. If our class $\mathcal{C}$ \gnote{were} a continuum, we \gnote{would}  need \emph{tolerant} identity testers, which tend to be more expensive in terms of \gnote{sample complexity} \cite{ValiantV11}, and result in substantially suboptimal testers for the classes we consider. Or we could use approaches related to generalized likelihood ratio test, but their behavior is not well-understood in our regime, and optimizing likelihood over our classes becomes computationally intense.


\paragraph{Our Contributions} In this paper, we obtain sample-optimal and computationally efficient testers for $\Pi(\mathcal{C},\varepsilon)$ for the most \gnote{fundamental} shape restrictions to a distribution. 
Our contributions are the following:
\begin{enumerate}
\item
\jnote{For a known distribution $q$ over $[n]$, and given samples from an unknown
distribution $p$, we show that distinguishing the cases: $(a)$ whether the $\chi^2$-distance
between $p$ and $q$ is at most $\varepsilon^2/2$,  versus $(b)$ the $\ell_1$ distance
between $p$ and $q$ is at least $\varepsilon$, requires $\Theta(\sqrt
n/\varepsilon^2)$ samples. 
As a corollary, we provide a simpler argument to
show that identity testing requires $\Theta(\sqrt n/\varepsilon^2)$ samples (previously shown in \cite{valiant2014automatic}). }
\item 
For the class $\mathcal{C}=\mathcal{M}_n^d$ of monotone distributions over
  $[n]^d$ we require an optimal $\Theta\left({n^{d/2} \over
      \varepsilon^2}\right)$ number of samples, where prior work requires
  $\Omega\left({\sqrt{n} \log n \over \varepsilon^4}\right)$ samples for $d=1$
  and $\tilde{\Omega}\left(n^{d-{1\over 2}} {\rm poly}\left({1 \over
      \varepsilon}\right)\right)$ for $d>1$~\cite{BatuKR04,Bhattacharyya11}. Our
  results improve the exponent of $n$ with respect to $d$, shave all
  logarithmic factors in $n$, and improve the exponent of $\varepsilon$ by at
  least a factor of $2$. 
\begin{enumerate}
\item A useful building block and interesting byproduct of our
  analysis is extending Birg\'e's oblivious decomposition for
  single-dimensional monotone distributions~\cite{Birge87} to monotone
  distributions in $d\ge1$, and to the stronger notion of $\chi^2$-distance. See Section~\ref{sec:hypergrid}. 

\item Moreover, we show that $O(\log^d n)$ samples suffice to learn a monotone
distribution over $[n]^d$ in $\chi^2$-distance. See
Lemma~\ref{lem:fin-learn-mon} for the precise statement.
\end{enumerate}
\item
\gnote{For the class $\mathcal{C} = \Pi_d$ of product distributions over $[n_1]\times\cdots\times[n_d]$,
our algorithm requires
$O\left(\left((\prod_\ell n_\ell)^{1/2} + \sum_\ell
n_\ell\right)/\varepsilon^2\right)$ samples.
We note that a product distribution is one where all marginals are independent, so this is equivalent to testing if a collection of random variables are all independent.}
\gnote{In the case where
$n_\ell$'s are large, then the first term dominates, and the sample
complexity is $O(\left(\prod_\ell n_\ell\right)^{1/2}/\varepsilon^2)$. 
In particular, when $d$ is a constant and all $n_\ell$'s are equal to $n$, we achieve the optimal sample complexity of $\Theta(n^{d/2}/\varepsilon^2)$.
To the best of our knowledge, this is the first result for $d \geq 3$, and when $d = 2$, this improves the previously known complexity from $O\left(\frac{n}{\varepsilon^6}{\rm
    polylog}(n/\varepsilon)\right)$ \cite{batu2001testing,levi2013testing}, significantly improving the dependence on $\varepsilon$ and shaving all logarithmic factors.
}
\item For the classes $\mathcal{C}=\mathcal{LCD}_n$, $\mathcal{C}=\mathcal{MHR}_n$ and $\mathcal{C}=\mathcal{U}_n$ of log-concave, monotone-hazard-rate and unimodal distributions over $[n]$, we require an optimal $\Theta\left({\sqrt{n} \over \varepsilon^2}\right)$ number of samples. Our testers for $\mathcal{LCD}_n$ and $\mathcal{C}=\mathcal{MHR}_n$ are to our knowledge the first for these classes for the low sample regime we are studying---see~\cite{hall2005testing} and its references for statistics literature on the asymptotic regime. Our tester for $\mathcal{U}_n$ improves the dependence of the sample complexity on $\varepsilon$ by at least a factor of $2$ in the exponent, and shaves all logarithmic factors in $n$, compared to testers based on testing monotonicity.
\begin{enumerate}
\item A useful building block and important byproduct of our analysis
  are the first computationally efficient algorithms for properly
  learning log-concave and monotone-hazard-rate distributions, to
  within $\varepsilon$ in total variation distance, from ${\rm
    poly}(1/\varepsilon)$ samples, independent of the domain size
  $n$. See Corollaries~\ref{cor:learning LCD}
  and~\ref{cor:learning MHR}. Again, these are the first
  computationally efficient algorithms to our knowledge in the low
  sample regime.~\cite{AcharyaDLS15, ChanDSS13b} provide algorithms for density
  estimation, which are non-proper, i.e. will approximate an unknown
  distribution from these classes with a distribution that does not
  belong to these classes. On the other hand, the statistics
  literature focuses on maximum-likelihood estimation in the
  asymptotic regime---see e.g. \cite{cule2010theoretical} and its
  references. 
\end{enumerate}

\item For all the above classes we obtain matching lower bounds,
  showing that the sample complexity of our testers is optimal with
  respect to $n$, $\varepsilon$ and when applicable $d$. See
  Section~\ref{sec:lower-bounds}. Our lower bounds are based on
  extending Paninski's lower bound for testing uniformity~\cite{Paninski08}. 
\end{enumerate}


At the heart of our tester lies a novel use of the $\chi^2$
statistic. Naturally, the $\chi^2$ and its related $\ell_2$ statistic
have  been used in several of the afore-cited results. We propose a
new use of the $\chi^2$ statistic enabling  our optimal sample
complexity. The essence of our approach is to first draw a small
number of samples (independent of $n$ for log-concave and
monotone-hazard-rate distributions and only logarithmic in $n$ for
monotone and unimodal distributions) to approximate the unknown
distribution $p$ in $\chi^2$ distance. If $p \in \mathcal{C}$, our learner
is required to output a distribution $q$ that is $O(\varepsilon)$-close
to $\mathcal{C}$ in total variation  and $O(\varepsilon^2)$-close to $p$ in
$\chi^2$ distance. Then some analysis reduces our testing problem
to distinguishing the following cases: 
\begin{itemize}
\item 
$p$ and $q$ are $O(\varepsilon^2)$-close in $\chi^2$ distance; this case
corresponds to $p \in \mathcal{C}$. 
\item 
$p$ and $q$ are $\Omega(\varepsilon)$-far in total variation distance; this
case corresponds to $d_{\mathrm {TV}}(p,\mathcal{C})>\varepsilon$. 
\end{itemize}

\gnote{
We draw a comparison with \emph{robust identity testing}, in which one
must distinguish whether $p$ and $q$ are $c_1\varepsilon$-close or
$c_2\varepsilon$-far in total variation distance, for constants $c_2> c_1 >
0$. 
In \cite{ValiantV11}, Valiant and Valiant show that $\Omega(n/\log n)$
samples are required for this problem -- a nearly-linear sample
complexity, which may be prohibitively large in many settings. 
In comparison, the problem we study tests for $\chi^2$ closeness
rather than total variation closeness: a relaxation of the previous
problem. 
However, our tester demonstrates that this relaxation allows us to
achieve a substantially sublinear complexity of $O(\sqrt{n}/\varepsilon^2)$. 
On the other hand, this relaxation is still tight enough to be useful,
demonstrated by our application in obtaining sample-optimal testers. 
}

\jnote{We note that while the $\chi^2$ statistic for testing
  hypothesis is prevalent in statistics providing optimal error
  exponents in the large-sample regime, to the best of our knowledge,
  in the small-sample regime, \emph{modified-versions} of the $\chi^2$
  statistic have only been recently used for \emph{closeness-testing}
  in~\cite{AcharyaDJOPS12, ChanDVV13} and for testing uniformity of
  monotone distributions in~\cite{AcharyaJOS13}. In
  particular,~\cite{AcharyaDJOPS12} design an unbiased statistic for
  estimating the $\chi^2$ distance between two \emph{unknown}
  distributions.} 

In Section~\ref{sec:testing}, we show that a version of the $\chi^2$
statistic, appropriately excluding certain elements of the support,
is sufficiently well-concentrated to distinguish between the above
cases. Moreover, the sample complexity of our algorithm is optimal for
most classes. 
Our base tester is combined with the afore-mentioned extension of
Birg\'e's decomposition theorem to test monotone distributions in
Section~\ref{sec:monotone} (see Theorem~\ref{thm:monotone-final} 
and Corollary~\ref{cor:high-d}), and is also used to test independence
of distributions in Section~\ref{sec:independence} (see
Theorem~\ref{thm:independence}). 

Naturally, there are several bells and whistles that we need to add to
the above skeleton to accommodate all classes of distributions that we
are considering. For log-concave and monotone-hazard distributions, we
are unable to obtain a cheap (in terms of samples) learner that
$\chi^2$-approximates the unknown distribution $p$ throughout its
support. Still, we can identify a subset of the support where the
$\chi^2$-approximation is tight and which captures almost all the
probability mass of $p$. We extend our tester to accommodate
excluding subsets of the support from the $\chi^2$-approximation. See
Theorems~\ref{thm:lcd-main} and~\ref{thm:mhr-main} in
Sections~\ref{sec:LCD-main} and \ref{sec:MHR-main}. 

For unimodal distributions, we are even unable to identify a large
enough subset of the support where the $\chi^2$ approximation is
guaranteed to be tight. But we can show that there exists a light
enough piece of the support (in terms of probability mass under $p$)
that we can exclude to make the $\chi^2$ approximation tight. Given
that we only use Chebyshev's inequality to prove the concentration of
the test statistic, it would seem that our lack of knowledge of the
piece to exclude would involve a union bound and a corresponding
increase in the required number of samples. We avoid this through a
careful application of Kolmogorov's max inequality in our setting. See
Theorem~\ref{thm:unimodality} of Section~\ref{sec:unimodal-main}. 

\paragraph{Related Work.} \jnote{For the problems that we study in
  thie paper, we have provided the 
related works in the previous section along with our contributions.} 
We cannot do justice to the role of shape
restrictions of probability distributions in probabilistic modeling
and testing. 
It suffices to say that the classes of distributions that
we study are fundamental, motivating extensive literature on their
learning and testing~\cite{BBBB:72}. In the recent times, there has
been work on shape restricted statistics, pioneered by Jon Wellner,
and others.~\cite{JW:09,BW10sn} study estimation of monotone and $k-$
monotone densities, and ~\cite{BJR11,SumardW14} study estimation of
log-concave distributions.

As we have mentioned, statistics has focused on the asymptotic
regime as the number of samples tends to infinity. Instead we are
considering the low sample regime and  are more stringent about the
behavior of our testers, requiring $2$-sided guarantees. We want to
accept if the unknown distribution is in our class of interest, and
also reject if it is far from the class. For this problem, as
discussed above, there are few results when $\mathcal{C}$ is a whole class of
distributions. Closer related to our paper is the line of
papers~\cite{BatuKR04, ACS10, Bhattacharyya11} for monotonicity testing, albeit these 
papers have sub-optimal sample complexity as discussed above.
\gnote{Testing independence of random variables has a long history in
  statisics~\cite{rao1981analysis, agresti2011categorical}. The theoretical
  computer science community has also considered 
  the problem of testing independence of two random
  variables~\cite{batu2001testing, levi2013testing}.  
While our results sharpen the case where the variables are over domains of equal size,
they demonstrate an interesting asymmetric upper bound when this is not the case.} 
More recently, Acharya and Daskalakis provide optimal testers for the
family of Poisson Binomial Distributions~\cite{AcharyaD15}. 

\gnote{
Finally,
contemporaneous work of Canonne et al~\cite{CanonneDGR15a,CanonneDGR15b} provides
a generic algorithm and lower bounds for the single-dimensional families of distributions considered here.
We note that their algorithm has a sample complexity which is suboptimal in both $n$ and $\varepsilon$, while our algorithms are optimal.
Their algorithm also extends to mixtures of these classes, though some of these extensions are not computationally efficient.
They also provide a framework for proving lower bounds, giving the optimal bounds for many classes when $\varepsilon$ is sufficiently large with respect to $1/n$. 
In comparison, we provide these lower bounds unconditionally by modifying Paninski's construction \cite{Paninski08} to suit the classes we consider.
}

 
 




\input{preliminaries}
\input{overview}
\input{testing}
\input{monotone}
\input{unimodal-main}
\input{independence}
\input{otherclasses}
\input{lower-bounds}


\section*{Acknowledgements}
The authors thank Cl\'ement Canonne and Jerry Li; the former for several useful comments and suggestions on previous drafts of this work, and both for helpful discussions and thoughts regarding independence testing.
\bibliographystyle{alpha}


\section{Details on MHR testing}
\label{sec:MHR}
\begin{prevproof}{Lemma}{lem:mhr-learn}
As with log-concave distributions, our method for MHR distributions can be split into two parts.
In the first step, if $p \in \mathcal{MHR}_n$, we obtain a distribution $q$ which is $O(\varepsilon^2)$-close to $p$ in $\chi^2$ distance on a set $\mathcal{A}$ of intervals such that $p(\mathcal{A}) \geq 1 - O(\varepsilon)$.
$q$ will achieve this by being a multiplicative $(1 + O(\varepsilon))$ approximation for each element within these intervals.
This step is very similar to the decomposition used for unimodal distributions (described in Section~\ref{sec:unimodal-appendix}), so we sketch the argument and highlight the key differences.

The second step will be to find a feasible point in a linear program.
If $p \in \mathcal{MHR}_n$, there should always be a feasible point, indicating that $q$ is close to a distribution in $\mathcal{MHR}_n$ (leveraging the particular guarantees for our algorithm for generating $q$).
If $d_{\mathrm {TV}}(p,\mathcal{MHR}_n) \geq \varepsilon$, there may or may not be a feasible point, but when there is, it should imply the existence of a distribution $p^* \in \mathcal{MHR}_n$ such that $d_{\mathrm {TV}}(q,p^*) \leq \varepsilon/2$. 

The analysis will rely on the following lemma from \cite{ChanDSS13a}, which roughly states that an MHR distribution is ``almost'' non-decreasing.
\begin{lemma}[Lemma 5.1 in \cite{ChanDSS13a}]
\label{lem:mhr-str}
Let $p$ be an MHR distribution over $[n]$.
Let $I = [a,b] \subset [n]$ be an interval, and $R = [b+1,n]$ be the elements to the right of $I$.
Let $\eta = p(I)/p(R)$. Then $p(b+1) \geq \frac{1}{1 + \eta}p(a)$. 
\end{lemma}

\paragraph{Part 1.}
As before, with unimodal distributions, we start by taking $O(\frac{b \log b}{\varepsilon^2})$ samples, with the goal of partitioning the domain into intervals of mass approximately $\Theta(1/b)$.
First, we will ignore the left and rightmost intervals of mass $\Theta(\varepsilon)$.
For all ``heavy'' elements with mass $\geq \Theta(1/b)$, we consider them as singletons.
We note that Lemma~\ref{lem:mhr-str} implies that there will be at most $O(1/\varepsilon)$ contiguous intervals of such elements.
The rest of the domain is greedily divided (from left to right) into intervals of mass $\Theta(1/b)$, cutting an interval short if we reach one of the heavy elements.
This will result in the guarantee that all but potentially $O(1/\varepsilon)$ intervals have $\Theta(1/b)$ mass.

Next, similar to unimodal distributions, considering the flattened distribution, we discard all intervals for which the per-element probability is not within a $(1 \pm O(\varepsilon))$ multiplicative factor of the same value for both neighboring intervals.
The claim is that all remaining intervals will have the property that the per-element probability is within a $(1 \pm O(\varepsilon))$ multiplicative factor of the true probability.
This is implied by Lemma~\ref{lem:mhr-str}. 
If there were a point in an interval which was above this range, the distribution must decrease slowly, and the next interval would have a much larger per-element weight, thus leading to the removal of this interval.
A similar argument forbids us from missing an interval which contains a point that lies outside this range.
Relying on the fact that truncating the left and rightmost intervals eliminates elements with low probability mass, similar to the unimodal case, one can show that we will remove at most $\log(n/\varepsilon)/\varepsilon$ intervals, and thus a $\log(n/\varepsilon)/b\varepsilon$ probability mass. 
Choosing $b = \Omega(\varepsilon^2/\log(n/\varepsilon))$ limits this to be $O(\varepsilon)$, as desired.
At this point, if $p$ is indeed MHR, the multiplicative estimates guarantee that the result is $O(\varepsilon^2)$-close in $\chi^2$-distance among the remaining intervals.

\paragraph{Part 2.}
We note that an equivalent condition for distribution $f$ being MHR is log-concavity of $\log(1 - F)$, where $F$ is the CDF of $f$.
Therefore, our approach for this part will be similar to the approach used for log-concave distributions.

Given the output distribution $q$ from the previous part of this algorithm, our goal will be check if there exists an MHR distribution $f$ which is $O(\varepsilon)$-close to $q$.
We will run a linear program with variables $\mathfrak{f}_i = \log(1 - F_i)$.
First, we ensure that $f$ is a distribution.
This can be done with the following constraints:
\begin{alignat*}{3}
&\mathfrak{f}_i &&\leq 0 \hphantom{space}&&\forall i \in [n] \\
&\mathfrak{f}_i &&\geq \mathfrak{f}_{i+1} &&\forall i \in [n-1] \\
&\mathfrak{f}_n &&= -\infty
\end{alignat*}
To ensure that $f$ is MHR, we use the following constraint:
\begin{alignat*}{3}
&\mathfrak{f}_{i-1} + \mathfrak{f}_{i+1}  &&\leq 2\mathfrak{f}_i \hphantom{space}&&\forall i \in [2,n-1]
\end{alignat*}
Now, ideally, we would like to ensure $f$ and $q$ are $\varepsilon$-close in total variation distance by ensuring they are pointwise within a multiplicative $(1 \pm \varepsilon)$ factor of each other:
$$(1 - \varepsilon) \leq f_i/q_i \leq (1 + \varepsilon)$$
We note that this is a stronger condition than $f$ and $q$ being $\varepsilon$-close, but if $p \in \mathcal{MHR}_n$, the guarantees of the previous step would imply the existence of such an $f$.

We have a separate treatment for the identified singletons (i.e., those with probability $\geq 1/b$) and the remainder of the support.
For each element $q_i$ identified to have $\geq 1/b$ mass, we add two constraints:
$$\log((1-\varepsilon/2b)(1 - Q_i)) \leq \mathfrak{f}_i \leq \log((1+\varepsilon/2b)(1 - Q_i))$$
$$\log((1-\varepsilon/2b)(1 - Q_{i-1})) \leq \mathfrak{f}_{i-1} \leq \log((1+\varepsilon/2b)(1 - Q_{i-1}))$$
If we satisfy these constraints, it implies that
$$q_i - \varepsilon/b \leq f_{i} \leq q_i + \varepsilon/b.$$
Since $q_i \geq 1/b$, this implies $$(1 - \varepsilon)q_i \leq f_i \leq (1 + \varepsilon)q_i$$
as desired. 

Now, the remaining elements each have $\leq 1/b$ mass.
For each such element $q_i$, we create a constraint
$$(1 - O(\varepsilon))\frac{q_i}{1 - Q_{i-1}} \leq \mathfrak{f}_{i-1} - \mathfrak{f}_i  \leq (1 + O(\varepsilon))\frac{q_i}{1 - Q_{i-1}} $$
Note that the middle term is
$$-\log\left(\frac{1 - F_i}{1 - F_{i-1}}\right) = -\log\left(1 - \frac{f_i}{1 - F_{i-1}}\right) \in \frac{f_i}{1 - F_{i-1}}\left(1 \pm 2\varepsilon\right),$$
where the second equality uses the Taylor expansion and the facts that $f_i \leq 1/b$ and $1 - F_{i-1} \geq \varepsilon$ (since during the previous part, we ignored the rightmost $O(\varepsilon)$ probability mass).
If we satisfy the desired constraints, it implies that
$$f_i \in \frac{1}{\left(1 \pm 2\varepsilon\right)}\frac{1 - F_{i-1}}{1 - Q_{i-1}}(1 \mp O(\varepsilon))q_i.$$
Since we are taking $\Omega(1/\varepsilon^4)$ samples and $1 - F_{i-1} \geq \Omega(\varepsilon)$, Lemma~\ref{lem:dkw} implies that $f_i$ is indeed a multiplicative $(1 \pm \varepsilon)$ approximation for these points as well.

We note that all points which do not fall into these two cases make up a total of $O(\varepsilon)$ probability mass.
Therefore, $f$ may be arbitrary at these points and only incur $O(\varepsilon)$ cost in total variation distance.

If we find a feasible point for this linear program, it implies the existence of an MHR distribution within $O(\varepsilon)$ total variation distance.
In this case, we continue to the testing portion of the algorithm.
Furthermore, if $p \in \mathcal{MHR}_n$, our method for generating $q$ certifies that such a distribution exists, and we continue on to the testing portion of the algorithm.
\end{prevproof}

\section{Testing for Monotone Hazard Rate}
\label{sec:MHR}
\gnote{Insert prose}


\section{Details on Testing Monotonicity}
\label{sec:monotone-appendix}
In this section, we prove the lemmas necessary for our monotonicity testing result.




\subsection{A Structural Result for Monotone Distributions on the Hypergrid}
\label{sec:hypergrid}
Birg\'e~\cite{Birge87} showed that any monotone
distribution is estimated to a total variation $\varepsilon$ with a
$O(\log(n)/\varepsilon)$-piecewise constant distribution. Moreover, the
intervals over which the output is constant is 
independent of the distribution $p$. This result, was strengthened
to the Kullback-Leibler divergence by~\cite{AcharyaJOS14a} to study the compression of
monotone distributions. They upper bound the KL divergence by
$\chi^2$ distance and then bound the $\chi^2$ distance. 
We extend this result to $[n]^d$.
We divide $[n]^d$ into $b^d$ rectangles
as follows.
Let $\{I_1, \dots, I_b\}$ be a partition of
$[n]$ into consecutive intervals defined as:
\begin{align*}
|I_j|

\begin{cases}
1  & \text{ for } 1\le j\le\frac b2, \\
\lfloor 2(1+\gamma)^{j-b/2} \rfloor & \text{ for }\frac b2< j\le b.
 \end{cases}
\end{align*}
For ${\bf j}=(j_1,\ldots, j_d)\in[b]^d$, let $I_{\bf j}{\hfill\blksquare}\medskip I_{j_1}\times
I_{j_2}\times\ldots\times I_{j_d}$. 

 The $\chi^2$ distance between $p$ and $\bar{\dP}$
can be bounded as
\begin{align*}
\chi^2(p,\bar{\dP}) 
=& \left[\sum_{{\bf j}\in[b]^d}\sum_{{\bf i}\in
    I_{{\bf j}}}\frac{{\dP}_{{\bf i}}^2}{{\dPbar}_{{\bf i}}}\right]-1\\
\le& \left[\sum_{{\bf j}\in[b]^d}{\dP}_{{\bf j}}^+|I_{\bf j}|\right]-1\\
\end{align*}
For ${\bf j}=(j_1,\ldots, j_d)\in\mathcal{S}_{\rm large}$, let ${\bf j}^*=(j_1^*,\ldots,
j_b^*)$ be 
\begin{align*}
j_i^* =\begin{cases}
j_i& \text{ if } j_i\le b/2+1\\
j_i-1& \text{ otherwise}.
\end{cases}
\end{align*}

We bound the expression above as follows. 

Let $T \subseteq [d]$ be any subset of $d$. Suppose the size of $T$ is
$\ell$.  Let $\bar T$ be the set of all ${\bf j}$ that satisfy ${\bf j}_i=b/2+1$ for $i\in
T$. In other words, over the dimensions determined by $T$, the value
of the index is equal to $d/2+1$. The map ${\bf j}\rightarrow{\bf j}^*$ restricted to
$T$ is one-to-one, and since at most $d-\ell$ of the coordinates drop, 
\begin{align*}
|I_{{\bf j}}| \le |I_{{\bf j}^*}|\cdot(1+\gamma)^{d-\ell}. 
\end{align*}
Since there are $\ell$ coordinates that do not change, 
and each of them have $2(1+\gamma)$ coordinates, we obtain
\begin{align*}
\sum_{{\bf j}\in\bar T}p_{\bf j} \le& 
\ \sum_{{\bf j}\in\bar T}p_{{\bf j}^*}^-\cdot|I_{{\bf j}}|\cdot
(2(1+\gamma))^{\ell}\cdot(1+\gamma)^{d-\ell}\\
=&\ \sum_{{\bf j}\in\bar T}p_{{\bf j}^*}^-\cdot|I_{{\bf j}^*}| \cdot 2^{\ell} (1+\gamma)^{d}.
\end{align*}

Since the mapping is one-to-one, the probability of observing as element in
$\bar T$ is the probability of observing $b/2+1$ in $\ell$
coordinates, which is at most $(2/(b+2))^\ell$ under any monotone
distribution. Therefore,
\begin{align*}
\sum_{{\bf j}\in\bar T}p_{\bf j} \le&\left(\frac2{b+2}\right)^\ell \cdot 2^{\ell} (1+\gamma)^{d}.
\end{align*}
For any $\ell$ there are ${d\choose \ell}$ choices for $T$.
Therefore, 
\begin{align}
\chi^2(p,\bar{\dP})\le\ & \sum_{\ell=0}^d {d\choose \ell}
\left(\frac4{b+2}\right)^\ell (1+\gamma)^{d}-1\nonumber\\
=&\ (1+\gamma)^d\left(1+\frac4{b+2}\right)^d-1\nonumber\\
=&  \ \left(1+\gamma+\frac4{b+2}+\frac{4\gamma}{b+2}\right)^d-1\nonumber
\end{align}

Recall that $\gamma=2\log (n)/b>1/b$, implies that the expression above
is at most $(1+2\gamma)^d-1$. 
This implies Lemma~\ref{lem:mon-str}.

\subsection{Monotone Learning}

Our algorithm requires a distribution $q$ satisfying the properties discussed earlier. We learn a monotone distribution from samples as follows. 

Before proving this result, we prove a general result for $\chi^2$ learning of arbitrary discrete distributions, adapting the result from~\cite{KamathOPS15}.
For a distribution $p$, and a partition of the domain into $b$ intervals $I_1, \ldots, I_b$, let $\bar{\dP}_i= p(I_i)/|I_i|$ be the flattening of $p$ over these intervals. We saw that for monotone distributions there exists a partition of the domain such that $\bar{\dP}$ is \emph{close} to the underlying distribution in $\chi^2$ distance. 

Suppose we are given $m$ samples from a distribution $p$ and a partition $I_1, \ldots, I_b$. Let $m_j$ be the number of samples that fall in $I_j$. For $i\in I_j$, let
\[
\dQ_i {\hfill\blksquare}\medskip \frac{1}{|I_j|}\frac{m_j+1}{m+b}.
\]

Let $S_j=\sum_{i\in I_j}\dP_i^2$. The expected $\chi^2$ distance between $p$ and $q$ can be bounded as follows. 
\begin{align}
\expectation{\chi^2(p,q) } =& \left[\sum_{j=1}^b \sum_{i\in I_j} \sum_{\ell=0}^{m}{m\choose \ell}(p(I_j))^{\ell}(1-p(I_j))^{m-\ell}\frac{\dP_i^2}{(\ell+1)/(|I_j|(m+b))}\right]-1\nonumber\\
= &\left[\frac{m+b}{m+1}\sum_{j=1}^b \frac{S_j}{\bar{\dP}(I_j)/|I_j|} \left(\sum_{\ell=0}^{m}{m+1\choose \ell+1}(p(I_j))^{\ell+1}(1-p(I_j))^{m+1-\ell+1}\right)\right]-1 \nonumber\\
= & \left[\frac{m+b}{m+1}\sum_{j=1}^b \frac{S_j}{\bar{\dP}(I_j)/|I_j|} \left(1-(1-p(I_j)^{m+1}\right)\right]-1\nonumber\\
\le & \left[\frac{m+b}{m+1}\sum_{j=1}^b \frac{S_j}{\bar{\dP}(I_j)/|I_j|}\right]-1\nonumber\\
= & \left[\frac{m+b}{m+1}\left(\chi^2(p,\bar{\dP})+1\right)\right]-1\nonumber\\
= & \frac{m+b}{m+1}\cdot\chi^2(p,\bar{\dP})+\frac{b}{m+1}\label{eqn:chi-learn}.
\end{align}

Suppose $\gamma = O(\log (n)/b)$, and $b=O(d\cdot\log (n)/\varepsilon^2)$. Then, by Lemma~\ref{lem:mon-str}, 
\begin{align}
\chi^2(p,\bar{\dP})\le \varepsilon^2.
\end{align}

Combining this with~\eqref{eqn:chi-learn} gives Lemma~\ref{lem:fin-learn-mon}.



\section{Testing Monotonicity}
\label{sec:monotone}
As an application of our testing framework, we will demonstrate how to
test for monotonicity. Let $d\geq1$, and ${\bf i}=(i_1, \ldots,
i_d),{\bf j}=(j_1, \ldots, j_d)\in[n]^d$. We say
${\bf i}\succcurlyeq{\bf j}$ if $i_l>j_l$ for $l=1,\ldots,d$. 
\begin{definition}
A distribution $p$ over $[n]^d$
is monotone (decreasing) if for all ${\bf i}\succcurlyeq{\bf j}$, $p_{{\bf i}}\le
p_{{\bf j}}$. 
\end{definition}
Our main result of this section is as follows:
\begin{theorem}
\label{thm:monotone-final}
For any $d \geq 1$, there exists an algorithm for testing monotonicity over $[n]^d$ with sample complexity
$$O\left(\frac{n^{d/2}}{\varepsilon^2}+\left(\frac{d\log n}{\varepsilon^2}\right)^d\cdot \frac1{\varepsilon^2}\right)$$
and time complexity $O\left(\frac{n^{d/2}}{\varepsilon^2} + \poly(\log n, 1/\varepsilon)^d \right)$. 
\end{theorem}

In particular, this implies the following optimal algorithms for
monotonicity testing for all $d \geq 1$: 
\begin{corollary}
\label{cor:high-d}
Fix any $d \geq 1$, and suppose $\varepsilon>\frac{\sqrt{d\log n}}{n^{1/4}}$. 
Then there exists an algorithm for testing monotonicity over $[n]^d$
with sample complexity $O\left(n^{d/2}/\varepsilon^2\right)$. 
\end{corollary}

Our analysis starts with a structural lemma about monotone distributions.
In \cite{Birge87}, Birg\'e showed that any monotone distribution $p$
over $[n]$ can be \emph{obliviously} decomposed into $O(\log(n)/\varepsilon)$
intervals, such that the flattening $\bar p$ (recall
Definition~\ref{def:flattening}) of $p$ over these intervals is 
$\varepsilon$-close to $p$ in total variation distance. 
\cite{AcharyaJOS14a} extend this result, giving a bound between the
$\chi^2$-distance of $p$ and $\bar p$. 
We strengthen these results by extending them to monotone distributions over $[n]^d$.
In particular, we partition the domain $[n]^d$ of $p$ into $O(
(d\log(n)/\varepsilon^2)^d)$ rectangles, and compare it with $\bar p$, the
flattening over these rectangles. 

\begin{lemma}
\label{lem:mon-str}
Let $d\ge1$. There is an oblivious decomposition of $[n]^d$ into
$O((d\log(n)/\varepsilon^2)^d)$ rectangles such that for any monotone
distribution $p$ over $[n]^d$, its flattening $\bar p$ over these
rectangles satisfy $\chi^2(p,\bar p) \leq \varepsilon^2$.  
\end{lemma}

This effectively reduces the support size to logarithmic in $n$.
At this point, we can apply the Laplace estimator (along the lines
of~\cite{KamathOPS15}) and learn a $q$ such
that if $p$ was monotone, then $q$ will be $O(\varepsilon^2)$-close in
$\chi^2$-distance. 

\begin{lemma}
\label{lem:fin-learn-mon}
Let $d\ge1$, and $p$ be a monotone distribution over $[n]^d$. 
There is an algorithm which outputs a distribution $q$ such that $\expectation{\chi^2(p,q)}\le \frac{\varepsilon^2}{500}$.
The time and sample complexity are both $O((d\log (n)/\varepsilon^2)^d/\varepsilon^2)$.
\end{lemma}

The final step before we apply our $\chi^2$-tester is to compute the
distance between $q$ and $\mathcal{M}_n^d$. 
This subroutine is similar to the one introduced by~\cite{BatuKR04}. 
The key idea is to write a linear program, which searches for any distribution $f$ which is close to $q$ in total variation distance.
We note that the desired properties of $f$ (i.e., monotonicity, normalization, and $\varepsilon$-closeness to $q$) are easy to enforce as linear constraints.
If we find that such an $f$ exists, we will apply our $\chi^2$-test to $q$.
If not, we output $\textsc{Reject}\xspace$, as this is sufficient evidence to conclude that $p \not \in \mathcal{M}_n^d$.
Note that the linear program operates over the oblivious decomposition
used in our structural result, so the complexity is polynomial in
$(d\log(n)/\varepsilon)^d$, rather than the naive $n^d$. 

At this point, we have precisely the guarantees needed to apply Theorem~\ref{thm:chisq-test}, directly implying Theorem~\ref{thm:monotone-final}.
Proof of the lemmas in this section are provided in Section~\ref{sec:monotone-appendix}.

\section{Testing Log-Concavity}
\label{sec:LCD-main}
In this section we describe our results for testing log-concavity of
distributions. Our main result is as follows:
\begin{theorem}
\label{thm:lcd-main}
There exists an algorithm for testing log-concavity over $[n]$ with sample complexity
$$O\left(\frac{\sqrt{n}}{\varepsilon^2}+\frac{1}{\varepsilon^5}\right)$$
and time complexity $\poly(n,1/\varepsilon)$.
\end{theorem}

In particular, this implies the following optimal tester for this class:
\begin{corollary}
Suppose $\varepsilon > 1/n^{1/5}$.
Then there exists an algorithm for testing log-concavity over $[n]$ with sample complexity $O\left(\sqrt{n}/\varepsilon^2\right)$.
\end{corollary}

Our algorithm will fit into the structure of our general framework.
We first perform a very particular type of learning algorithm, whose
guarantees are summarized in the following lemma: 
\begin{lemma}
\label{lem:lcd-learn}
Given $\varepsilon > 0$ and sample access to a distribution $p$, there exists an algorithm with the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{LCD}_n$, the algorithm outputs a distribution $q \in \mathcal{LCD}_n$ and an $O(\varepsilon)$-effective support $S$ of $p$ such that $\chi^2(p_S,q_S) \leq \frac{\varepsilon^2}{500}$ with probability at least $5/6$;
\item If $d_{\mathrm {TV}}(p,\mathcal{LCD}_n) \geq \varepsilon$, the algorithm either outputs a
  distribution $q \in \mathcal{LCD}_n$ or \textsc{Reject}\xspace. 
\end{itemize}
The sample complexity is $O(1/\varepsilon^5)$ and the time complexity is $\poly(n,1/\varepsilon)$.
\end{lemma}

We note that as a corollary, one immediately obtains a $O(1/\varepsilon^5)$ proper learning algorithm for log-concave distributions.
The result is immediate from the first item of Lemma~\ref{lem:lcd-learn} and Proposition~\ref{prop:distance-relations}. 
We can actually do a bit better -- in the proof of Lemma~\ref{lem:lcd-learn}, we partition $[n]$ into intervals of probability mass $\Theta(\varepsilon^{3/2})$.
If one instead partitions into intervals of probability mass $\Theta(\varepsilon/\log(1/\varepsilon))$ and works directly with total variation distance instead of $\chi^2$ distance, one can show that $\tilde O(1/\varepsilon^4)$ samples suffice.
\begin{corollary}
\label{cor:learning LCD}
Given $\varepsilon > 0$ and sample access to a distribution $p \in \mathcal{LCD}_n$, there exists an algorithm which outputs a distribution $q \in \mathcal{LCD}_n$ such that $d_{\mathrm {TV}}(p,q) \leq \varepsilon$.
The sample complexity is $\tilde O(1/\varepsilon^4)$ and the time complexity is $\poly(n,1/\varepsilon)$.
\end{corollary}

Then, given the guarantees of Lemma \ref{lem:lcd-learn},
Theorem~\ref{thm:lcd-main} follows from
Theorem~\ref{thm:chisq-test}\footnote{To be more precise, we require
  the modification of Theorem~\ref{thm:lcd-main} which is described in
  Section~\ref{sec:testing}, in order to handle the case where the
  $\chi^2$-distance guarantees only hold for a known effective
  support.}. The details of these results are presented in Section~\ref{sec:LCD}.


\section{Testing for Monotone Hazard Rate}
\label{sec:MHR-main}
In this section, we obtain our main result for testing for monotone hazard rate:
\begin{theorem}
\label{thm:mhr-main}
There exists an algorithm for testing monotone hazard rate over $[n]$ with sample complexity
$$O\left(\frac{\sqrt{n}}{\varepsilon^2}+\frac{\log(n/\varepsilon)}{\varepsilon^4}\right)$$
and time complexity $\poly(n,1/\varepsilon)$.
\end{theorem}

This implies the following optimal tester for the class:
\begin{corollary}
Suppose $\varepsilon > \sqrt{\log(n/\varepsilon)}/n^{1/4}$.
Then there exists an algorithm for testing monotone hazard rate over $[n]$ with sample complexity $O\left(\sqrt{n}/\varepsilon^2\right)$.
\end{corollary}

We obey the same framework as before, first applying a $\chi^2$-learner with the following guarantees:
\begin{lemma}
\label{lem:mhr-learn}
Given $\varepsilon > 0$ and sample access to a distribution $p$, there exists an algorithm with the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{MHR}_n$, the algorithm outputs a distribution $q \in \mathcal{MHR}_n$ and an $O(\varepsilon)$-effective support $S$ of $p$ such that $\chi^2(p_S,q_S) \leq \frac{\varepsilon^2}{500}$ with probability at least $5/6$;
\item If $d_{\mathrm {TV}}(p,\mathcal{MHR}_n) \geq \varepsilon$, the algorithm either outputs a distribution $q \in \mathcal{MHR}_n$ and a set $S \subseteq [n]$ or \textsc{Reject}\xspace.
\end{itemize}
The sample complexity is $O(\log(n/\varepsilon)/\varepsilon^4)$ and the time complexity is $\poly(n,1/\varepsilon)$.
\end{lemma}

As with log-concave distributions, this implies the following proper learning result:
\begin{corollary}
\label{cor:learning MHR}
Given $\varepsilon > 0$ and sample access to a distribution $p \in \mathcal{MHR}_n$, there exists an algorithm which outputs a distribution $q \in \mathcal{MHR}_n$ such that $d_{\mathrm {TV}}(p,q) \leq \varepsilon$.
The sample complexity is $O(\log(n/\varepsilon)/\varepsilon^4)$ and the time complexity is $\poly(n,1/\varepsilon)$.
\end{corollary}

Again, combining the learning guarantees of Lemma \ref{lem:mhr-learn}
with the appropriate variant of Theorem~\ref{thm:chisq-test}, we
obtain Theorem~\ref{thm:mhr-main}. The details of the argument and
proofs are presented in Section~\ref{sec:MHR}.



\section{Overview}
Our algorithm for testing a distribution $p$ can be decomposed into three steps.

\paragraph{Near-proper learning in $\chi^2$-distance.}
Our first step requires a learning algorithm with very specific guarantees.
In proper learning, we are given sample access to a distribution $p \in \mathcal{C}$, where $\mathcal{C}$ is some class of distributions, and we wish to output $q \in \mathcal{C}$ such that $p$ and $q$ are close in total variation distance.
In our setting, given sample access to $p \in \mathcal{C}$, we wish to output $q$ such that $q$ is \emph{close} to $\mathcal{C}$ in total variation distance, and $p$ and $q$ are close in $\chi^2$-distance on an effective support\footnote{We also require the algorithm to output a description of an effective support for which this property holds.
This requirement can be slightly relaxed, as we show in our results for testing unimodality.} of $p$.
From an information theoretic standpoint, this problem is harder than proper learning, since $\chi^2$-distance is more restrictive than total variation distance.
Nonetheless, this problem can be shown to have comparable sample complexity to proper learning for the structured classes we consider in this paper.


\paragraph{Computation of distance to class.}
The next step is to see if the hypothesis $q$ is close to the class $\mathcal{C}$ or not.
Since we have an explicit description of $q$, this step requires no further samples from $p$, i.e. it is purely computational.
If we find that $q$ is far from the class $\mathcal{C}$, then it must be that $p \not \in \mathcal{C}$, as otherwise the guarantees from the previous step would imply that $q$ is close to $\mathcal{C}$. Thus, if it is not, we can terminate the algorithm at this point.

\paragraph{$\chi^2$-testing.}
At this point, the previous two steps guarantee that our distribution $q$ is such that:
\begin{itemize}
\item If $p \in \mathcal{C}$, then $p$ and $q$ are close in $\chi^2$ distance on a (known) effective support of $p$;
\item If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, then $p$ and $q$ are far in total variation distance.
\end{itemize}
We can distinguish between these two cases using $O(\sqrt{n}/\varepsilon^2)$ samples with a simple statistical $\chi^2$-test, that we describe in Section~\ref{sec:testing}. 


\smallskip Using the above three-step approach, our tester, as described in the next section, can directly test monotonicity, log-concavity, and monotone hazard rate. With an extra trick, using Kolmogorov's max inequality, it can also test unimodality.




\section{Probability distances}
We use the following probability distances in our paper.
\begin{definition}
The \emph{total variation distance} between distributions
$p$ and $q$ is defined as 
$$d_{\mathrm {TV}}(p,q) {\hfill\blksquare}\medskip \sup_{A} |p(A) - q(A)| = \frac12\|p - q\|_1.$$
\end{definition}
For a subset of the domain, the total variation distance is defined as
half of the $\ell_1$ distance restricted to the subset.  

\begin{definition} \label{def:chisq}
The \emph{$\chi^2$-distance} between $p$ and $q$ over $[n]$ is defined by
$$  \chi^2(p,q) {\hfill\blksquare}\medskip \sum_{i \in [n]}\frac{(p_i-q_i)^2}{q_i} = \left[\sum_{i \in [n]}\frac{p_i^2}{q_i}\right]-1.$$
\end{definition}

\begin{definition}
  The \emph{Kolmogorov distance} between two probability measures
  $p$ and $q$ over an ordered set ($e.g.$, $\Rho$) with cumulative
  density functions (CDF) $F_p$ and $F_q$ is defined as
  $$d_{\mathrm K}(p,q) {\hfill\blksquare}\medskip \sup_{x \in \mathbb{R}} |F_p(x) - F_q(x)|.$$
\end{definition}


\section{Preliminaries}



We use the following probability distances in our paper.
\begin{definition}
The \emph{total variation distance} between distributions
$p$ and $q$ is defined as 
$$d_{\mathrm {TV}}(p,q) {\hfill\blksquare}\medskip \sup_{A} |p(A) - q(A)| = \frac12\|p - q\|_1.$$
\end{definition}
For a subset of the domain, the total variation distance is defined as
half of the $\ell_1$ distance restricted to the subset.  

\begin{definition} \label{def:chisq}
The \emph{$\chi^2$-distance} between $p$ and $q$ over $[n]$ is defined by
$$  \chi^2(p,q) {\hfill\blksquare}\medskip \sum_{i \in [n]}\frac{(p_i-q_i)^2}{q_i} = \left[\sum_{i \in [n]}\frac{p_i^2}{q_i}\right]-1.$$
\end{definition}

\begin{definition}
  The \emph{Kolmogorov distance} between two probability measures
  $p$ and $q$ over an ordered set ($e.g.$, $\Rho$) with cumulative
  density functions (CDF) $F_p$ and $F_q$ is defined as
  $$d_{\mathrm K}(p,q) {\hfill\blksquare}\medskip \sup_{x \in \mathbb{R}} |F_p(x) - F_q(x)|.$$
\end{definition}

Our paper is primarily concerned with testing against classes of distributions, defined formally as follows:
\begin{definition}
Given $\varepsilon \in (0,1]$ and sample access to a distribution $p$, an algorithm is said to \emph{test} a class $\mathcal{C}$ if it has the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{C}$, the algorithm outputs \textsc{Accept}\xspace with probability at least $2/3$;
\item If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, the algorithm outputs \textsc{Reject}\xspace with probability at least $2/3$.
\end{itemize}
\end{definition}

The Dvoretzky-Kiefer-Wolfowitz (DKW) inequality gives a generic algorithm for learning any distribution with respect to the Kolmogorov distance~\cite{DvoretzkyKW56}. 
\begin{lemma}{(See~\cite{DvoretzkyKW56},\cite{Massart90})}
 \label{lem:dkw}
  Suppose we have $n$ \textit{i.i.d.} samples $X_1, \dots X_n$ from a distribution with CDF $F$.
  Let $F_n(x) {\hfill\blksquare}\medskip \frac{1}{n}\sum_{i=1}^n \mathbf{1}_{\{X_i \leq x\}}$ be the empirical CDF.
  Then $\Pr[d_{\mathrm K}(F,F_n) \geq \varepsilon] \leq 2e^{-2n\varepsilon^2}$.
  In particular, if $n = \Omega((1/\varepsilon^2) \cdot \log(1/\delta))$, then $\Pr[d_{\mathrm K}(F,F_n) \geq \varepsilon] \leq \delta$.
\end{lemma}

We note the following useful relationships between these distances~\cite{GibbsS02}:
\begin{proposition}
\label{prop:distance-relations}
  $d_{\mathrm K}(p,q)^2 \leq d_{\mathrm {TV}}(p,q)^2 \leq \frac14 \chi^2(p,q)$.
\end{proposition}

In this paper, we will consider the following classes of distributions:
\begin{itemize}
\item Monotone distributions over $[n]^d$ (denoted by $\mathcal{M}_n^d$), for which $i \lesssim j$ implies $f_i \geq f_j$\footnote{This definition describes monotone non-increasing distributions. By symmetry, identical results hold for monotone non-decreasing distributions.};
\item Unimodal distributions over $[n]$ (denoted by $\mathcal{U}_n$), for which there exists an $i^*$ such that $f_i$ is non-decreasing for $i \leq i^*$ and non-increasing for $i \geq i^*$;
\item Log-concave distributions over $[n]$ (denoted by $\mathcal{LCD}_n$), the
  sub-class of unimodal distributions for which $f_{i-1}f_{i+1} \leq f_i^2$; 
\item Monotone hazard rate (MHR) distributions over $[n]$ (denoted by $\mathcal{MHR}_n$), for which $i < j$ implies $\frac{f_i}{1 - F_i} \leq \frac{f_j}{1 - F_j}$.
\end{itemize}



\begin{definition}
An \emph{$\eta$-effective support} of a distribution $p$ is any set $S$ such that $p(S) \geq 1 - \eta$. 
\end{definition}

The \emph{flattening} of a function $f$ over a subset $S$ is the function $\bar{f}$ such that
$\bar{f}_i= p(S)/|S|$. 

\begin{definition}
\label{def:flattening}
Let $p$ be a distribution, and support $I_1, \ldots$ is a partition of
the domain. The flattening of $p$ with respect to $I_1,
\ldots$ is the distribution $\bar p$ which is the flattening of $p$
over the intervals $I_1, \ldots$. 
\end{definition}







\paragraph{Poisson Sampling}
Throughout this paper, we  use the standard Poissonization
approach.
Instead of drawing exactly $m$ samples from a distribution $p$, we first draw $m' \sim \rm Poisson(m)$, and then draw $m'$ samples from $p$.
As a result, the number of times different elements in the support of $p$ occur in the sample become independent, giving much simpler analyses.
In particular, the number of times we will observe domain element $i$ will be distributed as $\rm Poisson(m\dP_i)$, independently for each $i$.
Since $\rm Poisson(m)$ is tightly concentrated around $m$, this additional flexibility comes only at a sub-constant cost in the sample complexity
with an inversely exponential in $m$, additive increase in the error probability. 

\section{Introduction} \label{sec:introduction}

The quintessential scientific question is whether an unknown object
has some property, i.e. whether a model from a specific class fits  
the object's observed behavior. If the unknown object is
a probability distribution, $p$, to which we have sample access,
we are typically called to distinguish from samples whether $p$ belongs to some class 
$\mathcal{C}$ or whether it is sufficiently far from it.

This question has received tremendous attention in Statistics, where
test statistics for important properties such as the ones we
consider here have been proposed. Nevertheless, the 
emphasis has been on asymptotic
analysis, characterizing rates of convergence of test statistics under null
hypotheses, as the number of samples tends to infinity. In contrast, 
we want to study the small sample regime, studying the following 
problem:

\vspace{2ex}
\begin{center}
\smallskip \framebox{
\begin{minipage}{13.5cm} $\Pi(\mathcal{C},\epsilon)$: Given a family of distributions $\mathcal{C}$, some
$\epsilon>0$, and sample access to an unknown distribution $p$ over some
discrete support, how many samples are required to distinguish
between $p \in \mathcal{C}$ versus $d_{\mathrm {TV}}(p,\mathcal{C})>\epsilon$?
\end{minipage}
}
\end{center}

\vspace{2ex}
The problem has been studied  intensely in the literature on Property
Testing and Sublinear Algorithms, where the emphasis has been on
characterizing the optimal tradeoff between $p$'s support size and
the accuracy $\epsilon$ in the number of samples. Several results have
been obtained, roughly clustering into three groups, where (i) $\mathcal{C}$
is the class of monotone distributions over $[n]$, or more generally a
poset---see e.g.~\cite{BatuKR04,Bhattacharyya11}; (ii) $\mathcal{C}$ is the
class of independent, or $k$-wise independent distributions over a
hypergrid---see, e.g., \cite{batu2001testing,alon2007testing}; and
(iii) $\mathcal{C}$ contains a single-distribution $q$, and the problem
becomes that of testing whether $p$ equals $q$ or is far from
it---see, e.g.~\cite{batu2001testing,Paninski08, valiant2014automatic}. 

With respect to (iii), \cite{valiant2014automatic} characterize exactly the number of samples required to test identity to each distribution $q$, providing a single tester matching this bound simultaneously for all $q$. Nevertheless, this tester and its precursors are not applicable to the composite identity testing problem that we consider. If our class $\mathcal{C}$ were finite, we could test against each element in the class, albeit this would not necessarily be sample optimal. If our class $\mathcal{C}$ is a continuum though, we  need tolerant identity testers, which tend to be more expensive in terms of number of samples, and result in substantially suboptimal testers for the classes we consider. Or we could use approaches related to generalized likelihood ratio test, but their behavior is not well-understood in our regime, and optimizing likelihood over our classes becomes computationally intense.

\jnote{Our problem falls in the general framework of property
testing~\cite{Goldreich98, Fischer01, Rubinfeld06, Ron08,canonne2015survey}, where the
objective is to decide if an object has a certain property, or is
\emph{far} from having the property. The objects in our case are
distributions, and properties are belongingness to \emph{simple}
classes of distributions $\mathcal{C}$. The objective of property testing is
to solve such problems with as few samples as possible, and as fast
(computationally) as possible. This is different from the extensively
studied (See \emph{e.g.,}~\cite{Fisher25,lehmann2006testing}) classic
problem of (composite) hypothesis testing in statistics where the
number of samples are taken to infinity and error exponents and
consistency are studied. }

In this paper, we obtain sample-optimal and computationally efficient testers for $\Pi(\mathcal{C},\epsilon)$ for the most basic shape restrictions to a distribution. Our contributions are the following:
\begin{enumerate}
\item
\jnote{For a known distribution $q$ over $[n]$, and samples from an unknown
distribution $p$, we show that to distinguishing the cases $(a)$ whether the $\chi^2$ distance
between $p$ and $q$ is at most $\varepsilon^2/2$,  versus $(b)$ the $\ell_1$ distance
between $p$ and $q$ is at least $\varepsilon$ requires $O(\sqrt
n/\varepsilon^2)$ samples. As a corollary, we provide simpler arguments to
show that identity testing requires $\Theta(\sqrt n/\varepsilon^2)$ samples. }
\item For the class $\mathcal{C}=\mathcal{M}_n^d$ of monotone distributions over
  $[n]^d$ we require an optimal $\Theta\left({n^{d/2} \over
      \varepsilon^2}\right)$ number of samples, where prior work requires
  $\Omega\left({\sqrt{n} \log n \over \varepsilon^4}\right)$ samples for $d=1$
  and $\tilde{\Omega}\left(n^{d-{1\over 2}} {\rm poly}({1 \over
      \varepsilon})\right)$ for $d>1$~\cite{BatuKR04,Bhattacharyya11}. Our
  results improve the exponent of $n$ with respect to $d$, shave all
  logarithmic factors in $n$, and improve the exponent of $\varepsilon$ by at
  least a factor of $2$. 
\begin{enumerate}
\item A useful building block and interesting byproduct of our
  analysis is extending Birg\'e's oblivious decomposition for
  single-dimensional monotone distributions~\cite{Birge87} to monotone
  distributions in $d\ge1$, and to the stronger notion of $\chi^2$
  distance. See Section~\ref{sec:hypergrid}. 

\item Moreover, we show that $O(\log(n)^{d})$ samples suffice to learn a monotone
distribution over $[n]^d$ in $\chi^2$ distance. See
Lemma~\ref{lem:fin-learn-mon} for the precise statement.
\end{enumerate}
\item For the classes $\mathcal{C}=\mathcal{LCD}_n$, $\mathcal{C}=\mathcal{MHR}_n$ and $\mathcal{C}=\mathcal{U}_n$ of log-concave, monotone-hazard-rate and unimodal distributions over $[n]$, we require an optimal $\Theta\left({\sqrt{n} \over \varepsilon^2}\right)$ number of samples. Our testers for $\mathcal{LCD}_n$ and $\mathcal{C}=\mathcal{MHR}_n$ are to our knowledge the first for these classes for the low sample regime we are studying---see~\cite{hall2005testing} and its references for Statistics literature on the asymptotic regime. Our tester for $\mathcal{U}_n$ improves the dependence of the sample complexity on $\varepsilon$ by at least a factor of $2$ in the exponent, and shaves all logarithmic factors in $n$, compared to testers based on testing monotonicity.
\begin{enumerate}
\item A useful building block and important byproduct of our analysis are the first computationally efficient algorithms for properly learning log-concave and monotone-hazard-rate distributions, to within $\epsilon$ in total variation distance, from ${\rm poly}(1/\epsilon)$ samples, independent of the domain size $n$. See Corollaries~\ref{cor:learning LCD} and~\ref{cor:learning MHR}. Again, these are the first computationally efficient algorithms to our knowledge in the low sample regime. \cite{ChanDSS13b} provide algorithms for density estimation, which are non-proper, i.e. will approximate an unknown distribution from these classes with a distribution that does not belong to these classes. On the other hand, the Statistics literature focuses on maximum-likelihood estimation in the asymptotic regime---see e.g. \cite{cule2010theoretical} and its references.
\end{enumerate}

\item For all the above classes we obtain matching lower bounds, showing that the sample complexity of our testers is optimal with respect to $n$, $\varepsilon$ and when applicable $d$. See Section~\ref{sec:lower-bounds}. Our lower bounds are based on extending Paninski's lower bound~\cite{Paninski08}.
\end{enumerate}


In the heart of our tester lies a novel use of the $\chi^2$ statistic. Naturally, the $\chi^2$ and its related $\ell_2$ statistic have  been used in several of the afore-cited results. We propose a new use of the $\chi^2$ statistic enabling  our optimal sample complexity. The essence of our approach is to first draw a small number of samples (independent of $n$ for log-concave and monotone-hazard-rate distributions and only logarithmic in $n$ for monotone and unimodal distributions) to approximate the unknown distribution $p$ in $\chi^2$ distance. If $p \in \mathcal{C}$, our learner is required to output a distribution $q$ that is $O(\epsilon)$-close to $\mathcal{C}$ in total variation  and $O(\epsilon^2)$-close to $p$ in $\chi^2$ distance. Then a small analysis reduces our testing problem to telling apart the following cases:
\begin{itemize}
\item $p$ and $q$ are $O(\epsilon^2)$-close in $\chi^2$ distance; this case corresponds to $p \in \mathcal{C}$.
\item $p$ and $q$ are $\Omega(\epsilon)$-far in total variation distance; this case corresponds to $d_{\mathrm {TV}}(p,\mathcal{C})>\varepsilon$.
\end{itemize}
In Section~\ref{sec:testing}, we show that a version of the $\chi^2$
statistic, appropriately excluding certain elements of the support,
is sufficiently well-concentrated to distinguish between the above
cases. Moreover, the sample complexity of our algorithm is optimal for
most classes. \explainindetail{Should we put the chi-sq L1 as a
  separate result, and say why this is a robust identity testing.}
Our tester, combined with the afore-mentioned extension of
Birg\'e's decomposition theorem is used in Section~\ref{sec:monotone}
to test monotone distributions. See Theorem~\ref{thm:monotone-final}
and Corollary~\ref{cor:high-d}. 

Naturally, there are several bells and whistles that we need to add to the above skeleton to accommodate all classes of distributions that we are considering. For log-concave and monotone-hazard distributions, we are unable to obtain a cheap (in terms of samples) learner that $\chi^2$-approximates the unknown distribution $p$ throughout its support. Still, we can identify a subset of the support where the $\chi^2$-approximation is tight and which captures almost all the probability mass of $p$. We extend our tester to accommodate excluding subsets of the support from the $\chi^2$-approximation. See Theorems~\ref{thm:lcd-main} and~\ref{thm:mhr-main} in Sections~\ref{sec:LCD} and \ref{sec:MHR}, which are in the appendix due to lack of space. Some discussion is provided in Section~\ref{sec:blurb}.

For unimodal distributions, we are even unable to identify a large enough subset of the support where the $\chi^2$ approximation is guaranteed to be tight. But we can show that there exists a light enough piece of the support (in terms of probability mass under $p$) that we can exclude to make the $\chi^2$ approximation tight. Given that we only use Chebyshev's inequality to prove the concentration of the test statistic, it would seem that our lack of knowledge of the piece to exclude would involve a union bound and a corresponding increase in the required number of samples. We avoid this through a nice application of Kolmogorov's max inequality in our setting. See Theorem~\ref{thm:unimodality} of Section~\ref{sec:unimodal}, which is in the appendix due to lack of space. Some discussion is provided in Section~\ref{sec:blurb}.

\paragraph{Related Work.} We cannot do justice to the role of shape
restrictions of probability distributions in probabilistic modeling
and testing. It suffices to say that the classes of distributions that
we study are fundamental, motivating extensive literature on their
learning and testing~\cite{BBBB:72}. In the recent times, there has
been work on shape restricted statistics, pioneered by Jon Wellner,
and others~\cite{JW:09,BW10sn, BJR11,SumardW14}.
Due to the sheer volume of literature in statistics in this field, we
will restrict ourselves to those already referenced. 

As we have mentioned, Statistics has focused on the asymptotic
regime as the number of samples tends to infinity. Instead we are
considering the low sample regime and  are more stringent about the
behavior of our testers, requiring $2$-sided guarantees. We want to
accept if the unknown distribution is in our class of interest, and
also reject if it is far from the class. For this problem, as
discussed above, there are few results when $\mathcal{C}$ is a whole class of
distributions. Closer related to our paper is the line of
papers~\cite{BatuKR04, ACS10, Bhattacharyya11} for monotonicity testing, albeit these 
papers have sub-optimal sample complexity as discussed above. More
recently, Acharya and Daskalakis provide optimal testers for the
family of Poisson Binomial Distributions~\cite{AcharyaD15}. Finally,
contemporaneous work of Canonne et al~\cite{CanonneDGR15} provide
testers for the single-dimensional families of distributions
considered here, albeit their sample complexity is suboptimal in both
$n$ and $\epsilon$. \insertref{Put a bunch of
  statistics citations.} 

 
 




\input{preliminaries}
\input{overview}
\input{testing}
\input{monotone}
\input{unimodal-main}
\input{otherclasses}
\input{lower-bounds}
\input{experiments}


\bibliographystyle{IEEEtran}

\section{Moments of the Chi-Squared Statistic}
\label{sec:chisq-moments}
We analyze the mean and variance of the statistic
$$ Z = \sum_{i \in \mathcal{A}} \frac{(X_i - mq_i)^2 - X_i}{mq_i},$$
where each $X_i$ is independently distributed according to $\rm Poisson(\text{$m p_i$})$.

We start with the mean:
\begin{align*}
\expectation{Z} &= \sum_{i \in \mathcal{A}} \expectation{\frac{(X_i - mq_i)^2 - X_i}{mq_i}} \nonumber \\
     &= \sum_{i \in \mathcal{A}} \frac{\expectation{X_i^2} -
       2mq_i\expectation{X_i} + m^2q_i^2 - \expectation{X_i}}{mq_i} \nonumber \\ 
     &= \sum_{i \in \mathcal{A}} \frac{m^2p_i^2 + m p_i - 2m^2q_ip_i + m^2q_i^2 - m p_i}{mq_i} \nonumber \\
     &= m \sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^2}{q_i} \nonumber\\
     &= m \cdot \chi^2(p_\mathcal{A},q_\mathcal{A}) \nonumber
\end{align*}

Next, we analyze the variance.
Let $\lambda_i = {\bf E}{X_i}=m p_i$ and $\lambda_i' = m q_i$.
\begin{align}
\Var{Z} &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}\Var{(X_i - \lambda_i)^2 + 2(X_i - \lambda_i)(\lambda_i - \lambda_i') - (X_i - \lambda_i)} \nonumber \\
       &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}\Var{(X_i - \lambda_i)^2 + (X_i - \lambda_i)(2\lambda_i -2\lambda_i' - 1) } \nonumber\\
       &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}{\bf E}{(X_i - \lambda_i)^4 + 2(X_i - \lambda_i)^3(2\lambda_i -2\lambda_i' - 1) + (X_i - \lambda_i)^2(2\lambda_i -2\lambda_i' - 1)^2 - \lambda_i^2} \nonumber\\
       &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}[3\lambda_i^2 + \lambda_i + 2\lambda_i(2\lambda_i - 2\lambda_i' - 1) + \lambda_i(2\lambda_i - 2\lambda_i' - 1)^2 - \lambda_i^2] \nonumber\\
       &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}[2\lambda_i^2 + \lambda_i + 4\lambda_i(\lambda_i - \lambda_i') - 2\lambda_i + \lambda_i(4(\lambda_i - \lambda_i')^2 -4(\lambda_i - \lambda_i') + 1)] \nonumber\\
       &= \sum_{i \in \mathcal{A}}\frac{1}{\lambda_i'^2}[2\lambda_i^2  + 4\lambda_i(\lambda_i - \lambda_i')^2 ] \nonumber\\
       &= \sum_{i \in \mathcal{A}}\left[2 \frac{p_i^2}{q_i^2}+4m\cdot\frac{p_i\cdot(p_i-q_i)^2}{q_i^2}\right] 
\end{align}
The third equality is by noting the random variable has expectation $\lambda_i $ and the fourth equality substitutes the values of centralized moments of the Poisson distribution.

\section{Analysis of our $\chi^2$-Test Statistic}
\label{sec:chisq-analysis}
We first prove the key lemmas in the analysis of our $\chi^2$-test.

\begin{prevproof}{Lemma}{lem:means}
The former case is straightforward from (\ref{eqn:mean}) and \ref{prp:in-chisq} of $q$.

We turn to the latter case. 
Recall that $\mathcal{A}= \{i:q_i \geq \varepsilon/50n\}$, and thus $q(\bar \mathcal{A}) \leq \varepsilon/50$.
We first show that $d_{\mathrm {TV}}(p_{\mathcal{A}}, q_{\mathcal{A}})\geq \frac{6\varepsilon}{25}$, where $p_{\mathcal{A}}, q_{\mathcal{A}}$ are
defined as above and in our slight abuse of notation we use $d_{\mathrm {TV}}(p_{\mathcal{A}}, q_{\mathcal{A}})$ for non-probability vectors to denote $\frac12\|p_{\mathcal{A}} - q_{\mathcal{A}}\|_1.$

Partitioning the support into $\mathcal{A}$ and $\compl{\mathcal{A}}$, we have
\begin{align}
d_{\mathrm {TV}}(p,q)=d_{\mathrm {TV}}(p_{\mathcal{A}}, q_{\mathcal{A}})+d_{\mathrm {TV}}(p_{\compl{\mathcal{A}}}, q_{\compl{\mathcal{A}}}).\label{eqn:tv-decomp}
\end{align}

We consider the following cases separately:
\begin{itemize}
\item 
{\bf $p(\compl{\mathcal{A}})\le \varepsilon/2$:} In this case, 
\begin{align}
  d_{\mathrm {TV}}(p_{\compl{\mathcal{A}}}, q_{\compl{\mathcal{A}}}) = \frac12 \sum_{i \in \compl{\mathcal{A}}} |p_i - q_i| \leq 
  \frac12 (p(\compl{\mathcal{A}})+q(\compl{\mathcal{A}})) \le \frac{1}{2}\left(\frac{\varepsilon}{2} + \frac{\varepsilon}{50}\right) = \frac{13\varepsilon}{50}.\nonumber
\end{align}
Plugging this in~\eqref{eqn:tv-decomp}, and using the fact that $d_{\mathrm {TV}}(p,q) \geq \varepsilon$
shows that $d_{\mathrm {TV}}(p_{\mathcal{A}}, q_{\mathcal{A}}) \geq \frac{6\varepsilon}{25}$. 
\item 
{\bf $p(\compl{\mathcal{A}})> \varepsilon/2$:} In this case, by the reverse triangle inequality,  
\begin{align}
  d_{\mathrm {TV}}(p_{\mathcal{A}}, q_{\mathcal{A}}) \geq \frac12 (q(\mathcal{A})-p(\mathcal{A})) \geq \frac12 ((1 - \varepsilon/50) - (1 - \varepsilon/2)) = \frac{6\varepsilon}{25} .\nonumber
\end{align}
\end{itemize}

By the Cauchy-Schwarz inequality,
\begin{align}
  \chi^2(p_{\mathcal{A}}, q_{\mathcal{A}}) &\ge 4\frac{d_{\mathrm {TV}}(p_{\mathcal{A}},q_{\mathcal{A}})^2}{q(\mathcal{A})}\nonumber\\
                           &\geq \frac{\varepsilon^2}{5}. \nonumber
\end{align}
We conclude by recalling~\eqref{eqn:mean}.

\end{prevproof}

\begin{prevproof}{Lemma}{lem:vars}
We bound the terms of (\ref{eqn:variance}) separately, starting with the first.

\begin{align}
2\sum_{i \in \mathcal{A}} \frac{p_i^2}{q_i^2} &= 2\sum_{i \in \mathcal{A}} \left(\frac{(p_i - q_i)^2}{q_i^2} + \frac{2p_iq_i - q_i^2}{q_i^2}\right) \nonumber \\
                                             &= 2\sum_{i \in \mathcal{A}} \left(\frac{(p_i - q_i)^2}{q_i^2} + \frac{2q_i(p_i - q_i) + q_i^2}{q_i^2}\right) \nonumber\\
                                             &\leq 2n + 2\sum_{i \in \mathcal{A}} \left(\frac{(p_i - q_i)^2}{q_i^2} + 2\frac{(p_i - q_i)}{q_i}\right) \nonumber\\
                                             &\leq 4n + 4\sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^2}{q_i^2} \nonumber\\
                                             &\leq 4n + \frac{200n}{\varepsilon} \sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^2}{q_i}\nonumber\\
                                             &= 4n + \frac{200n}{\varepsilon}\frac{E[Z]}{m} \nonumber\\
                                             &\leq 4n + \frac{1}{100}\sqrt{n} E[Z]\label{eq:first-var-term-in}
\end{align}
The second inequality is the AM-GM inequality, the third inequality uses that $q_i \geq \frac{\varepsilon}{50n}$ for all $i \in \mathcal{A}$, the last equality uses \eqref{eqn:mean}, and the final inequality substitutes a value $m \geq 20000\frac{\sqrt{n}}{\varepsilon^2}$.

The second term can be similarly bounded:
\begin{align*}
4m \sum_{i \in \mathcal{A}} \frac{p_i(p_i - q_i)^2}{q_i^2} &\leq 4m \left(\sum_{i \in \mathcal{A}} \frac{p_i^2}{q_i^2}\right)^{1/2}\left(\sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^4}{q_i^2}\right)^{1/2} \\
                                                          &\leq 4m \left(4n + \frac{1}{100}\sqrt{n} E[Z] \right)^{1/2}\left(\sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^4}{q_i^2}\right)^{1/2} \\
                                                          &\leq 4m \left(2\sqrt{n} + \frac{1}{10}n^{1/4} E[Z]^{1/2}\right)\left(\sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^2}{q_i}\right) \\
                                                          &= \left(8\sqrt{n} + \frac{2}{5}n^{1/4} E[Z]^{1/2}\right)E[Z] \\
\end{align*}
The first inequality is Cauchy-Schwarz, the second inequality uses (\ref{eq:first-var-term-in}), the third inequality uses the monotonicity of the $\ell_p$ norms, and the equality uses~\eqref{eqn:mean}.

Combining the two terms, we get
$$\Var{Z} \leq 4n + 9\sqrt{n} {\bf E}{Z}  + \frac{2}{5}n^{1/4} {\bf E}{Z}^{3/2}  .$$

We now consider the two cases in the statement of our lemma.
\begin{itemize}
\item
When $p \in \mathcal{C}$, we know from Lemma~\ref{lem:means} that ${\bf E}{Z} \leq \frac{1}{500} m\varepsilon^2$. Combined with a choice of $m \geq 20000 \frac{\sqrt{n}}{\varepsilon^2}$ and the above expression for the variance, this gives:
$$\Var{Z} \leq \frac{4}{20000^2}m^2\varepsilon^4 + \frac{9}{20000 \cdot 500}m^2\varepsilon^4  + \frac{\sqrt{10}}{12500000}m^2\varepsilon^4 \leq \frac{1}{500000}m^2\varepsilon^4.$$

\item When $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, Lemma~\ref{lem:means} and  $m \geq 20000\frac{\sqrt{n}}{\varepsilon^2}$ give:
$${\bf E}{Z} \geq \frac{1}{5}m\varepsilon^2 \geq 4000\sqrt{n}.$$

Combining this with our expression for variance we get:
$$\Var{Z} \leq \frac{4}{4000^2}{\bf E}{Z}^2 + \frac{9}{4000}{\bf E}{Z}^2 + \frac{2}{5\sqrt{4000}}{\bf E}{Z}^2 \leq \frac{1}{100}{\bf E}{Z}^2.$$
\end{itemize}
\end{prevproof}


\section{A Robust $\chi^2$-$\ell_1$ Identity Test} \label{sec:testing}

{
Our main result in the Section is Theorem~\ref{thm:chisq-test}. 
As an immediate corollary, we obtain the following result on testing
whether an unknown distribution is close in $\chi^2$ or far in
$\ell_1$ distance to a known distribution. In
particular, we show the following:
\begin{theorem}
\label{thm:rob-iden}
For a known distribution $q$, there exists an algorithm with sample
complexity
\[
O(\sqrt n/\varepsilon^2)
\]
distinguishes between the cases
\begin{itemize}
\item 
$\chi^2(p,q)<\varepsilon^2/10$\ \ \ \ \emph{versus}
\item
$\|p-q\|>\varepsilon^2$.
\end{itemize}
with probability at least $5/6$.
\end{theorem}
This theorem follows from our main result of this section, stated
next, slightly more generally for classes of distributions. 
}
\begin{theorem}
\label{thm:chisq-test}
Suppose we are given $\varepsilon \in (0,1]$, a class of probability distributions $\mathcal{C}$, sample access to a distribution $p$ over $[n]$, and an explicit description of a distribution $q$ with the following properties:
\begin{enumerate}[label=\textbf{Property \arabic*.},ref=Property \arabic*,align=left]
\item $d_{\mathrm {TV}}(q,\mathcal{C}) \leq \frac{\varepsilon}{2}$.\label{prp:q-tv}
\item If $p \in \mathcal{C}$, then $\chi^2(p,q) \leq \frac{\varepsilon^2}{500}$. \label{prp:in-chisq}
\end{enumerate}
Then there exists an algorithm with the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{C}$, the algorithm outputs \textsc{Accept}\xspace with probability at least $2/3$;
\item If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, the algorithm outputs \textsc{Reject}\xspace with probability at least $2/3$.
\end{itemize}
The time and sample complexity of this algorithm are $O\left(\frac{\sqrt{n}}{\varepsilon^2}\right)$.
\end{theorem}

\begin{remark}
\label{rmk}
As stated in Theorem~\ref{thm:chisq-test}, \ref{prp:in-chisq} requires that $q$ is $O(\varepsilon^2)$-close in $\chi^2$-distance to $p$ over its entire domain.
For the class of monotone distributions, we are able to efficiently obtain such a $q$, which immediately implies sample-optimal learning algorithms for this class.
However, for some classes, we cannot learn a $q$ with such strong guarantees, and we must consider modifications to our base testing algorithm.

For example, for log-concave and monotone hazard rate distributions, we can obtain a distribution $q$ and a set $S$ with the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{C}$, then $\chi^2(p_S,q_S) \leq O(\varepsilon^2)$ and $p(S) \geq 1 - O(\varepsilon)$;
\item If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, then $d_{\mathrm {TV}}(p,q) \geq \varepsilon/2$.
\end{itemize}
In this scenario, the tester will simply pretend the support of $p$ and $q$ is $S$, ignoring any samples and support elements in $[n] \setminus S$.
Analysis of this tester is extremely similar to what we present below.
In particular, we can still show that the statistic $Z$ will be separated in the two cases.
When $p \in \mathcal{C}$, excluding $[n] \setminus S$ will only reduce $Z$.
On the other hand, when $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, since $p(S) \geq 1 - O(\varepsilon)$, $p$ and $q$ must still be far on the remaining support, and we can show that $Z$ is still sufficiently large.
Therefore, a small modification allows us to handle this case with the same sample complexity of $O(\sqrt{n}/\varepsilon^2)$.

A further modification can handle even weaker learning guarantees.
We could handle the previous case because the tester ``knows what we don't know'' -- it can explicitly ignore the support over which we do not have a $\chi^2$-closeness guarantee. 
A more difficult case is when there may be a low measure interval hidden in our effective support, over which $p$ and $q$ have a large $\chi^2$-distance.
While we may have insufficient samples to reliably identify this interval, it may still have a large effect on our statistic.
A naive solution would be to consider a tester which tries all possible ``guesses'' for this ``bad'' interval, but a union bound would incur an extra logarithmic factor in the sample complexity.
We manage to avoid this cost through a careful analysis involving Kolmogorov's max inequality, maintaining the $O(\sqrt{n}/\varepsilon^2)$ sample complexity even in this more difficult case. 

Being more precise, we can handle cases where we can obtain a distribution $q$ and a set of intervals $S = \{I_1,\dots, I_b\}$ with the following guarantees:
\begin{itemize}
\item If $p \in \mathcal{C}$, then $p(S) \geq 1 - O(\varepsilon)$, $p(I_j) = \Theta(p(S)/b)$ for all $j \in [b]$, and there exists a set $T \subseteq [b]$ such that $|T| \geq b - t$ (for $t = O(1)$) and $\chi^2(p_R,q_R) \leq O(\varepsilon^2)$, where $R = \cup_T I_j$;
\item If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, then $d_{\mathrm {TV}}(p,q) \geq \varepsilon/2$.
\end{itemize}
This allows us to additionally test against the class of unimodal distributions.

The tester requires that an effective support is divided into several intervals of roughly equal measure.
It computes our statistic over each of these intervals, and we let our statistic $Z$ be the sum of all but the largest $t$ of these values.
In the case when $p \in \mathcal{C}$, $Z$ will only become smaller by performing this operation.
We use Kolmogorov's maximal inequality to show that $Z$ remains large when $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$. 
More details on this tester are provided in Section~\ref{sec:unimodal-appendix}.
\end{remark}


\begin{algorithm}[h]
 \caption{Chi-squared testing algorithm}\label{alg:testing}
\begin{algorithmic}[1]
\State \textbf{Input:} $\varepsilon$; an explicit distribution $q$; (Poisson) $m$ samples from a distribution $p$, where $N_i$ denotes the number of occurrences of the $i$th domain element.
\State $\mathcal{A} \leftarrow \{i:q_i \geq \varepsilon/50n\}$
\State $Z \leftarrow \sum_{i \in \mathcal{A}} \frac{(N_i - mq_i)^2 - N_i}{mq_i}$
\If {$Z \leq m\varepsilon^2/10$}
\State \Return \textsc{Accept}\xspace
\Else 
\State \Return \textsc{Reject}\xspace
\EndIf 
\end{algorithmic}
\end{algorithm}

\begin{prevproof}{Theorem}{thm:chisq-test}
Theorem \ref{thm:chisq-test} is proven by analyzing Algorithm \ref{alg:testing}.
As shown in Section~\ref{sec:chisq-moments}, $Z$ has the following mean and variance:
\begin{equation}
{\bf E}{Z} = m \cdot \sum_{i \in \mathcal{A}} \frac{(p_i - q_i)^2}{q_i} = m \cdot \chi^2(p_\mathcal{A},q_\mathcal{A})  \label{eqn:mean}
\end{equation}
\begin{equation}
\Var{Z} = \sum_{i \in \mathcal{A}}\left[2 \frac{p_i^2}{q_i^2}+4m\cdot\frac{p_i\cdot(p_i-q_i)^2}{q_i^2}\right]  \label{eqn:variance}
\end{equation}
where by $p_\mathcal{A}$ and $q_\mathcal{A}$ we denote respectively the vectors $p$ and $q$ restricted to the coordinates in $\mathcal{A}$, and we slightly abuse notation when we write $\chi^2(p_\mathcal{A},q_\mathcal{A})$, as these do not then correspond to probability distributions. 

Lemma~\ref{lem:means} demonstrates the separation in the means of the statistic $Z$ in the
two cases of interest, $i.e.,$ $p \in \mathcal{C}$ versus $d_{\mathrm {TV}}(p,\mathcal{C}) \geq
\varepsilon$, and Lemma~\ref{lem:vars} shows the separation in the variances in the
two cases. These two results are proved in
Section~\ref{sec:chisq-analysis}. 

\begin{lemma}
\label{lem:means}
If $p \in \mathcal{C}$, then ${\bf E}{Z} \leq \frac{1}{500}m\varepsilon^2$.
If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, then ${\bf E}{Z} \geq \frac{1}{5}m\varepsilon^2$.
\end{lemma}

\begin{lemma}
\label{lem:vars}
If $p \in \mathcal{C}$, then $\Var{Z} \leq \frac{1}{500000}m^2\varepsilon^4$.
If $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$, then $\Var{Z} \leq \frac{1}{100}E[Z]^2$.
\end{lemma}

Assuming Lemmas~\ref{lem:means} and~\ref{lem:vars},
Theorem~\ref{thm:chisq-test} is now a simple application of
Chebyshev's inequality. 

When $p \in \mathcal{C}$, we have that
$${\bf E}{Z} + \sqrt{3}\Var{Z}^{1/2} \leq \left(\frac{1}{500} + \sqrt{3}\left(\frac{1}{500000}\right)^{1/2}\right)m\varepsilon^2 \leq \frac{1}{200}m\varepsilon^2.$$
Thus, Chebyshev's inequality gives
$$\Pr\left[Z \geq m\varepsilon^2/10\right] \leq \Pr\left[Z \geq m\varepsilon^2/200\right] \leq \Pr\left[Z - {\bf E}{Z} \geq \sqrt{3}\Var{Z}^{1/2}\right] \leq \frac13.$$

The case for $d_{\mathrm {TV}}(p,\mathcal{C}) \geq \varepsilon$ is similar.
Here,
$${\bf E}{Z} - \sqrt{3}\Var{Z}^{1/2} \geq \left(1 - \sqrt{3}\left(\frac{1}{100}\right)^{1/2}\right)E[Z] \geq 3m\varepsilon^2/20.$$
Therefore,
\[\Pr\left[Z \leq m\varepsilon^2/10\right] \leq \Pr\left[Z \leq 3m\varepsilon^2/20\right] \leq \Pr\left[Z - {\bf E}{Z} \leq - \sqrt{3}\Var{Z}^{1/2}\right] \leq \frac13.\qedhere\qedhere\]
\end{prevproof}



\section{$t$-modal Distributions}
\label{sec:t-modal}
For a distribution, $p$ over $[n]$, $i\in[n]$ is said to be a \emph{mode} if $\left(p(i)-p(i-1)\right)\cdot \left(p(i+1)-p(i)\right)$ are of different signs. The family of distributions with at most $t$ modes are called $t$-modal distributions. Monotone distributions are 0-modal, and unimodal and 1-modal distribution classes. These classes of distributions are extremely useful to model mixture distributions, for example it is well known that mixture of $t$ Gaussians is a $t$-modal.

PUT SOME RELATED WORK HERE. 

Recently, in a personal communication~\cite{CanonneDGR15} it was shown that testing $t$-modal distributions is possible with $\tilde{O}(t\sqrt{n}/\varepsilon^4)$ samples. Their time complexity is not mentioned explicitly, however we believe there is an implementation in polynomial time. 

Indeed, note that any $t$-modal distribution can be written as a mixture of a $t+1$-mixture of unimodal distributions. Indeed, our results will hold for this more general class of distributions. Since, log-concave and monotone hazard distributions are unimodal, our results also apply to them. 

\subsection{Structural Results} 

We observed that for monotone distributions, there is an oblivious decomposition of the domain, such that the flattening of the underlying distribution over these intervals has a small $\chi^2$ distance. We now extend this argument to $t$-modal distributions. Consider any distribution with at most $t$ modes. Let $b$ be a number, to be decided later. 

Consider a partition of $[n]$ into $L\le 1/b+1/t$ intervals such that:
\begin{enumerate}
\item
For each element $i$ with probability at least $1/b$, there is an $I_\ell=\{i\}$.
\item
There are at most two intervals with $p(I)\le 1/b$.
\item
Every other interval $I$ satisfies $p(I)\in[\frac1b,\frac2b]$.
\end{enumerate}
Let $\bar{\dP}$ be the flattening of $p$ over the intervals satisfying these properties.

We first prove our decomposition result for unimodal distributions and then describe appropriate extension to mixtures and more modes. 
Consider any decomposition given as above, and we remove the intervals $I_j$ for which $p(I_j)/|I_j|<\frac{\varepsilon}{50n}$. This operation essentially removes the tiny probability elements. 
Our key lemma is as follows. 
\begin{lemma}
Let $C>2$. For a unimodal distribution over $[n]$, there are at most most $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals $I_j$  satisfy $\frac{\dP_j^+}{\dP_j^-}<(1+\varepsilon/C)$. 
\end{lemma}
\begin{proof}
To the contrary, if there are more than $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals, then at least half of them are on one side of the mode, however this implies that the ratio of the largest probability and smallest probability is at least $(1+\varepsilon/C)^j$, and if $j>\frac{2\log(50n/\varepsilon)}{C\varepsilon}$, is at least $50n/\varepsilon$, contradicting that we have removed all such elements. 
\end{proof}

Let $R{\hfill\blksquare}\medskip \frac{4\log(50n/\varepsilon)}{C\varepsilon}$ denote the highest number of intervals with large ratios. 
For $A\in [L]$, let $I_A$ denote the subset of intervals with indices in $A$. The following result is now immediate from the previous lemma. 
\begin{lemma}
For any unimodal distribution $p$ over $[n]$, there is a subset of intervals $I_A$, such that 
\begin{itemize}
\item
$p(I_A)>1-\varepsilon/50-2R/b$, and
\item
$\chi^2_{{I_A}}(p,\bar{\dP})\le \varepsilon^2/C^2$.
\end{itemize}
\end{lemma}

Consider the following test. 
\begin{enumerate}
\item
Take $O(b\log b)$ samples to obtain the intervals satisfying the condition above.  
\item
Use $O(b/\varepsilon^2)$ samples obtain a $q$ that is flat over the
intervals of interest. 
\item
If $q$ is at least $\varepsilon/2$ far from the class, output \textsc{Reject}\xspace.
\item
Remove all singletons and intervals with $q(I)/|I|<\varepsilon/(50n)$. 
\item
\jnote{Remove the intervals with mass at most $1/10b$-- may not be
  necessary}
\item 
Let $\mathcal{I}_{\rm rem}$ be the remaining intervals
\item
Let $Z_j$ be the $\chi^2$ statistic over the $i$th interval, computed
with $O(\sqrt{n}/\varepsilon^2)$ samples. 
\item 
Remove the $R$ largest $Z_j$'s.
\item 
If $\sum_{j}Z_j>m\varepsilon^2/10$, output \textsc{Reject}\xspace.
\item
Output \textsc{Accept}\xspace. 
\end{enumerate}

We now prove the accuracy of the algorithm. 
\begin{claim}
If the output $q$ is at most $\varepsilon/2$-far from $\mathcal{C}$, and $p$ is
at least $\varepsilon-$  far from $\mathcal{C}$, then $p$ is at least $\varepsilon/5$-far
from $q$ over the intervals in $\mathcal{I}_{\rm rem}$. 
\end{claim}
\begin{proof}
Over the singletons, the largest error introduced is at most $\varepsilon/4$
since we take $O(b/\varepsilon^2)$ samples, 
\end{proof}




\section{Real Todos}
\begin{itemize}
\item 
\sout{Everything}
\item 
Test previous item
\item
Put in the lower bound derivation from Paninski/PBD's.
\item
$t$-modal distributions -- Think Think Think
\end{itemize}

\section{Details on testing Unimodality}
\label{sec:unimodal}
\label{sec:unimodal-appendix}

Recall that to circumvent Birg\'e's decomposition, we want to decompose the interval into disjoint intervals such that the probability of each interval is about $O(1/b)$, where $b$ is a parameter, specified later. In particular we consider a decomposition of $[n]$ with the following properties:
\begin{enumerate}
\item
For each element $i$ with probability at least $1/b$, there is an $I_\ell=\{i\}$.
\item
There are at most two intervals with $p(I)\le 1/{2b}$.
\item
Every other interval $I$ satisfies $p(I)\in\left[\frac1{2b},\frac2b\right]$.
\end{enumerate}

Let $I_1, \ldots, I_L$ denote the partition of $[n]$ corresponding to these intervals. Note that $L= O(b)$. 
\begin{claim}
There is an algorithm that takes $O(b\log b)$ samples and outputs $I_1, \ldots, I_L$ satisfying the properties above.
\end{claim}

The first step in our algorithm is to estimate the \emph{total probability} within each of these intervals. 
In particular, 
\begin{lemma}
There is an algorithm that takes $m'=O(b\log b/\varepsilon^2)$ samples from a distribution $p$, and with probability at least 9/10 outputs a distribution $\bar{\dQ}$ that is constant on each $I_L$. Moreover, for any $j$ such that $p(I_j)>1/2b$, 
$\bar{\dQ}(I_j)\in(1\pm\varepsilon)p(I_j)$. 
\end{lemma}
\begin{proof}
Consider any interval $I_j$ with $p(I_j)\ge 1/2b$. The number of
samples $N_{I_j}$ that fall in that interval is distributed
$Binomial(m', p(I_j)$. Then by Chernoff bounds for $m'>12 b\log
b/\varepsilon^2$,  
\begin{align}
\probof{|N_{I_j}-m'p(I_j)|>\varepsilon m'p(I_j) }\le& 2\exp\left(\varepsilon^2m'p(I_j)/2\right)\\
\le & \frac1{b^2},
\end{align}
where the last inequality uses the fact that $p(I_j)\ge 1/2b$. 
\end{proof}

The next step is estimate the distance of $q$ from $\mathcal{U}_n$. This is possible by a simple dynamic program, similar to the one used for monotonicity. 
If the estimated distance is more than $\varepsilon/2$, we output \textsc{Reject}\xspace. 

Our next step is to remove certain intervals. This will be to ensure that when the underlying distribution is unimodal, we are able to estimate the distribution \emph{multiplicatively} over the remaining intervals. 
In particular, we do the following preprocessing step:
\begin{itemize}
\item 
$A= \emptyset$. 
\item 
For interval $I_j$,  
\begin{itemize}
\item 
If \begin{align}
q(I_j) &\notin\left((1-\varepsilon)\cdotq(I_{j+1}),
  (1+\varepsilon)\cdotq(I_{j+1})\right)\ \ \text{ OR }\\
q(I_j) &\notin\left((1-\varepsilon)\cdotq(I_{j-1}), (1+\varepsilon)\cdotq(I_{j-1})\right),
\end{align}
 add $I_j$ to $A$. 
\end{itemize}
\item
Add the (at most 2) intervals with mass at most $1/2b$ to $A$.
\item
Add all intervals $j$ with $q(I_j)/|I_j|<\varepsilon/50n$ to $A$
\end{itemize}

If the distribution is unimodal, we can prove the following about the set of intervals ${A^c}$. 
\begin{lemma}
If $p$ is unimodal then, 
\begin{itemize}
\item
$p(I_{A^c})\ge 1-\varepsilon/25-1/b - O\left(\log n/(\varepsilon b)\right).$
\item
Except \emph{at most one} interval in ${A^c}$ every other interval $I_j$ satisfies, 
\[
\frac{\dP_j^+}{\dP_j^-}\le (1+\varepsilon). 
\]
\end{itemize}
\end{lemma}

If this holds, then the $\chi^2$ distance between $p$ and $q$ constrained to $A^c$, is at most $\varepsilon^2$. 
This lemma follows from the following result. 
\begin{lemma}
Let $C>2$. For a unimodal distribution over $[n]$, there are at
most $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals $I_j$  that satisfy $\frac{\dP_j^+}{\dP_j^-}<(1+\varepsilon/C)$. 
\end{lemma}
\begin{proof}
To the contrary, if there are more than $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals, then at least half of them are on one side of the mode, however this implies that the ratio of the largest probability and smallest probability is at least $(1+\varepsilon/C)^j$, and if $j>\frac{2\log(50n/\varepsilon)}{C\varepsilon}$, is at least $50n/\varepsilon$, contradicting that we have removed all such elements. 
\end{proof}

We have one additional pre-processing step here. We compute $q(A^c)$ and if it is smaller than $1-\varepsilon/25$, we output \textsc{Reject}\xspace. 

Suppose there are $L'$ intervals in $A^c$. Then, except at most one interval in $L'$ we know that the $\chi^2$ distance between $p$ and $q$ is at most $\varepsilon^2$ when $p$ is unimodal, and the TV distance between $p$ and $q$ is at least $\varepsilon/2$ over $A^c$. We propose the following simple modification to take into account, the one interval that might introduce a high $\chi^2$ distance in spite of having a small total variation. If we knew the interval, we can simply remove it and proceed. Since we do not know where the interval lies, we do the following. 

\begin{enumerate}
\item
Let $Z_j$ be the $\chi^2$ statistic over the $i$th interval in $A^c$, computed
with $O(\sqrt{n}/\varepsilon^2)$ samples. 
\item 
Let $Z_l$ be the largest among all $Z_j$'s.
\item 
If $\sum_{j, j\ne l}Z_j>m\varepsilon^2/10$, output \textsc{Reject}\xspace.
\item
Output \textsc{Accept}\xspace. 
\end{enumerate}

The objective of removing the largest $\chi^2$ statistic is our substitute for not knowing the largest interval. 
We now prove the correctness of this algorithm. 

\paragraph{Case 1 $p\in UM_n$:} We only concentrate on the final step. The $\chi^2$ statistic over all but one interval are at most $c\cdot m\varepsilon^2$, and the variance is bounded as before. Since we remove the largest statistic, the expected value of the new statistic is \emph{strictly dominated} by that of these intervals. Therefore, the algorithm outputs \textsc{Accept}\xspace with at least the same probability as if we removed the spurious interval. 

\paragraph{Case 2 $p\notin UM_n$:} 
This is the hard case to prove for unimodal distributions. We know that the $\chi^2$ statistic is large in this case, and we therefore have to prove that it remains large even after removing the largest test statistic $Z_l$. 

We invoke Kolmogorov's Maximal Inequality to this end. 
\begin{lemma}[Kolmogorov's Maximal Inequality]
For independent zero mean random variables $X_1,\ldots, X_L$ with finite variance, let $S_\ell=X_1+\ldots X_\ell$. Then for any $\lambda>0$, 
\begin{align}
\probof{\max_{1\le\ell\le L}\left|S_\ell\right|\ge\lambda}\le \frac{1}{\lambda^2}\cdot Var\left(S_L\right). 
\end{align}
\end{lemma}

As a corollary, it follows that $\probof{\max_{\ell}|X_\ell|>2\lambda}\le \frac{1}{\lambda^2}\cdot Var\left(S_L\right)$. 

In the case we are interested in, we let $X_i=Z_\ell-\expectation{Z_\ell}$. Then, similar to the computations before, and the fact that each interval has a small mass, it follows that that the variance of the summation is at most $\expectation{Z_\ell}^2/100$. Taking $\lambda = \expectation{S_L-m\varepsilon^2/3}^2/100$, it follows that the statistic does not fall below to $\sqrt n$. 
This completes the proof of Theorem~\ref{thm:unimodality}.

\section{Testing Unimodality}

One striking feature of Birge's result is that the decomposition of the domain is oblivious to the samples, and therefore to the unknown distribution. However, such an oblivious decomposition will not work for the unimodal distribution, since the mode is unknown. Suppose we know where the mode of the unknown distribution might be, then the problem can be decomposed into monotone functions over two intervals. Therefore, in theory, one can modify the monotonicity testing algorithm by iterating over all the possible $n$ modes. Indeed, applying a union bound, it then follows that 
\begin{theorem}(Follows from Monotone)
For $\varepsilon>1/n^{1/4}$, there is an algorithm that takes $O(\frac{n^{1/2}}{\varepsilon^2}\log n)$ samples, and can test unimodality.  
\end{theorem}

\jnote{It is then easy to extend this to $t$-modal distributions in the most trivial way by taking $t\sqrt{n}/\varepsilon^2$ samples. We dont like it, and will strive for the separation of testing and learning.}.

However, this is not satisfactory since we were only able to prove a lower bound of $\sqrt{n}/\varepsilon^2$. In order to overcome the logarithmic barrier introduced by the union bound, we resort to a non-oblivious decomposition of the domain. 

We want to decompose the interval into disjoint intervals such that the probability of each interval is about $O(1/b)$, where $b$ is a parameter, specified later. In particular we consider the following decomposition of  $[n]$. 
\begin{enumerate}
\item
For each element $i$ with probability at least $1/b$, there is an $I_\ell=\{i\}$.
\item
There are at most two intervals with $p(I)\le 1/{2b}$.
\item
Every other interval $I$ satisfies $p(I)\in\left[\frac1{2b},\frac2b\right]$.
\end{enumerate}

Let $I_1, \ldots, I_L$ denote the partition of $[n]$ corresponding to these intervals. Note that $L= O(b)$. 
\begin{claim}
There is an algorithm that takes $O(b\log b)$ samples and outputs $I_1, \ldots, I_L$ satisfying the properties above.
\end{claim}

The first step in our algorithm is to estimate the \emph{total probability} within each of these intervals. 
In particular, 
\begin{lemma}
There is an algorithm that takes $m'=O(b\log b/\varepsilon^2)$ samples from a distribution $p$, and with probability at least 9/100 outputs a distribution $\bar{\dQ}$ that is constant on each $I_L$. Moreover, for any $j$ such that $p(I_j)>1/2b$, 
$q(I_j)\in(1\pm\varepsilon)q(I_j)$. 
\end{lemma}
\begin{proof}
Consider any interval $I_j$ with $p(I_j)\ge 1/2b$. The number of samples $N_{I_j}$ that fall in that interval is distributed $Binomial(m', p(I_j)$. Then by Chernoff bounds for $m'>12 b\log b/\varepsilon^2$, 
\begin{align}
\probof{|N_{I_j}-m'p(I_j)|>\varepsilon m'p(I_j) }\le& 2\exp\left(\varepsilon^2m'p(I_j)/2\right)\\
\le & \frac1{b^2},
\end{align}
where the last inequality uses the fact that $p(I_j)\ge 1/2b$. 
\end{proof}

The next step is estimate the distance of $q$ from $UM_{n}$. This is possible by a simple dynamic program. If the estimated distance is more than $\varepsilon/2$, we output \textsc{Reject}\xspace. 

Our next step is to remove certain intervals. This will be to ensure that when the underlying distribution is unimodal, we are able to estimate the distribution \emph{multiplicatively} over the remaining intervals. 
In particular, we do the following preprocessing step:
\begin{itemize}
\item 
$A= \emptyset$. 
\item 
For interval $I_j$,  
\begin{itemize}
\item 
If \begin{align}
q(I_j) &\notin\left((1-\varepsilon)\cdotq(I_{j+1}),
  (1+\varepsilon)\cdotq(I_{j+1})\right)\ \ \text{ OR }\\
q(I_j) &\notin\left((1-\varepsilon)\cdotq(I_{j-1}), (1+\varepsilon)\cdotq(I_{j-1})\right),
\end{align}
 add $I_j$ to $A$. 
\end{itemize}
\item
Add the (at most 2) intervals with mass at most $1/2b$ to $A$.
\item
Add all intervals $j$ with $q(I_j)/|I_j|<\varepsilon/50n$ to $A$
\end{itemize}

If the distribution is unimodal, we can prove the following about the set of intervals ${A^c}$. 
\begin{lemma}
If $p$ is unimodal then, 
\begin{itemize}
\item
$p(I_{A^c})\ge 1-\varepsilon/25-1/b - O\left(\log n/(\varepsilon b)\right).$
\item
Except \emph{at most one} interval in ${A^c}$ every other interval $I_j$ satisfies, 
\[
\frac{\dP_j^+}{\dP_j^-}\le (1+\varepsilon). 
\]
\end{itemize}
\end{lemma}

If this holds, then the $\chi^2$ distance between $p$ and $q$ constrained to $A^c$, is at most $\varepsilon^2$. 
This lemma follows from the following result. 
\begin{lemma}
Let $C>2$. For a unimodal distribution over $[n]$, there are at most most $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals $I_j$  satisfy $\frac{\dP_j^+}{\dP_j^-}<(1+\varepsilon/C)$. 
\end{lemma}
\begin{proof}
To the contrary, if there are more than $\frac{4\log(50n/\varepsilon)}{C\varepsilon}$ intervals, then at least half of them are on one side of the mode, however this implies that the ratio of the largest probability and smallest probability is at least $(1+\varepsilon/C)^j$, and if $j>\frac{2\log(50n/\varepsilon)}{C\varepsilon}$, is at least $50n/\varepsilon$, contradicting that we have removed all such elements. 
\end{proof}

We have one additional pre-processing step here. We compute $q(A^c)$ and if it is smaller than $1-\varepsilon/25$, we output \textsc{Reject}\xspace. 

Suppose there are $L'$ intervals in $A^c$. Then, except at most one interval in $L'$ we know that the $\chi^2$ distance between $p$ and $q$ is at most $\varepsilon^2$ when $p$ is unimodal, and the TV distance between $p$ and $q$ is at least $\varepsilon/2$ over $A^c$. We propose the following simple modification to take into account, the one interval that might introduce a high $\chi^2$ distance in spite of having a small total variation. If we knew the interval, we can simply remove it and proceed. Since we do not know where the interval lies, we do the following. 

\begin{enumerate}
\item
Let $Z_j$ be the $\chi^2$ statistic over the $i$th interval in $A^c$, computed
with $O(\sqrt{n}/\varepsilon^2)$ samples. 
\item 
Let $Z_l$ be the largest among all $Z_j$'s.
\item 
If $\sum_{j, j\ne l}Z_j>m\varepsilon^2/10$, output \textsc{Reject}\xspace.
\item
Output \textsc{Accept}\xspace. 
\end{enumerate}

The objective of removing the largest $\chi^2$ statistic is our substitute for not knowing the largest interval. 
We now prove the correctness of this algorithm. 

\paragraph{Case 1 $p\in UM_n$:} We only concentrate on the final step. The $\chi^2$ statistic over all but one interval are at most $c\cdot m\varepsilon^2$, and the variance is bounded as before. Since we remove the largest statistic, the expected value of the new statistic is \emph{strictly dominated} by that of these intervals. Therefore, the algorithm outputs \textsc{Accept}\xspace with at least the same probability as if we removed the spurious interval. 

\paragraph{Case 2 $p\notin UM_n$:} 
This is the hard case to prove for unimodal distributions. We know that the $\chi^2$ statistic is large in this case, and we therefore have to prove that it remains large even after removing the largest test statistic $Z_l$. 

We invoke Kolmogorov's Maximal Inequality to this end. 
\begin{lemma}[Kolmogorov's Maximal Inequality]
For independent zero mean random variables $X_1,\ldots, X_L$ with finite variance, let $S_\ell=X_1+\ldots X_\ell$. Then for any $\lambda>0$, 
\begin{align}
\probof{\max_{1\le\ell\le L}\left|S_\ell\right|\ge\lambda}\le \frac{1}{\lambda^2}\cdot Var\left(S_L\right). 
\end{align}
\end{lemma}

As a corollary, it follows that $\probof{\max_{\ell}|X_\ell|>2\lambda}\le \frac{1}{\lambda^2}\cdot Var\left(S_L\right)$. 

In the case we are interested in, we let $X_i=Z_\ell-\expectation{Z_\ell}$. Then, similar to the computations before, and the fact that each interval has a small mass, it follows that that the variance of the summation is at most $\expectation{Z_\ell}^2/100$. Taking $\lambda = \expectation{S_L-m\varepsilon^2/3}^2/100$, it follows that the statistic does not fall below to $\sqrt n$. 

\begin{theorem}
\label{thm:unimodality}
For $\varepsilon>n^{-1/4}$ samples, there is an algorithm that takes $O(\sqrt{n}/\varepsilon^2)$ samples to test unimodality. 
\end{theorem}





\section{Testing Unimodality}
\label{sec:unimodal-main}
One striking feature of Birg\'e's result is that the decomposition of the domain is oblivious to the samples, and therefore to the unknown distribution. However, such an oblivious decomposition will not work for the unimodal distribution, since the mode is unknown. Suppose we know where the mode of the unknown distribution might be, then the problem can be decomposed into monotone functions over two intervals. Therefore, in theory, one can modify the monotonicity testing algorithm by iterating over all the possible $n$ modes. Indeed, by applying a union bound, it then follows that 
\begin{theorem}(Follows from Monotone)
For $\varepsilon>1/n^{1/4}$, there exists an algorithm for testing unimodality over $[n]$ with sample complexity $O\left(\frac{\sqrt{n}}{\varepsilon^2}\log n\right)$.
\end{theorem}

However, this is unsatisfactory, since our lower bound (and as we will demonstrate, the true complexity of this problem) is $\sqrt{n}/\varepsilon^2$.
We overcome the logarithmic barrier introduced by the union bound, by
employing a non-oblivious decomposition of the domain, and using 
Kolmogorov's max-inequality. 

Our main result for testing unimodality is the following theorem,
which is proved in Section~\ref{sec:unimodal-appendix}.
\begin{theorem}
\label{thm:unimodality}
Suppose $\varepsilon>n^{-1/4}$.
Then there exists an algorithm for testing unimodality over $[n]$ with
sample complexity $O(\sqrt{n}/\varepsilon^2)$. 
\end{theorem}


2015-12-09T02:14:38.000
1507
1507.05952
en
https://arxiv.org/abs/1507.05952
Given samples from an unknown distribution $p$, is it possible to distinguish whether $p$ belongs to some class of distributions $\mathcal{C}$ versus $p$ being far from every distribution in $\mathcal{C}$? This fundamental question has received tremendous attention in statistics, focusing primarily on asymptotic analysis, and more recently in information theory and theoretical computer science, where the emphasis has been on small sample size and computational complexity. Nevertheless, even for basic properties of distributions such as monotonicity, log-concavity, unimodality, independence, and monotone-hazard rate, the optimal sample complexity is unknown.We provide a general approach via which we obtain sample-optimal and computationally efficient testers for all these distribution families. At the core of our approach is an algorithm which solves the following problem: Given samples from an unknown distribution $p$, and a known distribution $q$, are $p$ and $q$ close in $\chi^2$-distance, or far in total variation distance?The optimality of our testers is established by providing matching lower bounds with respect to both $n$ and $\varepsilon$. Finally, a necessary building block for our testers and an important byproduct of our work are the first known computationally efficient proper learners for discrete log-concave and monotone hazard rate distributions.
Data Structures and Algorithms (cs.DS); Information Theory (cs.IT); Machine Learning (cs.LG); Statistics Theory (math.ST)
Optimal Testing for Properties of Distributions
Qwen/Qwen-72B
1. YES
2. YES