Efficient Representation of Large-Alphabet Probability Distributions

Abstract: A number of engineering and scientific problems require representing and manipulating probability distributions over large alphabets, which we may think of as long vectors of reals summing to $1$. In some cases it is required to represent such a vector with only $b$ bits per entry. A natural choice is to partition the interval $[0,1]$ into $2^b$ uniform bins and quantize entries to each bin independently. We show that a minor modification of this procedure -- applying an entrywise non-linear function (compander) $f(x)$ prior to quantization -- yields an extremely effective quantization method. For example, for $b=8 (16)$ and $10^5$-sized alphabets, the quality of representation improves from a loss (under KL divergence) of $0.5 (0.1)$ bits/entry to $10^{-4} (10^{-9})$ bits/entry. Compared to floating point representations, our compander method improves the loss from $10^{-1}(10^{-6})$ to $10^{-4}(10^{-9})$ bits/entry. These numbers hold for both real-world data (word frequencies in books and DNA $k$-mer counts) and for synthetic randomly generated distributions. Theoretically, we set up a minimax optimality criterion and show that the compander $f(x) ~\propto~ \mathrm{ArcSinh}(\sqrt{(1/2) (K \log K) x})$ achieves near-optimal performance, attaining a KL-quantization loss of $\asymp 2^{-2b} \log^2 K$ for a $K$-letter alphabet and $b\to \infty$. Interestingly, a similar minimax criterion for the quadratic loss on the hypercube shows optimality of the standard uniform quantizer. This suggests that the $\mathrm{ArcSinh}$ quantizer is as fundamental for KL-distortion as the uniform quantizer for quadratic distortion.

\section{Introduction}

\vspace{-0.5pc}

\section{Compander Basics and Definitions}

Consider the problem of \emph{quantizing} the probability simplex $\triangle_{\ensuremath{K}-1} = \{\bx \in \bbR^\ensuremath{K} : \bx \geq \bzero, \sum_i x_i = 1 \}$ of alphabet size $\ensuremath{K}$,\footnote{While the alphabet has $\ensuremath{K}$ letters, $\triangle_{\ensuremath{K}-1}$ is $\ensuremath{K}-1$ dimensional due to the constraint that the entries sum to $1$.} i.e. of finding a finite subset $\cY \subseteq \triangle_{\ensuremath{K}-1}$ to represent the entire simplex.  Each $\bx \in \triangle_{\ensuremath{K}-1}$ is associated with some $\by = \by(\bx) \in \cY$, and the objective is to find a set $\cY$ and an assignment such that the difference between the values $\bx \in \triangle_{\ensuremath{K}-1}$ and their representations $\by \in \cY$ are minimized; while this can be made arbitrarily small by making $\cY$ arbitrarily large, the objective is to do this efficiently for any given fixed size $M$. Since $\bx, \by \in \triangle_{\ensuremath{K}-1}$, they both represent probability distributions over a size-$\ensuremath{K}$ alphabet. Hence, a natural way to measure the quality of the quantization is to use the KL (Kullback-Leibler) divergence $D_{\kl}(\bx \| \by)$, which corresponds to the excess code length for lossless compression and is commonly used as a way to compare probability distributions. (Note that we want to minimize the KL divergence.)

While one can consider how to best represent the vector $\bx$ as a whole, in this paper we consider only \emph{scalar quantization} methods in which each element $x_j$ of $\bx$ is handled separately, since we showed in \cite{adler_ratedistortion_2021} that for Dirichlet priors on the simplex, methods using scalar quantization perform nearly as well as optimal vector quantization. Scalar quantization is also typically simpler and faster to use, and can be parallelized easily.
Our scalar quantizer is based on \emph{companders} (portmanteau of `compressor' and `expander'), a simple, powerful and flexible technique first explored by Bennett in 1948 \cite{bennett1948} in which the value $x_j$ is passed through a nonlinear function $f$ before being quantized. We discuss the background in greater depth in \Cref{sec::previous_works}, in particular to compare certain previously-known results to \Cref{thm::asymptotic-normalized-expdiv}.

In what follows, $\log$ is always base-$e$ unless otherwise specified.

\subsubsection{Encoding}

Companders require two things: a monotonically increasing function $\ensuremath{f}:[0,1] \to [0, 1]$ (we denote the set of such functions as $\ensuremath{\cF}$) and an integer $N$ representing the number of quantization levels, or \emph{granularity}. To simplify the problem and algorithm, we use the same $\ensuremath{f}$ for each element of the vector $\bx = (x_1, \dots, x_{\ensuremath{K}}) \in \triangle_{\ensuremath{K}-1}$ (see \Cref{rmk::symmetric-distribution}). To quantize $x \in [0, 1]$, the compander computes $\ensuremath{f}(x)$ and applies a uniform quantizer with $N$ levels, i.e. encoding $x$ to $n_N(x) \in [N]$ if $\ensuremath{f}(x) \in (\frac{n-1}{N}, \frac{n}{N}]$; this is equivalent to $n_N(x) = \lceil \ensuremath{f}(x) N \rceil$.

This encoding system partitions $[0,1]$ into \emph{bins} $I^{(n)}$:
\begin{align}\label{eq::bins}
    x \in I^{(n)} = \ensuremath{f}^{-1} \Big(\Big(\frac{n-1}{N}, \frac{n}{N}\Big] \Big) \iff n_N(x) = n
\end{align}
where $\ensuremath{f}^{-1}$ denotes the preimage under $f$.

\subsubsection{Decoding} \label{sec::decoding}

To decode $n \in [N]$, we pick some $\widehat{y}_{(n)} \in I^{(n)}$ to represent all $x \in I^{(n)}$; for a given $x$ (at granularity $N$), its representation is denoted $\widehat{y}(x) = \widehat{y}_{(n_N(x))}$. This is usually the \emph{midpoint} of the bin or, if $x$ is drawn randomly from a prior\footnote{Priors on $\triangle_{\ensuremath{K}-1}$ induce priors over $[0,1]$ for each entry.} $p$, the \emph{centroid} (the mean within bin $I^{(n)}$).
The midpoint of $I^{(n)}$ can be computed as $(\ensuremath{f}^{-1}(\frac{n-1}{N}) + \ensuremath{f}^{-1}(\frac{n}{N}) )/ 2$ and we denote it as $\bar{y}_{(n)}$. The centroid of $I^{(n)}$ is defined as
\begin{align}
    \widetilde{y}_{(n)} = \bbE_{X \sim p} [X \, | \, X \in I^{(n)}] \,.
\end{align}
We will discuss this in greater detail in \Cref{sec::x-from-prior}.

Handling each element of $\bx$ separately means the decoded values may not sum to $1$, so we normalize the vector after decoding. Thus, if $\bx$ is the input,
\begin{align}\label{eq::norm_step}
    y_i(\bx) = \frac{\widehat{y}(x_i)}{\sum_{j = 1}^\ensuremath{K} \widehat{y}(x_j)}
\end{align}
the vector $\by = \by(\bx) = (y_1(\bx), \dots, y_\ensuremath{K}(\bx)) \in \triangle_{\ensuremath{K}-1}$ is the output of the compander.\footnote{This notation reflects the fact that each entry of the normalized reconstruction depends on the entire vector $\bx$ due to the normalization step.} We refer to $\widehat{\by} = \widehat{\by}(\bx) = (\widehat{y}(x_1), \dots, \widehat{y}(x_\ensuremath{K}))$ as the \emph{raw} reconstruction of $\bx$, and $\by$ as the \emph{normalized} reconstruction. (We generally use the $\widehat{\cdot}$ accent to mark values dependent on the raw reconstruction.)
If the raw reconstruction uses centroid decoding, we likewise denote it using $\widetilde{\by} = \widetilde{\by}(\bx) = (\widetilde{y}(x_1), \dots, \widetilde{y}(x_\ensuremath{K}))$.

Thus, any $\bx \in \triangle_{\ensuremath{K}-1}$ requires $\ensuremath{K} \lceil \log_2 N \rceil$  bits to store; to encode and decode, only $\ensuremath{f}$ and $N$ need to be stored (as well as the prior if using centroid decoding). Another major advantage is that a single $\ensuremath{f}$ can work well over many or all choices of $N$, making the design more flexible.

\subsubsection{KL divergence loss}

The loss incurred by representing $\bx$ as $\by(\bx)$ is the KL divergence
\begin{align}\label{eq::kl_loss_norm}
    D_{\kl}(\bx\| \by(\bx)) = \sum_{i=1}^{\ensuremath{K}} x_i \log \frac{x_i}{y_i(\bx)} \,.
\end{align}
Although this loss function has some strange properties (for instance $D_{\kl}(\bx \| \by) \neq D_{\kl}(\by \| \bx)$ and it doesn't obey the triangle inequality) it measures the amount of `mis-representation' created by representing the probability vector $\bx$ by another probability vector $\by(\bx)$, and is hence is a natural thing to minimize. In particular, it represents the excess code length created by trying to encode the output of $\bx$ using a code built for $\by(\bx)$, as well as having connections to hypothesis testing (a natural setting in which the `difference' between probability distributions is studied).

\subsubsection{Distributions from a prior} \label{sec::x-from-prior}

Much of our work concerns the case where $\bx \in \triangle_{\ensuremath{K}-1}$ is drawn from some prior $P_\bx$ (to be commonly denoted as simply $P$). Using a single $\ensuremath{f}$ for each entry means we can WLOG assume that $P$ is symmetric over the alphabet, as permuting the letter indices does not affect the KL divergence. We denote the set of such priors as $\cP^\triangle_{\ensuremath{K}}$.

\begin{remark} \label{rmk::symmetric-distribution}
In principle, given a nonsymmetric prior $P_\bx$ over $\triangle_{\ensuremath{K}-1}$ with marginals $p_1, \dots, p_\ensuremath{K}$, we could quantize each letter's value with a different compander $f_1, \dots, f_\ensuremath{K}$, giving more accuracy than using a single $f$ (at the cost of higher complexity). However, the symmetrization of $P_\bx$ over the letters (by permuting the indices randomly after generating $\bX \sim P_\bx$) yields a prior in $\cP^\triangle_{\ensuremath{K}}$ on which any single $f$ will have the same (overall) performance and cannot be improved on by using varying $f_i$. Thus, considering symmetric $P_\bx$ suffices to derive our minimax compander (which performs well across all $P_\bx$ whose marginals are continuous probability distributions).
\end{remark}

While the random probability vector comes from a prior $P \in \cP^\triangle_{\ensuremath{K}}$, our analysis will rely on decomposing the loss so we can deal with one letter at a time. Hence, we work with the marginals $p$ of $P$ (which are identical since $P$ is symmetric), which we refer to as \emph{single-letter distributions} and are probability distributions over $[0,1]$.

We let $\cP$ denote the class of probability distributions over $[0,1]$ that are absolutely continuous with respect to the Lebesgue measure. We denote elements of $\cP$ by their probability density functions (PDF), e.g. $p \in \cP$; the cumulative distribution function (CDF) associated with $p$ is denoted $F_p$ and satisfies $F'_p(x) = p(x)$ and $F_p(x) = \int_0^x p(t) \, dt$ (since $F_p$ is monotonic, its derivative exists almost everywhere). Note that while $p \in \cP$ does not have to be continuous, its CDF $F_p$ must be absolutely continuous. Following common terminology~\cite{grimmett2001}, we refer to such probability distributions as \emph{continuous}.

Let $\cP_{1/\ensuremath{K}} = \{p \in \cP : \bbE_{X\sim p}[X] = 1/\ensuremath{K}\}$. Note that $P \in \cP^\triangle_{\ensuremath{K}}$ implies its marginals are in $\cP_{1/\ensuremath{K}}$.

\subsubsection{Expected loss and preliminary results}

For $P \in \cP^\triangle_{\ensuremath{K}}$, $\ensuremath{f} \in \ensuremath{\cF}$ and granularity $N$, we define the \emph{expected loss}:
\begin{align}
    \cL_{\ensuremath{K}}(P, \ensuremath{f}, N) = \bbE_{\bX \sim P}[D_{\kl}(\bX \| \by(\bX))]\,.
\end{align}
This is the value we want to minimize.

\begin{remark}
While $\bX$ and $\by(\bX)$ are random, they are also probability vectors. The KL divergence $D_{\kl}(\bX \| \by(\bX))$ is the divergence between $\bX$ and $\by(\bX)$ themselves, not the prior distributions over $\triangle_{\ensuremath{K}-1}$ they are drawn from.
\end{remark}

Note that $\cL_{\ensuremath{K}}(P,\ensuremath{f},N)$ can almost be decomposed into a sum of $\ensuremath{K}$ separate expected values (one per entry), except the normalization step \eqref{eq::norm_step} depends on the vector as a whole. Hence, we define the \emph{raw loss}:
\begin{align} \label{eq::raw-loss}
    \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P, \ensuremath{f}, N) = \bbE_{\bX \sim P}\Big[\sum_{i=1}^\ensuremath{K} X_i \log(X_i/\widetilde{y}(X_i))\Big]\,.
\end{align}
We also define for $p \in \cP$, the \emph{single-letter loss} as
\begin{align} \label{eq::raw-ssl}
    \ensuremath{\widetilde{L}}(p, \ensuremath{f}, N) =  \bbE_{X \sim p} \big[ X \log ( X/\widetilde{y}(X)) \big]
\end{align}
The raw loss is useful because it bounds the (normalized) expected loss and is decomposable into single-letter losses. Note that both raw and single-letter loss are defined with centroid decoding.
\begin{proposition}\label{lem::im-a-barby-girl}
For $P \in \cP^\triangle_{\ensuremath{K}}$ with marginals $p$,
    \begin{align}
        \cL_{\ensuremath{K}}(P, \ensuremath{f}, N) \leq \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P, \ensuremath{f}, N) = \ensuremath{K} \, \ensuremath{\widetilde{L}}(p,\ensuremath{f},N)\,.
    \end{align}
\end{proposition}

\iflon
\begin{proof}
Since
\begin{align}
    \cL  (P, \ensuremath{f}, N) \ensuremath{} & =  \bbE_{\bX \sim P} D_{\kl}(\bX || \bY) \\

    & =  \ensuremath{\widetilde{\cL}}_\ensuremath{K}(P, \ensuremath{f}, N) + \bbE_{\bX \sim P}  \left[ \log \left( \sum_{k=1}^\ensuremath{K} \widetilde{Y}_k \right)\right] \,.
\end{align}
Since $\bbE[\widetilde{Y}_k] = \bbE[X_k]$ for all $k$, $\sum_{k = 1}^\ensuremath{K} \bbE[\widetilde{Y}_k] =\sum_{k = 1}^\ensuremath{K} \bbE[{X}_k] = 1 $.
Because $\log$ is concave, by Jensen's Inequality
\begin{align}
    \bbE_{\bX \sim P} \bigg[\log \Big( \sum_{k=1}^\ensuremath{K} \widetilde{Y}_k \Big)\bigg] \leq \log \Big( \bbE \Big[\sum_{k=1}^\ensuremath{K} \widetilde{Y}_k\Big] \Big) = \log(1) = 0
\end{align}
and we are done.
\end{proof}
\fi
To derive our results about worst-case priors (for instance, \Cref{thm::minimax_compander}), we will also be interested in $\ensuremath{\widetilde{L}}(p,\ensuremath{f},N)$ even when $p$ is not known to be a marginal of some $P \in \cP^\triangle_{\ensuremath{K}}$.

\begin{remark} \label{rmk::centroid-needed}
Though one can define raw loss and single-letter loss without centroid decoding (replacing $\widetilde{y}$ in \eqref{eq::raw-loss} or \eqref{eq::raw-ssl} with another decoding method $\widehat{y}$), doing so removes much of their usefulness. This is because the resulting expected loss can be dominated by the difference between $\bbE[X]$ and $\bbE[\widehat{y}(X)]$, potentially even making it negative; specifically, the Taylor expansion of $X \log(X/\widehat{y}(X))$ has $X - \widehat{y}(X)$ in its first term, which can have negative expectation.

While this can make the expected `raw loss' general decoding negative, it cannot be exploited to make the (normalized) expected loss negative\footnote{As expected, since negative KL loss between probability distributions is not possible.} because the normalization step $y_i(\bX) = \widehat{y}(X_i)/\sum_j \widehat{y}(X_j)$ cancels out the problematic term. Centroid decoding avoids this problem by ensuring $\bbE[X] = \bbE[\widehat{y}(X)]$, removing the issue.
\end{remark}

As we will show, when $N$ is large these values are roughly proportional to $N^{-2}$ (for well-chosen $\ensuremath{f}$) and hence we define the \emph{asymptotic single-letter loss}:
\begin{align} \label{eq::raw-assl}
    \ensuremath{\widetilde{L}}(p,\ensuremath{f}) = \lim_{N \to \infty} N^2 \ensuremath{\widetilde{L}}(p,\ensuremath{f},N)\,.
\end{align}
We similarly define $\ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P,\ensuremath{f})$ and $\cL_{\ensuremath{K}}(P,\ensuremath{f})$.
While the limit in \eqref{eq::raw-assl} does not exist for every $p, \ensuremath{f}$, we will show that one can ensure it exists by choosing an appropriate $\ensuremath{f}$ (which works against any $p \in \cP$), and cannot gain much by not doing so.

\section{Main Results}

\label{sec::main-theorems}

We demonstrate, theoretically and experimentally, the efficacy of companding for quantizing probability distributions with KL divergence loss.
Though our theoretical results are asymptotic as $N \to \infty$ and focus on raw loss (which uses centroid decoding), the experimental loss of the various companders with midpoint decoding (normalized) closely tracks the raw loss predicted theoretically, even for quantization levels as low as $N = 256$ ($8$ bits per value).

\subsection{Theoretical Results}

We first define a set of `well-behaved' companders:
\begin{definition}
Define $\ensuremath{\cF}^\dagger \subseteq \ensuremath{\cF}$ to be the set of $\ensuremath{f}$ such that for each $f$ there exist constants $c > 0$ and $\alpha \in (0,1/2]$ for which $\ensuremath{f}(x) - c x^{\alpha}$ is still monotonically increasing.
\end{definition}
This is equivalent to $\ensuremath{f'}(x) \geq c\,\alpha\,x^{\alpha-1}$ for all $x$ where $\ensuremath{f'}$ is defined (which is almost everywhere since $\ensuremath{f}$ is monotonic). We also define the following function on $p$ and $\ensuremath{f}$:
\begin{definition}
For $p \in \cP$ and $\ensuremath{f} \in \ensuremath{\cF}$, let
\begin{align}\label{eq::raw_loss}
    L^\dagger(p,\ensuremath{f}) &= \frac{1}{24} \int_0^1 p(x) \ensuremath{f'}(x)^{-2} x^{-1} \, dx
    \\ &= \int_{[0,1]} \frac{1}{24} \ensuremath{f'}(x)^{-2} x^{-1} \, dp\,.
\end{align}
\end{definition}

Then the asymptotic loss of $\ensuremath{f}$ against $p$ satisfies:
\begin{theorem}  \label{thm::asymptotic-normalized-expdiv}
For any $p \in \cP$ and $\ensuremath{f} \in \ensuremath{\cF}$,
\begin{align}
    \liminf_{N \to \infty} N^2 \ensuremath{\widetilde{L}}(p,\ensuremath{f},N) \geq L^\dagger(p,\ensuremath{f}) \,. \label{eq::fatou-bound}
\end{align}
Furthermore, if $\ensuremath{f} \in \ensuremath{\cF}^\dagger$ then an exact result holds:
\begin{align}
    \ensuremath{\widetilde{L}}(p,\ensuremath{f}) &= L^\dagger(p,\ensuremath{f}) < \infty

    \label{eq::norm_loss}
    \,.
\end{align}

\end{theorem}

Essentially, as long as you select a compander $\ensuremath{f}$ from the `well-behaved' set $\ensuremath{\cF}^\dagger$, for large granularities $N$ the single-letter loss will be approximated by
\begin{align}
    \ensuremath{\widetilde{L}}(p,\ensuremath{f},N) \approx N^{-2} L^\dagger (p,\ensuremath{f}) \, .
\end{align}
The lower bound \eqref{eq::fatou-bound} shows that even for $\ensuremath{f} \not \in \ensuremath{\cF}^\dagger$,
\begin{align}
    \ensuremath{\widetilde{L}}(p,\ensuremath{f},N) \gtrsim N^{-2} L^\dagger (p,\ensuremath{f})
\end{align}
i.e. the quantizer cannot do better than $N^{-2} L^\dagger(p,\ensuremath{f})$ loss (as $N \to \infty$) by choosing $\ensuremath{f} \not \in \ensuremath{\cF}^\dagger$.

\begin{theorem} \label{thm::optimal_compander_loss}
The best loss against source $p \in \cP$ is
\begin{align}
    \hspace{-0.75pc}  \inf_{\ensuremath{f} \in \ensuremath{\cF}} \ensuremath{\widetilde{L}}(p,\ensuremath{f}) \hspace{-0.1pc} = \hspace{-0.1pc} \min_{\ensuremath{f} \in \ensuremath{\cF}} L^\dagger(p,\ensuremath{f}) \hspace{-0.1pc} = \hspace{-0.1pc} \frac{1}{24} \Big(\int_0^1 (p(x)x^{-1})^{1/3} dx\Big)^3
    \label{eq::raw_overall_dist}
\end{align}
where the \emph{optimal compander against $p$} is
\begin{align}
    &\ensuremath{f}_p(x) = \argmin_{\ensuremath{f} \in \ensuremath{\cF}} L^\dagger(p,\ensuremath{f}) = \frac{\int_0^x (p(t)t^{-1})^{1/3} \, dt}{\int_0^1 (p(t)t^{-1})^{1/3} \, dt} \label{eq::best_f_raw}
\end{align}
(satisfying  $\ensuremath{f'}_p(x) \, \propto \, (p(x) x^{-1})^{1/3}$).
\end{theorem}
If $\ensuremath{f}_p \in \ensuremath{\cF}^\dagger$, it achieves the value from \eqref{eq::raw_overall_dist} and (as minimizer of $L^\dagger(p,\ensuremath{f})$) it has the smallest asymptotic loss against $p$. If $\ensuremath{f}_p \not \in \ensuremath{\cF}^\dagger$, we use the following:
\begin{proposition} \label{prop::approximate-compander}
For any $\ensuremath{f} \in \ensuremath{\cF}$ and $\delta \in (0,1]$, the functions
\begin{align}
    \ensuremath{f}_{p,\delta}(x) = (1-\delta) \ensuremath{f}_p(x) + \delta x^{1/2} \label{eq::approximate-compander}
\end{align}
satisfy $\ensuremath{f}_{p,\delta} \in \ensuremath{\cF}^\dagger$ and
\begin{align}
    \lim_{\delta \to 0} \ensuremath{\widetilde{L}}(p,\ensuremath{f}_{p,\delta}) = \lim_{\delta \to 0} L^\dagger(p,\ensuremath{f}_{p,\delta}) = L^\dagger(p,\ensuremath{f}_p) \label{eq::approximate-optimal-compander}\,.
\end{align}
\end{proposition}

Thus, you can imitate $\ensuremath{f}_p$ arbitrarily closely by mixing it with $x^{1/2}$ (or any $x^\alpha$ for $\alpha \in (0,1/2]$ will also work); the mixture is by definition in $\ensuremath{\cF}^\dagger$. This (with \Cref{thm::asymptotic-normalized-expdiv}) shows there is no real advantage to using $\ensuremath{f} \not \in \ensuremath{\cF}^\dagger$, so we restrict our analysis to $\ensuremath{f} \in \ensuremath{\cF}^\dagger$, for which
\eqref{eq::raw_loss} holds.

Since the prior $P$ generating $\bx$ is usually unknown, we give a compander which performs well against \emph{any} prior. This is closely linked to the following probability density on $[0,1]$:

\begin{proposition}
For alphabet size $\ensuremath{K} > 4$, there is a unique $c_{\ensuremath{K}} \in [\frac{1}{4}, \frac{3}{4}]$ such that if $a_{\ensuremath{K}} = (4/(c_{\ensuremath{K}} \ensuremath{K} \log \ensuremath{K} + 1))^{1/3}$ and $b_{\ensuremath{K}} = 4/a_{\ensuremath{K}}^2 - a_{\ensuremath{K}}$, then the following density is in $\cP_{1/\ensuremath{K}}$:
\begin{align}
    &p^*_{\ensuremath{K}}(x) = (a_{\ensuremath{K}} x^{1/3} + b_{\ensuremath{K}} x^{4/3})^{-3/2} \label{eq::maximin-density}
\end{align}
Furthermore, $\lim_{\ensuremath{K} \to \infty} c_{\ensuremath{K}} = 1/2$.
\end{proposition}

We call $p^*_{\ensuremath{K}}$ the \emph{maximin single-letter density}.

\noindent The optimal compander against $p^*_{\ensuremath{K}}$ is the \emph{minimax compander}:
\begin{align}\label{eq::minimax-compander}
    \ensuremath{f}^*_{\ensuremath{K}}(x) = \frac{\ensuremath{\mathrm{ArcSinh}}(\sqrt{c_{\ensuremath{K}}  (\ensuremath{K} \log \ensuremath{K}) \, x})}{\ensuremath{\mathrm{ArcSinh}}(\sqrt{c_{\ensuremath{K}}  \ensuremath{K} \log \ensuremath{K}})}\,.
\end{align}
Note that $\ensuremath{f}^*_{\ensuremath{K}} \in \ensuremath{\cF}^\dagger$ (see \Cref{rmk::minimax-is-closed-form}). The source $p^*_{\ensuremath{K}}$ and compander $\ensuremath{f}^*_{\ensuremath{K}}$ then form an `equilibrium':
\begin{theorem}\label{thm::minimax_compander}
The minimax compander $\ensuremath{f}^*_{\ensuremath{K}}$ and maximin single-letter density $p^*_{\ensuremath{K}}$ satisfy
\begin{align}
    &\sup_{p \in \cP_{1/\ensuremath{K}}} \ensuremath{\widetilde{L}}(p,\ensuremath{f}^*_{\ensuremath{K}}) = \inf_{\ensuremath{f} \in \ensuremath{\cF}^\dagger} \sup_{p \in \cP_{1/\ensuremath{K}}} \ensuremath{\widetilde{L}}(p,\ensuremath{f})   \label{eq::minmax}
    \\ = & \sup_{p \in \cP_{1/\ensuremath{K}}} \inf_{\ensuremath{f} \in \ensuremath{\cF}^\dagger} \ensuremath{\widetilde{L}}(p,\ensuremath{f}) = \inf_{\ensuremath{f} \in \ensuremath{\cF}^\dagger} \ensuremath{\widetilde{L}}(p^*_{\ensuremath{K}}, \ensuremath{f}) \label{eq::maxmin}
\end{align}
which is equal to $\ensuremath{\widetilde{L}}(p^*_{\ensuremath{K}}, \ensuremath{f}^*_{\ensuremath{K}})$ and satisfies
\begin{align} \label{eq::raw_loss_saddle}
    \ensuremath{\widetilde{L}}(p^*_{\ensuremath{K}}, \ensuremath{f}^*_{\ensuremath{K}}) = \Theta(\ensuremath{K}^{-1}\log^2 \ensuremath{K})\,.
\end{align}
\end{theorem}

This theorem importantly implies the following:
\begin{corollary}\label{cor::worstcase_prior}
For any prior $P \in \cP^{\triangle}_{\ensuremath{K}}$,
\begin{align}
    \cL_{\ensuremath{K}}(P,\ensuremath{f}^*_{\ensuremath{K}}) \leq \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P,\ensuremath{f}^*_{\ensuremath{K}}) = \Theta(\log^2 \ensuremath{K} )\,.
\end{align}

There also exists $P^* \in \cP^{\triangle}_{\ensuremath{K}}$ such that for any $P \in \cP^\triangle_{\ensuremath{K}}$
\begin{align}\label{eq::bound_worstcase_prior}
    \inf_{\ensuremath{f} \in \ensuremath{\cF}} \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P^*, \ensuremath{f}) \geq \frac{\ensuremath{K} - 1}{2\ensuremath{K}} \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P, \ensuremath{f}^*_{\ensuremath{K}}) = \Theta(\log^2 \ensuremath{K} )\,.
\end{align}
\end{corollary}

\vspace{-0.2pc}
The constant-factor gap exists because $P^* \in \cP^{\triangle}_{\ensuremath{K}}$ is a stronger constraint than $p^*_{\ensuremath{K}} \in \cP_{1/\ensuremath{K}}$.

For any $\ensuremath{K}$, $c_{\ensuremath{K}}$ can be approximated numerically.
To simplify the quantizer, we can use $c_{\ensuremath{K}} \approx \frac{1}{2}$ for large $\ensuremath{K}$ to get the \emph{approximate minimax compander}:
\begin{align} \label{eq::appx-minimax-compander}
\ensuremath{f}^{**}_\ensuremath{K}(x) = \frac{\ensuremath{\mathrm{ArcSinh}}(\sqrt{(1/2) (\ensuremath{K} \log \ensuremath{K}) \, x})}{\ensuremath{\mathrm{ArcSinh}}(\sqrt{(1/2) \ensuremath{K} \log \ensuremath{K}})}\,.
\end{align}
This is close to optimal without needing to compute $c_\ensuremath{K}$:

\begin{theorem} \label{thm::approximate-minimax-compander}
Suppose that $\ensuremath{K}$ is sufficiently large so that $c_\ensuremath{K} \in [\frac{1}{2 (1 + \varepsilon)}, \frac{1 + \varepsilon}{2}]$. Then for any $p \in \cP$,
\begin{align}
    \ensuremath{\widetilde{L}}(p,\ensuremath{f}^{**}_\ensuremath{K}) \leq (1+ \varepsilon) \ensuremath{\widetilde{L}}(p,\ensuremath{f}^*_\ensuremath{K})\,.
\end{align}
\end{theorem}

\begin{remark} \label{rmk::minimax-is-closed-form}
While $\ensuremath{f}^*_\ensuremath{K}$ and $\ensuremath{f}^{**}_\ensuremath{K}$ might appear complicated,
$
    \ensuremath{\mathrm{ArcSinh}}(\sqrt{z}) = \log(\sqrt{z} + \sqrt{z+1})
$
is fairly simple.
Taking the Taylor expansion also confirms that they are in $\ensuremath{\cF}^\dagger$.

Note that \eqref{eq::best_f_raw} (\Cref{thm::optimal_compander_loss}) suggests that the natural form of an optimal compander against $p$ is a normalized incomplete integral,
which is hard to use. Thus, the closed-form expressions of $\ensuremath{f}^*_\ensuremath{K}$ and $\ensuremath{f}^{**}_\ensuremath{K}$ is a welcome surprise.
\end{remark}

Using the minimax compander $\ensuremath{f}^*_{\ensuremath{K}}$ or approximate minimax compander $\ensuremath{f}^{**}_{\ensuremath{K}}$ on $P \in \cP^\triangle_{\ensuremath{K}}$ with granularity $N$, we have a bound on the average KL divergence:
\begin{align}\label{eq::compander_achieve}
    \bbE_{\bX \sim P}[D_{\kl}(\bX \| \bY)] = O\left(N^{-2} \log^2 \ensuremath{K} \right)\,.
\end{align}

\begin{remark}
If we use the uniform quantizer, by contrast, there exists a $P \in \cP^\triangle_{\ensuremath{K}}$ where we have
\begin{align}\label{eq::uniform_achieve}
    \bbE_{\bX \sim P}[D_{\kl}(\bX \| \bY)] = \Theta\left(\ensuremath{K}^2 N^{-2} \log N \right)\,.
\end{align}
The dependence on $N$ is greater than $N^{-2}$ (thus yielding $\widetilde{L}(p,f) = \infty$) and the dependence on $\ensuremath{K}$ is quadratic. This shows that, generally speaking, the uniform quantizer is very bad since it risks increasingly high loss relative to more suitable companders; specifically, by \Cref{thm::asymptotic-normalized-expdiv}, any $\ensuremath{f} \in \cF^\dagger$ is guaranteed to have loss $\propto \, N^{-2}$ for all priors.

Additionally, \eqref{eq::uniform_achieve} is by no means the worst possible for the uniform quantizer; it is just sufficient to show how it may perform much, much worse than well-chosen companders.
\end{remark}

\begin{remark}
Instead of the KL divergence loss on the simplex, we can do a similar analysis to find the minimax
compander for $L_2^2$ loss on the unit hypercube. It turns out, the solution
is given by the identity function $\ensuremath{f}(x)=x$ corresponding to the standard (non-companded) uniform quantization. (See \Cref{sec::other_losses}.)
\end{remark}

The above are all `average case' results, where $\bX$ is drawn from a prior $P$ (which is fixed as $N \to \infty$). In the worst-case problem, $\bx$ is chosen to maximize loss and can depend on $N$ (decoding is midpoint by default since there is no prior and hence no well-defined centroid):

\begin{theorem}
\label{thm::worstcase_power_minimax}
The minimax compander with midpoint decoding achieves \emph{worst-case loss}
\vspace{-0.2pc}
\begin{align}
    \max_{\bx \in \triangle_{\ensuremath{K}-1}}D_{\kl}(\bx\|\by) = O\left( N^{-2} \log^2 \ensuremath{K} \right)\,.
\end{align}
\end{theorem}

\begin{remark}
When $b$ is the number of bits used to quantize each value in the probability vector, using the approximate minimax compander yields a worst-case loss  on the order of $2^{-2b} \log ^2 \ensuremath{K}$. Using
optimal vector quantization (as explored in \cite{phdthesis}), the loss is an order between $2^{-2 b\frac{\ensuremath{K}}{\ensuremath{K} - 1}}$ and  $2^{-2 b\frac{\ensuremath{K}}{\ensuremath{K} - 1}} \log \ensuremath{K}$.
Thus, our result using companders is within a factor $2^{2b/(\ensuremath{K} - 1)}\log^2 \ensuremath{K}$ of the optimal loss. (The bound $2^{-2 b\frac{\ensuremath{K}}{\ensuremath{K} - 1}} \log \ensuremath{K}$ is not associated with an explicit quantization scheme. One is only shown to exist.)
\end{remark}

\iflong
\else
Due to space constraints, we omit the proofs of \Cref{rmk::best-constant,thm::worstcase_power_minimax}. We sketch the other proofs in \Cref{sec::asymt_single,sec::minimax}.
\fi

\subsection{Experimental Results}

\label{sec::experimental_results}

We compare the performance of five quantizers, with granularities $N = 2^8$ and $N = 2^{16}$, on three types of datasets: (i) random synthetic distributions drawn from the uniform prior over the simplex;\footnote{We draw 1000 random samples and average over these for our results.} (ii) frequency of words in books;\footnote{These frequencies are computed from text available on the Natural Language Toolkit (NLTK) libraries for Python. For each text, we get tokens (single words or punctuation) from each text and simply count the occurrence of each token} and (iii) frequency of $k$-mers in DNA.\footnote{For a given sequence of DNA, the set of $k$-mers are the set of length $k$ substrings which appear in the sequence. We use the human genome as the source for our DNA sequences. Parts of the sequence marked as repeats are removed.} Our quantizers are:
\begin{itemize}
    \item \textbf{Approximate Minimax Compander:} As given by \eqref{eq::appx-minimax-compander}. Using the approximate minimax compander is much simpler than the minimax compander since the constant $c_\ensuremath{K}$ does not need to be computed, and by \Cref{thm::approximate-minimax-compander} it has almost identical performance for large $\ensuremath{K}$.
    \item \textbf{Truncation:} Uniform quantization (equivalent to $\ensuremath{f}(x) = x$), which truncates the least significant bits. This is the natural way of quantizing values in $[0,1]$.
    \item \textbf{Float and bfloat16:} For 8-bit encodings ($N = 2^8$), we use a floating point implementation which allocates 4 bits to the exponent and 4 bits to the mantissa. For 16-bit encodings ($N = 2^{16}$), we use bfloat16, a standard which is commonly used in machine learning~\cite{kalamkar2019study}.
    \item \textbf{Exponential Density Interval (EDI):} This is the quantization method we used in an achievability proof in  \cite{adler_ratedistortion_2021}. It is designed for the uniform prior over the simplex.
    \item \textbf{Power Compander:} The compander is $ \ensuremath{f}(x) = x^{s}$, a natural class of functions from $[0,1]$ to $[0,1]$. We optimize $s$ and find that $s = \frac{1}{\log_e \ensuremath{K}}$ asymptotically minimizes KL divergence, and also gives close to the best performance empirically. To see the effects of different powers $s$ on the performance of the power compander, see \Cref{fig::books_power}.
\end{itemize}

Because a well-defined prior doesn't always exist for these datasets (and for simplicity) we use midpoint decoding for all the companders. When a probability value of exactly $0$ appears, we do not use companding and instead quantize the value to $0$, i.e. the value $0$ has its own bin.

\begin{figure}
    \centering
    \includegraphics[scale = .6, trim = {25 0 0 0}
    ]{figures/all_books_frequencies_256.pdf}
    \caption{Power compander $\ensuremath{f}(x) = x^s$ performance with different powers $s$ used to quantize frequency of words in books. Number of distinct words in each book is shown in the legend. The theoretical optimal power $s = \frac{1}{\log \ensuremath{K}}$ is plotted where $\ensuremath{K}$ is the number of distinct words.}
    \label{fig::books_power}
\end{figure}

Our main experimental results are given in \Cref{fig::first_compare_compander}, showing the KL divergence between the original distribution $\bx$ and its quantized version $\by$ versus alphabet size $\ensuremath{K}$. The approximate minimax compander performs well against all sources.
\begin{figure}
    \centering
    \includegraphics[scale = .6]{figures/kl_ucsc_dna_books_dir_1_truncate_exp_power_sinh_float_256_fig.pdf}
    \includegraphics[scale = .6]{figures/kl_ucsc_dna_books_dir_1_truncate_exp_power_sinh_bfloat16_65536_fig.pdf}
    \caption{Plot comparing the performance of the truncation compander, the EDI compander, floating points, the power compander, and the approximate minimax compander \eqref{eq::appx-minimax-compander} on probability distributions of various sizes.
    }
    \label{fig::first_compare_compander}
    \vspace{-1pc}
\end{figure}
For truncation, the KL divergence increases with $\ensuremath{K}$ and is generally fairly large.
The EDI quantizer works well for the synthetic uniform prior (as it should), but for real-world datasets like word frequency in books, it performs badly (sometimes even worse than truncation).
The power compander performs similarly to the minimax compander and is worse only by a constant.\footnote{\Cref{thm::worstcase_power_minimax} also holds for the power compander with different constants.}

The experiments demonstrate that the approximate minimax compander achieves low loss on the entire ensemble of data (even for relatively small granularity, such as $N = 256$) and outperforms both truncation and floating-point implementations on the same number of bits. Additionally, its closed-form expression (and entrywise application) makes it simple to implement and computationally inexpensive. Thus it can be easily added to existing systems to lower storage requirements at little or no cost to fidelity.

\subsection{Paper Organization}
We provide background and discuss previous work on companders in \Cref{sec::previous_works}.
We prove \Cref{thm::asymptotic-normalized-expdiv} in \Cref{sec::asymt_single} (though proofs of some lemmas and propositions leading up to it are given in \Cref{sec::appendix_proofs_asymptotic}).
In \Cref{sec::minimax}, we optimize over  \eqref{eq::raw_loss} to get the maximin single-letter distribution and minimax compander, thus showing \Cref{thm::optimal_compander_loss,thm::minimax_compander} and \Cref{cor::worstcase_prior} (leaving \Cref{thm::approximate-minimax-compander} for \Cref{sec::proof_L_appx_minimax_compander}).
We prove \Cref{thm::worstcase_power_minimax} in \Cref{sec::worst-case_analysis}. In \Cref{sec::other_losses} we discuss companders for losses other than KL divergence.

\section{Background}

\label{sec::previous_works}

Companders (also spelled ``compandors'') were introduced by Bennett in 1948 \cite{bennett1948} as a way to quantize speech signals, where it is advantageous to give finer quantization levels to weaker signals and coarser levels to larger signals. Bennett gives a first order approximation that the mean-square error in this system is given by
\begin{align}\label{eq::bennett_compander}
    \frac{1}{12 N^2} \int_{a}^b \frac{p(x)}{(f'(x))^2} dx
\end{align}
where $N$ is the number quantization levels, $a$ and $b$ are the minimum and maximum values of the input signal, $p$ is the probability density of the input signal,
and $f'$ is the slope of the compressor function placed before the uniform quantization.
This formula is similar to our \eqref{eq::raw_loss} except that we have an extra $x^{-1}$ since we are working with KL divergence.
Others have expanded on this line of work. In \cite{panter_dite}, the authors studied the same problem and determined the optimal compressor under mean-square error, a result which parallels our result \eqref{eq::raw_overall_dist}. However, results like those in \cite{bennett1948, panter_dite} are stated either as first order approximations or make simplfying assumptions. For example, in \cite{panter_dite}, the authors state that they assume the values $\widehat{y}_{(n)}$ are close together enough that probabilities over each bin are approximately uniform. Their results proceed from this assumption.
Our work rigorously addresses these assumptions, showing they hold under very general conditions.

Generalizations of Bennett's formula are also studied when instead of mean-square error, the loss is the expected $r$th moment loss $\bbE\norm{\cdot}^r$. This is computed for vectors of length $\ensuremath{K}$ in \cite{zador1982} and \cite{gersho1979}.
The typical examples of companders used in engineering are $\mu$-law and $A$-law companders \cite{lewis_mu-law}. For the $\mu$-law compander, \cite{panter_dite} and \cite{smith1957} argue that for a large enough constant $\mu$, in the case of mean-squared error, the distortion becomes independent of the signal.

Quantizing probability distributions is a common topic, though typically the loss function is a norm and not KL divergence \cite{graf2007}. Quantizing for KL divergence is considered in our earlier work \cite{adler_ratedistortion_2021}, where we focus on average KL loss for Dirichlet priors.

A similar problem to quantizing under KL divergence is \emph{information $k$-means}. This is the problem of clustering $n$ points $a_i$ to $k$ centers $\hat{a}_j$ to minimize the KL divergences between the points and their associated centers. Theoretical aspects of this are explored in \cite{slonim1999} and \cite{tishby2000}.
Information $k$-means has been implemented for several different applications \cite{Pereira93, jiang2013, cao2013}. There are also other works that study clustering with a slightly different but related metric \cite{dhillon2003,nielsen2013, veldhuis2002}; however, the focus of these works is to analyze data rather than reduce storage.

\section{Asymptotic Single-Letter Loss} \label{sec::asymt_single}

In this section we give the proof of \Cref{thm::asymptotic-normalized-expdiv} (though the proofs of some lemmas must be sketched). We use the following notation:
\begin{itemize}
\item Given an interval $I$ we define $\bar{y}_I$ to be its midpoint and $r_I$ to be its width, so that by definition
\begin{align}
I = [\bar{y}_I - r_I/2, \bar{y}_I + r_I/2]\,.
\end{align}
Note that if $I \subseteq [0,1]$ then $r_I \leq 2 \bar{y}_I$.
\item Given probability distribution $p$ and interval $I$, $p|_I$ denotes $p$ restricted to $I$, i.e. $X \sim p|_I$ is the same as $X \sim p$ conditioned on $X \in I$.  We also define the probability mass of $I$ under $p$ as $\pi_{p,I} = \bbP_{X \sim p}[ X \in I]$. If $\pi_{p,I} = 0$, we let $p|_I$ be uniform on $I$ by default.
\item Given probability distribution $p$ and interval $I$, we denote the \emph{centroid of $I$ under $p$} as
\begin{align}
\widetilde{y}_{p,I} := \bbE_{X \sim p|_I} [X] = \bbE_{X \sim p}[X \, | \, X \in I]\,.
\end{align}
If this is undefined because $\bbP_{X \sim p}[X \in I] = 0$ then by the convention on $p|_I$, we have $\widetilde{y}_{p, I} = \bar{y}_I$ (the centroid under a uniform distribution is the midpoint).
\item Given two probability distributions $p, q$ (over the same domain), we define their \emph{Kolmogorov-Smirnov distance} (KS distance) to be
\begin{align}
d_{KS}(p,q) = \norm{F_p - F_q}_{\infty} = \sup_x |F_p(x) - F_q(x)| \label{eq::infty-norm}
\end{align}
(recall that $F_p, F_q$ are the CDFs of $p,q$).
\item We use standard order-of-growth notation (which are also used in \Cref{sec::main-theorems}). We review these definitions here for clarity, especially as we will use some of the rarer concepts (in particular, small-$\omega$). For a parameter $t$ and functions $a(t), b(t)$, we say:
\begin{align}
    a(t) = O(b(t)) &\iff \limsup_{t \to \infty} |a(t)/b(t)| < \infty
    \\ a(t) = \Omega(b(t)) &\iff \liminf_{t \to \infty} |a(t)/b(t)| > 0
    \\ a(t) = \Theta(b(t)) &\iff a(t) = O(b(t)), \, a(t) = \Omega(b(t))\,.
\end{align}
We use small-$o$ notation to denote the strict versions of these:
\begin{align}
    a(t) = o(b(t)) &\iff \lim_{t \to \infty} |a(t)/b(t)| = 0
    \\  a(t) = \omega(b(t)) &\iff \lim_{t \to \infty} |a(t)/b(t)| = \infty\,.
\end{align}
Sometimes we will want to indicate order-of-growth as $t \to 0$ instead of $t \to \infty$; this will be explicitly mentioned in that case.
\end{itemize}
When $I = I^{(n)}$ is a bin of the compander, we can replace it with $(n)$ in the notation, i.e. $\bar{y}_{(n)} = \bar{y}_{I^{(n)}}$ (so the midpoint of the bin containing $x$ at granularity $N$ is denoted $\bar{y}_{(n_N(x))}$ and the width of the bin is $r_{(n_N(x))}$). When $I$ and/or $p$ are fixed, we will sometimes drop them from the notation, i.e. $\widetilde{y}_I$ or even just $\widetilde{y}$ to denote the centroid of $I$ under $p$.

\subsection{The Local Loss Function}

One key to the proof is the following perspective: instead of considering $X \sim p$ directly, we (equivalently) first select bin $I^{(n)}$ with probability $\pi_{p, (n)}$, and then select $X \sim p|_{(n)}$. The expected loss can then be considered within bin $I^{(n)}$. This makes it useful to define:
\begin{definition}
Given probability measure $p$ and interval $I \subseteq [0,1]$, the \emph{single-interval loss of $I$ under $p$} is
\begin{align}
\ell_{p,I} = \bbE_{X \sim p|_{I}}[X \log(X/\widetilde{y}_{p,I})]\,.
\end{align}
\end{definition}
As before, if $p$ and/or $I$ is fixed and clear, we can drop it from the notation (and if $I = I^{(n)}$ is a bin, we can denote the local loss as $\ell_{p, (n)}$).
This can be interpreted as follows: if we quantize all $x \in I$ to the centroid $\widetilde{y}_I$, then $\ell_{p,I}$ is the expected loss of $X \sim p$ conditioned on $X \in I$. Thus the values of $\ell_{p,(n)}$ can be used as an alternate means of computing the single-letter loss:
\begin{align}
	\ensuremath{\widetilde{L}}(p,f,N) &= \bbE_{X \sim p} [X \log(X/\widetilde{y}(X))]
	\\ &=  \sum_{n=1}^N \pi_{p, (n)} \bbE_{X \sim p|_{(n)}}[X \log(X/\widetilde{y}_{p,(n)})]
	\\ &=  \sum_{n=1}^N \pi_{p, (n)} \ell_{p, (n)}
	\\ &= \int_{[0,1]}  \ell_{p, (n_N(x))} \, dp\,.
\end{align}
Thus the normalized single-letter loss (whose limit is the asymptotic single-letter loss \eqref{eq::raw-assl}) is
\begin{align}
N^2 \, \ensuremath{\widetilde{L}}(p,f,N) = \int_{[0,1]} N^2 \, \ell_{p, (n_N(x))} \, dp\,.
\end{align}
For single-letter density $p$ and compander $f$, we define the \emph{local loss function at granularity $N$}:
\begin{align} \label{eq::loc-loss-fn}
    g_N(x) = N^2 \, \ell_{p, (n_N(x))}\,.
\end{align}
We also define the \emph{asymptotic local loss function}:
\begin{align}
    g(x) = \frac{1}{24} f'(x)^{-2} x^{-1} \,.
\end{align}
\Cref{thm::asymptotic-normalized-expdiv} is therefore equivalent to:
\begin{align}
&\liminf_{N \to \infty} \int g_N \, dp \geq \int g \, dp ~\text{ for all }~ p \in \cP, f \in \cF \label{eq::fatou-restated}
\\ \text{and }~ &\lim_{N \to \infty} \int g_N \, dp = \int g \, dp ~\text{ for all }~ p \in \cP, f \in \cF^\dagger \label{eq::raw-restated}\,.
\end{align}
To prove \eqref{eq::fatou-restated} and \eqref{eq::raw-restated}, we show:

\begin{proposition} \label{prop::locloss-convergence}
For all $p \in \cP$, $\ensuremath{f} \in \ensuremath{\cF}$, if $X \sim p$ then
\begin{align}
    \lim_{N \to \infty} \ensuremath{g}_N(X) = \ensuremath{g}(X) ~~~\text{almost surely.}
\end{align}
\end{proposition}

\begin{proposition} \label{prop::locloss-dominating}
Let $f$ be a compander and $c > 0$ and $\alpha \in (0,1]$ such that $f(x) - c x^\alpha$ is monotonically increasing. Letting $g_N$ be the local loss functions as in \eqref{eq::loc-loss-fn} and
\begin{align}
     h(x) = (2^{2/\alpha} + \alpha^2 2^{1/\alpha - 2}) (c \alpha)^{-2} x^{1-2\alpha} +  c^{-1/\alpha} 2^{1/\alpha - 2}
\end{align}
then $g_N(x) \leq h(x)$ for all $x, N$. Additionally, if $\alpha \leq 1/2$ then $\int_{[0,1]} h \, dp < \infty$.
\end{proposition}

The lower bound \eqref{eq::fatou-restated} then follows immediately from \Cref{prop::locloss-convergence} and Fatou's Lemma; and when $f \in \cF^\dagger$, by \Cref{prop::locloss-dominating} there is some $h$ which is integrable over $p$ and dominates all $g_N$, thus showing \eqref{eq::raw-restated} by the Dominated Convergence Theorem.

To prove \Cref{prop::locloss-convergence}, we use the following facts:
\begin{itemize}
\item For any $x$ at which $f$ is differentiable, when $N$ is large
\begin{align}
r_{(n_N(x))} \approx N^{-1} f'(x)^{-1} \, .
\end{align}
\item For any $x$ at which $F_p$ is differentiable, $p|_I$ will be approximately uniform over any sufficiently small $I$ containing $x$.
\item For a sufficently small interval $I$ containing $x$ and such that $p|_I$ approximately uniform,
\begin{align}\label{eq::local_loss_approx}
\ell_{p,I} \approx \frac{1}{24} r_I^2 x^{-1}\,.
\end{align}
\end{itemize}
Putting these together, we get that if $F_p$ and $f$ are both differentiable at $x$ then when $N$ is large,
\begin{align}
	g_N(x) &= N^2 \, \ell_{p,(n_N(x))} \\
	&\approx N^2 \frac{1}{24} r_{(n_N(x))}^2 x^{-1} \approx \frac{1}{24} f'(x)^{-2} x^{-1} = g(x)
\end{align}
as we wanted.  We formally state each of these steps in \Cref{sec::locloss-convergence-preliminaries} (due to space constraints, most proofs are in the appendix), and combine them to prove \Cref{prop::locloss-convergence} in \Cref{sec::locloss-convergence-pf}.

The proof of \Cref{prop::locloss-dominating} is given in \Cref{sec::locloss-dominating-proof}, along with its own set of definitions and lemmas needed to show it.

\subsection{Preliminaries for \Cref{prop::locloss-convergence}} \label{sec::locloss-convergence-preliminaries}

We first generalize the idea of \emph{bins} \eqref{eq::bins}. The bin around $x \in [0,1]$ at granularity $N$ is the interval $I = I^{(n)}$ containing $x$ such that $f(I) = [(n-1)/N, n/N]$ for some $n \in [N]$. This notion relies on integers because $f(I) = [(n-1)/N, n/N]$ for integers $n,N$. We remove the dependence on integers while keeping the basic structure (an interval $I$ about $x$ whose image $f(I)$ is a given size):
\begin{definition} \label{def::pseudobin}
For any $x \in [0,1]$, $\theta \in [0,1]$, and $\varepsilon > 0$, we define the \emph{pseudo-bin} $I^{(x, \theta, \varepsilon)}$ as the interval satisfying:
\begin{align}
    \hspace{-1pc} I^{(x,\theta,\varepsilon)} &= [x - \theta r^{(x, \theta, \varepsilon)}, x + (1-\theta) r^{(x, \theta, \varepsilon)}] \text{ where }
    \\ \hspace{-0.85pc} r^{(x, \theta, \varepsilon)} &= \inf \big(r : f(x + (1-\theta) r) - f(x - \theta r) \geq \varepsilon \big) \label{eq::defn-of-r}
\end{align}
\end{definition}
The interpretation of this is that $I^{(x,\theta,\varepsilon)}$ is the minimal interval $x$ such that $|f(I^{(x,\theta,\varepsilon)})| \geq \varepsilon$ and such that $x$ occurs at $\theta$ within $I^{(x,\theta,\varepsilon)}$, i.e. a $\theta$ fraction of $I^{(x,\theta,\varepsilon)}$ falls below $x$ and $1 - \theta$ falls above. Its width is $r^{(x,\theta,\varepsilon)}$. This implies that bins are a special type of pseudo-bins. Specifically, for any $x$ and $N$ (and any compander $f$),
\begin{align}
    I^{(n_N(x))} = I^{(x,\theta,1/N)} \text{ for some } \theta \in [0,1] \,.
\end{align}
We now consider the size of pseudo-bins as $\varepsilon \to 0$:
\begin{lemma} \label{lem::asymptotic-interval-sizes}
If $f$ is differentiable at $x$, then
\begin{align}
    \lim_{\varepsilon \to 0} \varepsilon^{-1} r^{(x,\theta,\varepsilon)} = f'(x)^{-1}
\end{align}
(including going to $\infty$ when $f'(x) = 0$). The limit converges uniformly over $\theta \in [0,1]$.
\end{lemma}

The proof is given in \Cref{sec::proof_asymptotic-interval-sizes}.
Note that applying this to bins means $\lim_{N \to \infty} N r^{(n_N(x))} = f'(x)^{-1}$, and hence when $f'(x) > 0$ we have $r^{(n_N(x))} = N^{-1} f'(x)^{-1} + o(N^{-1})$.

For any interval $I$, we want to measure how close $p$ is to uniform over $I$ using the distance measure $d_{KS}(p,q)$ from \eqref{eq::infty-norm}. We will show that when $F'_p(x) = p(x)$ is well-defined and positive at $x$, $p$ is approximately uniform on any sufficiently small interval $I$ around $x$. Formally:
\begin{proposition} \label{prop::uniform-at-small-scales}
If $p(x) = F'_p(x) > 0$ is well-defined, then for every $\varepsilon > 0$ there is a $\delta > 0$ such that for all intervals $I$ such that $x \in I$ and $r_I \leq \delta$,
\begin{align}
d_{KS}(p|_I, \unif_I) \leq \varepsilon \, .
\end{align}
\end{proposition}

We give the proof in \Cref{sec::proof_uniform-at-small-scales}. This allows us to use the following:
\begin{proposition} \label{prop::divergence-of-close-distributions}
Let $p$ be a probability measure and $I$ be an interval containing $x$ such that $r_I \leq x/4$ and $d_{KS}(p|_I,\unif_I) \leq \varepsilon$ where $\varepsilon \leq 1/2$. Then
\begin{align}
| \ell_{p,I} - \ell_{\unif_I} | \leq 2 \varepsilon r_I^2 x^{-1} + O(r_I^3 x^{-2})\,.
\end{align}
\end{proposition}

Recall that $\ell_{p,I}$ is the interval loss of $I$ under distribution $p$ when all points in $I$ are quantized to $\widetilde{y}_{p,I}$, the centroid of the interval.
We give the proof of \Cref{prop::divergence-of-close-distributions} in \Cref{sec::proof_divergence-of-close-distributions}.

\begin{proposition} \label{prop::loss-under-uniform-dist}
For any $x > 0$ and any sequence of intervals $I_1, I_2, \dots \subseteq [0,1]$ all containing $x$ such that $r_{I_i} \to 0$ as $i \to \infty$,
\begin{align}
    \ell_{\unif_{I_i}} = \frac{1}{24} r_{I_i}^2 x^{-1} + O(r_{I_i}^3 x^{-2}) \,.
\end{align}
\end{proposition}

The proof is in \Cref{sec::proof_loss-under-uniform-dist}.

Note that the above lemmas are all about asymptotic behavior as intervals shrink to $0$ in width; to deal with the (edge) case where they don't, we need the following lemma:
\begin{lemma} \label{lem::increase-interval-loss-positive}
For any $I$ such that $\bbP_{X \sim p} [X \in I] > 0$, there is some $a_I > 0$ such that
\begin{align}
    \ell_{p,J} \geq a_I \text{ for any } J \supseteq I \,.
\end{align}
\end{lemma}
We give the proof in
\Cref{sec::additional_lemmas}.

\subsection{Proof of \Cref{prop::locloss-convergence}}

\label{sec::locloss-convergence-pf}

We now combine the above results to prove \Cref{prop::locloss-convergence}, i.e. that $\lim_{N \to \infty} g_N(X) = g(X)$ almost surely when $X \sim p$. Because $p \in \cP$ (i.e. it is a continuous probability distribution) we will treat the bins as closed sets, i.e. $I^{(n)} = [f(\frac{n-1}{N})^{-1}, f(\frac{n}{N})^{-1}]$; this does not affect anything since the resulting overlap is only a finite set of points.

\begin{proof}

Since $p \in \cP$ then when $X \sim p$ the following hold with probability $1$:
\begin{enumerate}
    \item $0 < X < 1$;
    \item $f'(X)$ is well-defined;
    \item $p(X) = F'_p(X)$ is well defined;
    \item $p(X) > 0$.
\end{enumerate}
This is because if $p \in \cP$, and $|S|$ denotes the Lebesgue measure of set $S$, then
\begin{align}
    |S| = 0 \implies \bbP_{X \sim p} [X \in S] = 0
\end{align}
This immediately implies 1) since $\{0,1\}$ is measure-$0$.

Additionally, by Lebesgue's differentiation theorem for monotone functions, any monotonic function on $[0,1]$ is differentiable almost everywhere on $[0,1]$ (i.e. excluding at most a measure-$0$ set), and compander $f$ and CDF $F_p$ are monotonic. This implies 2) and 3). Finally, 4) follows because the set of $X$ such that $p(X) = 0$ has probability $0$ under $p$ by definition.

Therefore, we can fix $X \sim p$ and assume it satisfies the above properties.

We now consider the bin size $r_{(n_N(X))}$ as $N \to \infty$; there are two cases, (a) $\lim_{N \to \infty} r_{(n_N(X))} = 0$ and (b) $\limsup_{N \to \infty} r_{(n_N(X))} > 0$. For case (b), since the length of the interval does not go to zero, $g_N(X) = N^2 \ell_{p,(n_N(X))} \to \infty$; additionally, $g(X) = \infty$ by default since case (b) requires that $f'(X) = 0$, and so $g_N(X) \to g(X)$ as we want.

\emph{Case (a):} In this case (which holds for all $X$ if $f \in \cF^\dagger$), any $\delta > 0$ there is some sufficiently large $N^*$ (which can depend on $X$) such that
\begin{align}
    N \geq N^*\implies r_{(n_N(X))} \leq \delta \, .
\end{align}
By \Cref{prop::uniform-at-small-scales}, for any $\varepsilon > 0$ there is some $\delta > 0$ such that for all intervals $I$ where $X \in I$ and $r_I \leq \delta$, we have $d_{KS}(p|_I, \unif_I) \leq \varepsilon$. Putting this together implies that for any $\varepsilon > 0$, there is some sufficiently large $N^*_\varepsilon$ such that for all $N \geq N^*_\varepsilon$,
\begin{align}
    d_{KS}(p|_{(n_N(X))}, \unif_{(n_N(X))}) \leq \varepsilon \, .
\end{align}
i.e. $p$ is $\varepsilon$ close to uniform on $I^{(n_N(X))}$. Furthermore, we can always choose $\varepsilon \leq 1/2$ and $N^*_\varepsilon$ sufficiently large that $r_{(n_N(X))} \leq X/4$ (since $\lim_{N \to \infty} r_{(n_N(X))} = 0$). Under these conditions, for $N > N^*_\varepsilon$ we can apply \Cref{prop::divergence-of-close-distributions} and get
\begin{align}
    |\ell_{p,(n_N(X))} - \ell_{\unif_{(n_N(X))}}| \leq 2 \varepsilon r_{(n_N(X))}^2 X^{-1} + O(r_{(n_N(X))}^3 X^{-2}) \, .
\end{align}
We can then turn this around: as $N \to \infty$, we have $\varepsilon \to 0$ and hence $\varepsilon = o(1)$ (as $N \to \infty$), so
\begin{align} \label{eq::p-close-to-unif}
    |\ell_{p,(n_N(X))} - \ell_{\unif_{(n_N(X))}}| = o(r_{(n_N(X))}^2 X^{-1}) \, .
\end{align}

We then apply \Cref{prop::loss-under-uniform-dist} (note that since $r_{(n_N(X))} \leq X/4$ and $X \leq 2\bar{y}_{(n_N(X))}$, we know automatically that $r_{(n_N(X))} \leq \bar{y}_{(n_N(X))}/2$) to get that
\begin{align}
    \ell_{\unif_{(n_N(X))}} = \frac{1}{24} r_{(n_N(X))}^2 \bar{y}_{(n_N(X))}^{-1} + O(r_{(n_N(X))}^3 X^{-2})
\end{align}
However, since $X$ is fixed and $r_{(n_N(X))} \to 0$ as $N \to 0$ (and $|X - \bar{y}_{(n_N(X))}| \leq r_{(n_N(X))}$ since they are both in the bin $I^{(n_N(X))}$), we know that $\bar{y}_{(n_N(X))} = X (1 + o(1))$ where $o(1)$ is in terms of $N$ (as $N \to \infty$). Hence (noting that $(1 + o(1))^{-1}$ is still $1 + o(1)$ and $O(r_{(n_N(X))}^3 X^{-2})$ is $o(1) r_{(n_N(X))}^2 X^{-1}$) we can re-write the above and combine with \eqref{eq::p-close-to-unif} to get
\begin{align}
    \ell_{\unif_{(n_N(X))}} &= \frac{1}{24} (1 + o(1)) r_{(n_N(X))}^2 X^{-1}
    \\ \implies \ell_{p, (n_N(X))} &= \frac{1}{24} (1 + o(1)) r_{(n_N(X))}^2 X^{-1} \, .
\end{align}
We now split things into two cases: (i) $f'(X) > 0$; (ii) $f'(X) = 0$.

\emph{Case i ($f'(X) > 0$):} For all $N$ there is a $\theta \in [0,1]$ such that $I^{(n_N(X))} = I^{(X,\theta,1/N)}$ (bins are pseudo-bins, see \Cref{def::pseudobin}).
Thus, by \Cref{lem::asymptotic-interval-sizes} (which shows uniform convergence over $\theta$),
\begin{align}
    \lim_{N \to \infty} N r_{(n_N(X))} = f'(X)^{-1}
\end{align}
Thus, we may re-write as a little-$o$ and plug into $g_N(X)$:
\begin{align}
    r_{(n_N(X))} &= N^{-1} f'(X)^{-1} + o(N^{-1})
    \\ &= N^{-1} f'(X)^{-1} (1 + o(1))
    \\ \implies g_N(X) &= N^2 \ell_{p, (n_N(X))} \label{eq::gn-to-g-01}
    \\ &= N^2 \frac{1}{24} (1 + o(1)) r_{(n_N(X))}^2 X^{-1} \label{eq::gn-to-g-02}
    \\ &= N^2 \frac{1}{24} (1 + o(1)) N^{-2} f'(X)^{-2} X^{-1} \label{eq::gn-to-g-03}
    \\ &= \frac{1}{24} (1 + o(1)) f'(X)^{-2} X^{-1} \label{eq::gn-to-g-04}
\end{align}
implying $\lim_{N \to \infty} g_N(X) = g(X)$ as we wanted.

\emph{Case ii ($f'(X) = 0$):} As before, for any $N$ there is some $\theta \in [0,1]$ such that $I^{(n_N(X))} = I^{(X,\theta,1/N)}$. Thus, by \Cref{lem::asymptotic-interval-sizes} and as $f'(X) = 0$, we have
\begin{align}
    \lim_{N \to \infty} N r_{(n_N(X))} = \infty \, .
\end{align}
since the convergence in \Cref{lem::asymptotic-interval-sizes} is uniform over $\theta$. We can then re-write this as a little-$\omega$:
\begin{align}
    r_{(n_N(X))} = \omega(N^{-1}) \, .
\end{align}
This implies that
\begin{align}
    g_N(X) &= N^2 \ell_{p, (n_N(X))}
    \\ &= N^2 \frac{1}{24} (1 + o(1)) r_{(n_N(X))}^2 X^{-1}
    \\ &= N^2 \frac{1}{24} (1 + o(1)) \omega(N^{-2}) X^{-1}
    \\ &= \omega(1)
\end{align}
where $\omega(1)$ means $\lim_{N \to \infty} g_N(X) = \infty$. But since $f'(X) = 0$, by convention we have $g(X) = \frac{1}{24} f'(X)^{-2} X^{-1} = \infty$ and so $\lim_{N \to \infty} g_N(X) = g(X)$ as we wanted.

\emph{Case (b):} $\limsup_{N \to \infty} r_{(n_N(X))} > 0$. Note that this can only happen if $f'(X) = 0$, so $g(X) = \infty$; hence our goal is to show that $\lim_{N \to \infty} g_N(X) = \infty$.

Related to the above, this only happens if $f$ is not strictly monotonic at $X$, i.e. if there is some $a < X$ or some $b > X$ such that $f(X) = f(a)$ or $f(X) = b$ (or both). If both, $[a,b] \subseteq I^{(n_N(X))}$ for all $N$. Since $p(X)$ is well-defined and positive, any nonzero-width interval containing $X$ has positive probability mass under $p$. Thus, by \Cref{lem::increase-interval-loss-positive}, there exists some $\alpha > 0$ such that all $J \supseteq [a,b]$ satisfies $\ell_{p,J} \geq \alpha$. But then $g_N(X) \geq N^2 \alpha$ and goes to $\infty$.

If only $a$ exists, we divide the granularities $N$ into two classes: first, $N$ such that $I^{(n_N(X))}$ has lower boundary exactly at $X$ (which can happen if $f(X)$ is rational), and second, $N$ such that $I^{(n_N(X))}$ has lower boundary below $X$. Call the first class $N^{(1)}(1), N^{(1)}(2), \dots$ and the second $N^{(2)}(1), N^{(2)}(2), \dots$. Then as no $b$ exists, $\lim_{i \to \infty} r^{(n_{N^{(1)}(i)}(X))} = 0$, i.e. the bins corresponding to the first class shrink to $0$ and the asymptotic argument applies to them, showing $g_{N^{(1)}(i)}(X) \to \infty$. For the second class, for any $i$, we have $I^{(n_{N^{(2)}(i)}(X))} \supseteq [a,X]$ and so we have an $\alpha > 0$ lower bound of the interval loss, and multiplying by $N^2$ takes it to $\infty$. Thus since both subsequences of $N$ take $g_N(X)$ to $\infty$, we are done. An analogous argument holds if $b$ exists but not $a$.

As this holds for any $X$ under conditions 1-4, which happens almost surely, we are done.
\end{proof}

\subsection{Proof of \Cref{prop::locloss-dominating}} \label{sec::locloss-dominating-proof}

To finish our Dominated Convergence Theorem (DCT) argument, we to prove \Cref{prop::locloss-dominating}, which gives an integrable function $h$ dominating all the local loss functions $g_N$. As with \Cref{prop::locloss-convergence}, we do this in stages. We first define:

\begin{definition}
For any interval $I$, let
\begin{align}
    \ell^*_I = \sup_q \ell_{q,I}
\end{align}
where $q$ is a probability distribution over $[0,1]$. If $I = I^{(n)}$ we can denote this as $\ell^*_{(n)}$.
\end{definition}
Since $\ell_{q,I}$ is only affected by $q|_I$ (i.e. what $q$ does outside of $I$ is irrelevant), we can restrict $q$ to be a probability distribution over $I$ without affecting the value of $\ell^*_I$. The question is thus: what is the maximum single-interval loss which can be produced on interval $I$?

Then, we can use the upper bound
\begin{align} \label{eq::locloss-upper-bound-1}
    g_N(x) = N^2 \ell_{p, (n_N(x))} \leq N^2 \ell^*_{(n_N(x))} \,.
\end{align}
This has the benefit of simplifying the term by removing $p$. We now bound $\ell^*_I$:

\begin{lemma} \label{lem::interval-loss-ub}
For any interval $I$, $\ell^*_I \leq \frac{1}{2} r_I^2 \bar{y}_I^{-1}$.
\end{lemma}

We give the proof in \Cref{sec::proof_interval-loss-ub}.
We can then add the above result to \eqref{eq::locloss-upper-bound-1} in order to obtain
\begin{align} \label{eq::locloss-upper-bound-2}
    g_N(x) \leq N^2 \ell^*_{(n_N(x))} \leq N^2 \frac{1}{2} r_{(n_N(x))}^2 \bar{y}_{(n_N(x))}^{-1}
\end{align}

However, it is hard to use this as the boundaries of $I^{(n_N(x))}$ in relation to $x$ are inconvenient. Instead, use an interval which is `centered' at $x$ in some way, with the help of the following:
\begin{lemma} \label{lem::ell-star}
If $I \subseteq I'$, then $\ell^*_I \leq \ell^*_{I'}$.
\end{lemma}

\begin{proof}
This follows as any $q$ over $I$ is also a distribution over $I'$ (giving $0$ probability to $I'\backslash I$).
\end{proof}

Thus, if we can find some interval $J$ such that $I^{(n_N(x))} \subseteq J$ (but of the right size) and which had more convenient boundaries, we can use that instead. We define:
\begin{definition}
For compander $f$ at scale $N$ and $x \in [0,1]$, define the interval
\begin{align}
    J^{f,N,x} = f^{-1} \Big(\Big[f(x) - \frac{1}{N}, f(x) + \frac{1}{N}\Big] \cap [0,1] \Big)
\end{align}
\end{definition}
As mentioned, we want this because it contains $I^{(n_N(x))}$:
\begin{lemma} \label{lem::bin-ub}
For any strictly monotonic $f$ and integer $N$,
\begin{align}
    I^{(n_N(x))} \subseteq J^{f,N,x}
\end{align}
\end{lemma}

\begin{proof}
Since $f$ is strictly monotonic, it has a well-defined inverse $f^{-1}$.

By definition the bin $I^{(n_N(x))}$, when passed through the compander $f$, returns $[\frac{n-1}{N}, \frac{n}{N}]$, i.e.
\begin{align}
    f(I^{(n_N(x))}) = \Big[\frac{n-1}{N}, \frac{n}{N}\Big] \, .
\end{align}
Note that this interval has width $1/N$ and includes $f(x)$ and (by definition) it is in $[0,1]$. Hence,
\begin{align}
    f(I^{(n_N(x))}) &\subseteq \Big[f(x) - \frac{1}{N}, f(x) + \frac{1}{N}\Big] \cap [0,1]
    \\ \implies f(I^{(n_N(x))}) &\subseteq f(J^{f,N,x})
    \\ \implies I^{(n_N(x))} &\subseteq J^{f,N,x}
\end{align}
and we are done.
\end{proof}

Now we can consider the importance of $f \in \cF^\dagger$: by dominating a monomial $c x^\alpha$, we can `upper bound' the interval $J^{f,N,x}$ by the equivalent interval with the compander $f_*(x) = c x^\alpha$ (i.e. $J^{f,N,x} \subseteq J^{f_*,N,x}$), which is then much nicer to work with.\footnote{While $f_*(x)$ may not map to all of $[0,1]$, it's a valid compander (but sub-optimal as it only uses some of the $N$ labels).} This also guarantees that $f$ is strictly monotonic.

\begin{lemma} \label{lem::dominating-companders}
If $f_1, f_2 \in \cF$ are strictly monotonic increasing companders such that $f_2 - f_1$ is also monotonically increasing (not necessarily strictly) and $f_1(0) = 0$, then for any $x \in [0,1]$ and $N$,
\begin{align}
    J^{f_2, N, x} \subseteq J^{f_1, N, x}
\end{align}
\end{lemma}
The proof is given in \Cref{sec::proof_dominating-companders}.
Finally, we need a quick lemma concerning the guarantee that if $f \in \cF^\dagger$, the function $g(x) = \frac{1}{24} f'(x)^{-2} x^{-1}$ is integrable under any distribution $p$:

\begin{lemma}\label{lem::h_finite}
Let $f \in \cF^\dagger$, and let $g(x) = \frac{1}{24} f'(x)^{-2} x^{-1}$. Then for any probability distribution $p$ over $[0,1]$,
\begin{align}
    \int_{[0,1]} g \, dp < \infty \,.
\end{align}
\end{lemma}

\begin{proof}
If $f \in \cF^\dagger$, then there is some $c > 0$ and $\alpha \in (0,1/2]$ such that $f(x) - cx^\alpha$ is monotonically increasing. Thus (whenever it is well-defined, which is almost everywhere by Lebesgue's differentiation theorem for monotone functions) we have $f'(x) \geq c \alpha x^{\alpha-1}$ and since $\alpha \in (0,1/2]$, we have $1 - 2\alpha \geq 0$. Thus, for all $x \in [0,1]$,
\begin{align}
    0 \leq g(x) \leq \frac{1}{24} c^{-2} \alpha^{-2} x^{1 - 2\alpha} \leq \frac{1}{24} c^{-2} \alpha^{-2}
\end{align}
which of course implies that $\int_{[0,1]} g \, p < \infty$.
\end{proof}

We can now prove \Cref{prop::locloss-dominating}, which will complete the proof of \Cref{thm::asymptotic-normalized-expdiv}.

\begin{proof}[Proof of \Cref{prop::locloss-dominating}]
As before, let $f_*(x) = c x^\alpha$; thus $f_*(0) = 0$ so we can apply \Cref{lem::dominating-companders}.

We begin, as outlined in \eqref{eq::locloss-upper-bound-2}, with:
\begin{align}
    g_N(x) &= N^2 \ell_{p,(n_N(x))}
    \\  &\leq N^2 \ell^*_{(n_N(x))} \label{eq::1st-ineq}
    \\  &\leq N^2 \ell^*_{J^{f,N,x}} \label{eq::2nd-ineq}
    \\  &\leq N^2 \ell^*_{J^{f_*,N,x}} \label{eq::3rd-ineq}
\end{align}
where \eqref{eq::1st-ineq} follows from the definition of $\ell^*_I$; \eqref{eq::2nd-ineq} follows from \Cref{lem::ell-star,lem::bin-ub}; and \eqref{eq::3rd-ineq} follows from \Cref{lem::dominating-companders}. However, since $f_*(x) = c x^\alpha$, we have a specific formula we can work with. We have $f'_*(x) = \alpha c x^{\alpha-1}$ and $f_*^{-1}(z) = (z/c)^{1/\alpha} = c^{-1/\alpha} z^{1/\alpha}$. Note that this means we can re-write
\begin{align}
    h(x) = (2^{2/\alpha} + \alpha^2 2^{1/\alpha - 2}) f'_*(x)^{-2} x^{-1} +  c^{-1/\alpha} 2^{1/\alpha - 2}
\end{align}
which sheds some light on the structure of $h(x)$. Using \Cref{lem::h_finite} proves that $\int_{[0,1]} h \, dp$ is finite if $\ensuremath{f} \in \ensuremath{\cF}$, which occurs when $\alpha \leq 1/2$.

Fix a value of $x$. Let $r_N(x)$ be the width of $J^{f_*, N, x}$. We consider two cases: (i) $cx ^{\alpha} < 1/N$; and (ii) $cx ^{\alpha} \geq 1/N$.

\emph{Case (i):} This implies $f(J^{f_*, N, x}) \subseteq [0, 2/N]$ so
\begin{align}
    x &< c^{-1/\alpha} N ^{-1/\alpha}
    \\ \implies r_N(x) &\leq c^{-1/\alpha} (N/2) ^{-1/\alpha}
\end{align}
Then, as $J^{f_*, N, x}$ has lower boundary $0$ in this case, $\bar{y}_{(n_N(x))} = r_N(x)/2$. Thus, using \eqref{eq::locloss-upper-bound-2},
\begin{align}
    g_N(x) &\leq N^2 \frac{1}{2} r_N(x)^2 \bar{y}_{(n_N(x))}^{-1}
    \\ &\leq  c^{-1/\alpha} 2^{-1/\alpha} N ^{-1/\alpha + 2}
  \,.
\end{align}
If $\alpha \leq 1/2$, then $N ^{-1/\alpha + 2}$ is maximized at $N = 1$, and thus
\begin{align}
    g_N(x) \leq c^{-1/\alpha} 2^{-1/\alpha}\,.
\end{align}
If $\alpha > 1/2$, the value $N ^{-1/\alpha + 2}$ is maximized for the largest possible $N$ still satisfying Case (i). Since $cx^{\alpha} < 1/N$, this implies that $N < c^{-1} x^{-\alpha}$. Then,
\begin{align}
    g_N(x) &\leq  c^{-1/\alpha} ( c^{-1} x^{-\alpha})^{-1/\alpha + 2} 2^{-1/\alpha}
    \\ & =  c ^{-2 } x ^{1-2\alpha } 2^{-1/\alpha}
    \\& =  \alpha^2  (c\alpha x^{\alpha- 1})^{-2} x^{-1} 2^{-1/\alpha}
    \\ & =   \alpha^2  f_*'(x)^{-2} x^{-1} 2^{-1/\alpha}
    \,.
\end{align}
Thus, for Case (i) we have that for any $a \in (0, 1]$,
\begin{align}
    g_N(x) &\leq  \alpha^2  f_*'(x)^{-2} x^{-1} 2^{-1/\alpha} +  c^{-1/\alpha}2^{-1/\alpha}\,.
\end{align}

\emph{Case (ii):} When $cx ^{\alpha} \geq 1/N$,
since $x \in I \implies \bar{y}_I \geq x/2$ (the midpoint of an interval cannot be less than half the largest element of the interval), we can upper-bound $g_N(x)$ (using \eqref{eq::3rd-ineq} and \Cref{lem::interval-loss-ub}) by
\begin{align} \label{eq::4th-ineq}
    g_N(x) \leq N^2 \frac{1}{2} r_N(x)^2 \bar{y}^{-1}_{J^{f_*,N,x}} \leq N^2 r_N(x)^2 x^{-1}\,.
\end{align}
We then bound $r_N(x)$ using the Fundamental Theorem of Calculus: since $f$ is monotonically increasing, for any $a \leq b$,
\begin{align}
    \int_a^b f'(t) \, dt \leq f(b) - f(a)
\end{align}
(any discontinuities can only make $f$ increase faster). Additionally $r_N(x) = b_1 - a_1$ where $f(b_1) = \max(f(x) + 1/N,1)$ and $f(a_1) = f(x) - 1/N$ (since it's Case (ii) we know $f(x)-1/N \geq 0$ and since $f \in \cF^\dagger$ is strictly monotonic $a_1, b_1$ are unique). Thus, if we define $a_2, b_2$ such that
\begin{align}
    \int_{a_2}^x f'(t) \, dt = 1/N \text{ and } \int_x^{b_2} f'(t) \, dt = 1/N
\end{align}
(or $a_2 = 0$ or $b_2 = 1$ if they exceed the $[0,1]$ bounds) we have $r_N(x) \leq b_2 - a_2$. Then, because $f - f_*$ is monotonically increasing, we can define $a_3, b_3$ where
\begin{align}
    \int_{a_3}^x f'_*(t) \, dt = 1/N \text{ and } \int_x^{b_3} f'_*(t) \, dt = 1/N
\end{align}
and get that $r_N(x) \leq b_3 - a_3$ (also allowing $b_3 \geq 1$ if necessary). This yields:
\begin{align}
    r_N(x) &\leq c^{-1/\alpha} \int_{\max(0, cx^\alpha-1/N)}^{\min(1, cx^\alpha+1/N)} (f^{-1}_*)'(z) \, dz
    \\ &= c^{-1/\alpha}\int_{\max(0, cx^\alpha-1/N)}^{\min(1, cx^\alpha+1/N)} \alpha^{-1} z^{1/\alpha-1} \, dz
    \\ &\leq c^{-1/\alpha}\int_{\max(0,cx^\alpha-1/N)}^{\min(1, cx^\alpha+1/N)} \alpha^{-1} (cx^\alpha+1/N)^{1/\alpha-1} \, dz
    \\ &\leq c^{-1/\alpha}\int_{cx^\alpha-1/N}^{cx^\alpha+1/N} \alpha^{-1} (cx^\alpha+1/N)^{1/\alpha-1} \, dz
    \\ &= (2/N) c^{-1/\alpha} \alpha^{-1} (cx^\alpha+1/N)^{1/\alpha-1}
\end{align}
\begin{align}
    \implies r_N(x) &\leq (2/N) c^{-1/\alpha} \alpha^{-1} (cx^\alpha+1/N)^{1/\alpha-1}
    \\ &\leq 2 N^{-1}  c^{-1/\alpha} \alpha^{-1} (2 cx^\alpha)^{1/\alpha-1}
    \\ &= N^{-1} c^{-1/\alpha} \alpha^{-1}  2^{1/\alpha} (c x^\alpha)^{1/\alpha - 1}
    \\ &= 2^{1/\alpha} N^{-1}  \big(c^{-1} \alpha^{-1} x^{1-\alpha}\big)
    \\ &= 2^{1/\alpha} N^{-1} f'_*(x)^{-1}
\end{align}
Thus, we can incorporate this into our bound \eqref{eq::4th-ineq}
\begin{align}
    g_N(x) &\leq N^2 r_N(x)^2 x^{-1}
    \\& \leq 2^{2/\alpha} f'_*(x)^{-2} x^{-1}
    \,.
\end{align}
So, $h(x)$, as the sum of the two cases,
upper bounds $g_N(x)$ no matter what.

We can also note that if $\alpha \leq 1/2$, then $x^{1-2 \alpha} \leq 1$ and hence we can upper-bound $h$ by a constant. Thus $\int_{[0,1]} h \, dp = \bbE_{X \sim p} [h(X)] < \infty$ trivially, for any $p$, and we are done.
\end{proof}

This completes the proof of \eqref{eq::norm_loss} in \Cref{thm::asymptotic-normalized-expdiv}.

\section{Minimax Compander}

\label{sec::minimax}

\Cref{thm::asymptotic-normalized-expdiv} showed that for $f \in \cF^\dagger$, the asymptotic single-letter loss is equivalent to
\begin{align}
    \ensuremath{\widetilde{L}}(p, f) = \frac{1}{24} \int_0^1 p(x) f'(x)^{-2} x^{-1} dx\,.
\end{align}
Using this, we can analyze what is the `best' compander $f$ we can choose and what is the `worst' single-letter density $p$ in order to show \Cref{thm::optimal_compander_loss,thm::minimax_compander} and their related results.

\iflong
\subsection{Optimizing for Best Compander}

\fi

We show \Cref{thm::optimal_compander_loss} and \Cref{prop::approximate-compander} together. They follow from \Cref{thm::asymptotic-normalized-expdiv} by finding $\ensuremath{f} \in \ensuremath{\cF}$ which minimizes $L^\dagger(p,\ensuremath{f})$, by optimizing over $\ensuremath{f'}$. Since $\ensuremath{f} : [0,1] \to [0,1]$ is monotonic, we use constraints $\ensuremath{f'}(x) \geq 0$ and $\int_0^1 \ensuremath{f'}(x) \, dx = 1$.
\iflong
We solve the following:
\begin{align*}
    \text{minimize } &L^\dagger(p,\ensuremath{f}) = \frac{1}{24}\int_0^1 p(x) \ensuremath{f'}(x)^{-2} x^{-1} \, dx \\
    \text{subject to } &\int_0^1 \ensuremath{f'}(x) \, dx = 1 \text{ and } \ensuremath{f'}(x) \geq 0 \text{ for all } x \in [0,1]
\end{align*}
The function $L^\dagger(p,\ensuremath{f})$ is convex in $\ensuremath{f'}$, and thus first order conditions show optimality. Let $\lambda(x)$ be a function such that $\int_0^1 \lambda(x) dx = 0$.
We derive:
\begin{align}
    \frac{d}{dt}   \frac{1}{24} & \int_0^1 p(x) \big(\ensuremath{f'}(x) + t \, \lambda(x) \big)^{-2} x^{-1} \, dx \label{eq::perturbed_func_min_L} \\
    &= \frac{1}{24} \int_0^1 p(x) x^{-1} \frac{d}{dt} \big(\ensuremath{f'}(x) + t \, \lambda(x) \big)^{-2}  \, dx \\
    &= -\frac{1}{12} \int_0^1 p(x) x^{-1} \big(\ensuremath{f'}(x) + t \, \lambda(x) \big)^{-3} \lambda(x)  \, dx \\
     &= -\frac{1}{12} \int_0^1 p(x) x^{-1} \ensuremath{f'}(x)^{-3} \lambda(x)  \, dx  ~~(\text{at } t = 0) \,
     \\ & \propto \, -\frac{1}{12} \int_0^1  \lambda(x)  \, dx = 0 ~\text{ if }~ \ensuremath{f'}(x) \, \propto \, (p(x)x^{-1})^{1/3}  \label{eq::f_proportional_to}
\end{align}
since $\lambda(x)$ integrates to $0$. Thus, such $f$ satisfies the first-order optimality condition under the constraint $\int f'(x) \, dx = 1$. This gives
\else
Using calculus of variations, we get
\fi
$
    \ensuremath{f'}_p(x) \, \propto \, (p(x)x^{-1})^{1/3}
$
and $\ensuremath{f}(0) = 0$ and $\ensuremath{f}(1) = 1$, from which \eqref{eq::raw_overall_dist} and \eqref{eq::best_f_raw} follow.
If $\ensuremath{f}_p \in \ensuremath{\cF}^\dagger$, then $\ensuremath{f}_p = \argmin_f \ensuremath{\widetilde{L}}(p,\ensuremath{f})$, and for any other $\ensuremath{f} \in \ensuremath{\cF}$,
\begin{align}
    \hspace{-0.5pc} \ensuremath{\widetilde{L}}(p,\ensuremath{f}_p) &= L^\dagger(p,\ensuremath{f}_p)
     \leq L^\dagger(p,\ensuremath{f})
     \leq \liminf_{N \to \infty} N^2 \ensuremath{\widetilde{L}}(p,\ensuremath{f},N)
\end{align}

If $\ensuremath{f}_p \not \in \ensuremath{\cF}^\dagger$, for any $\delta > 0$ define $\ensuremath{f}_{p,\delta} = (1 - \delta) \ensuremath{f}_p + \delta x^{1/2}$ (as in \eqref{eq::approximate-compander}). Then $\ensuremath{f}_{p,\delta} - \delta x^{1/2} = (1 - \delta) \ensuremath{f}_p$ is monotonically increasing so $\ensuremath{f} \in \ensuremath{\cF}^\dagger$, so \Cref{thm::asymptotic-normalized-expdiv} applies to $\ensuremath{f}_{p,\delta}$; additionally, $\ensuremath{f}_{p,\delta} - (1-\delta) \ensuremath{f}_p = \delta x^{1/2}$ is monotonically increasing as well so $\ensuremath{f}'_{p,\delta} \geq (1-\delta) \ensuremath{f}'_p$. Hence, plugging into the $L^\dagger$ formula gives:
\begin{align}
 &\ensuremath{\widetilde{L}}(p,\ensuremath{f}_{p,\delta}) = L^\dagger(p,\ensuremath{f}_{p,\delta}) \leq L^\dagger(p,\ensuremath{f}_p)(1-\delta)^{-2}  \, .
\end{align}
Taking $\delta \to 0$ (and noting that $\ensuremath{\cF}^\dagger \subseteq \ensuremath{\cF}$) thus shows that
\begin{align}
    L^\dagger(p,\ensuremath{f}_p) = \inf_{\ensuremath{f} \in \ensuremath{\cF}^\dagger} \ensuremath{\widetilde{L}}(p,\ensuremath{f})\,.
\end{align}
This finishes the proofs of \Cref{thm::optimal_compander_loss} and \Cref{prop::approximate-compander}.

\begin{remark}
Since we know the corresponding single-letter source $p$ for a Dirichlet prior, using this $p$ with \Cref{thm::optimal_compander_loss} gives us the optimal compander for Dirichlet priors on any alphabet size. This gives us a better quantization method than EDI which was discussed in \Cref{{sec::experimental_results}}. This optimal compander is called the \emph{beta compander} and its details are given in \Cref{sec::beta_companding}.
\end{remark}

\iflong
\subsection{Minimax and Approximate Minimax Companders}
\fi

To prove \Cref{thm::minimax_compander} and \Cref{cor::worstcase_prior}, we first consider what density $p$ maximizes
\begin{align}
\frac{1}{24} \left(\int_0^1 (p(x)x^{-1})^{1/3} dx\right)^3
\end{align}
(equation \eqref{eq::raw_overall_dist}), i.e. is most difficult to quantize with a compander.

Using calculus of variations to maximize
\begin{align}
    \int_0^1 (p(x)x^{-1})^{1/3} \,dx \label{eq::loss-to-maximize}
\end{align}
(which of course maximizes \eqref{eq::raw_overall_dist}) subject to $p(x) \geq 0$ and $\int_0^1 p(x) \, dx = 1$, we find that maximizer is $p(x) = \frac{1}{2} x^{-1/2}$. However, while interesting, this is only for a single letter; and because $\bbE[X] = 1/3$ under this distribution, it is clearly impossible to construct a prior over the simplex (whose output vector \emph{must} sum to $1$) with this marginal (unless $\ensuremath{K} = 3$).

\iflong
Hence, we add an expected value constraint to the problem of maximizing \eqref{eq::loss-to-maximize}, giving:
\begin{align}
    \text{maximize } &\int_0^1 \big(p(x)x^{-1}\big)^{1/3} \, dx  \\
    \text{subject to } &\int_0^1 p(x) \, dx = 1\label{eq::p_constraint_sum}; \\
    &\int_0^1 p(x)x \,dx = \frac{1}{\ensuremath{K}} \label{eq::p_constraint_mean};\\
    &\text{and } p(x) \geq 0 \text{ for all } x
\end{align}
We can solve this again using variational methods (we are maximizing a concave function so satisfying first order conditions are enough to ensure optimality). A function $p(x) > 0$ is optimal if, for any $\lambda(x)$ where
\begin{align}
    \int_0^1 \lambda(x) \, dx = 0 \text{ and } \int_0^1 \lambda(x)x \, dx &= 0 \label{eq::variational_constraints}
\end{align}
the following holds:
\begin{align}
    \frac{d}{dt} \int_0^1 x^{-1/3} \big(p(x) + t \, \lambda(x)\big)^{1/3} \, dx = 0\,.
\end{align}
We have by the same logic as before:
\begin{align}
    \frac{d}{dt}  \int_0^1 \ensuremath{} x^{-1/3} \big(p(x) + t \, \lambda(x)\big)^{1/3} \, dx \ensuremath{}
    &= \frac{1}{3} \int_0^1 x^{-1/3} \big(p(x) + t \, \lambda(x)\big)^{-2/3} \lambda(x) \, dx \\
    &= \frac{1}{3} \int_0^1 x^{-1/3} p(x)^{-2/3} \lambda(x) \, dx ~~ (\text{at } t = 0) \label{eq::variational_condition}
\end{align}
Thus, if we can arrange things so that there are constants $a_{\ensuremath{K}}, b_{\ensuremath{K}}$ such that
\begin{align}
    x^{-1/3} p(x)^{-2/3} = a_\ensuremath{K}  + b_\ensuremath{K}  x
\end{align}
this ensures \eqref{eq::variational_condition} equals zero. In that case,
\begin{align}
    x^{-1/3} p(x)^{-2/3} &= a_\ensuremath{K}  + b_\ensuremath{K}  x \\
    \iff \hspace{2.5pc} p(x)^{-2/3} &= a_\ensuremath{K}  x^{1/3} + b_\ensuremath{K}  x ^{4/3} \\
    \iff \hspace{4.1pc}  p(x) &= \big(a_\ensuremath{K}  x^{1/3} + b_\ensuremath{K}  x^{4/3}\big)^{-3/2}\label{eq::maximin-density_repeat}
\end{align}
This yields the {maximin density} $p^*_{\ensuremath{K}}$ \eqref{eq::maximin-density} from \Cref{thm::minimax_compander}, where $a_\ensuremath{K} , b_\ensuremath{K} $ are set to meet the constraints \eqref{eq::p_constraint_sum} and \eqref{eq::p_constraint_mean}.
Exact formulas for $a_\ensuremath{K}, b_\ensuremath{K}$ are difficult to find. We will give more details on $a_\ensuremath{K}, b_\ensuremath{K}$ after the next step.

We want to determine the optimal compander for the maximin density \eqref{eq::maximin-density_repeat}. We know from
\eqref{eq::f_proportional_to} that we need to first compute
\begin{align} \label{eq::minimax-compander-proportional}
     \phi(x) = \int_0 ^{x} z^{-1/3} \left(a_\ensuremath{K}  z^{1/3} + b_\ensuremath{K}  z^{4/3} \right)^{-1/2} \,dz & = \frac{2 \ensuremath{\mathrm{ArcSinh}} \left(\sqrt{\frac{b_\ensuremath{K}  x}{a_\ensuremath{K} }} \right)}{\sqrt{b_\ensuremath{K} }}\,.
\end{align}

The best compander $f(x)$ is proportional to \eqref{eq::minimax-compander-proportional} and is exactly given by $f(x) = \phi(x)/\phi(1)$. The resulting compander, which we call the \emph{minimax compander}, is
\begin{align}
    f(x) &= \frac{\ensuremath{\mathrm{ArcSinh}} \left(\sqrt{\frac{b_\ensuremath{K}  x}{a_\ensuremath{K} }} \right)}{\ensuremath{\mathrm{ArcSinh}} \left(\sqrt{\frac{b_\ensuremath{K}  }{a_\ensuremath{K} }} \right)} \,.
\end{align}
Given the form of $f(x)$, it is natural to
determine an expression for the ratio $b_\ensuremath{K} / a_\ensuremath{K}$.  We can parameterize both $a_\ensuremath{K}$ and $b_\ensuremath{K}$ by $b_\ensuremath{K} / a_\ensuremath{K}$ and then examine how $b_\ensuremath{K} / a_\ensuremath{K}$ behaves as a function of $\ensuremath{K}$.
The constraints on $a_\ensuremath{K}$ and $b_\ensuremath{K}$ give that
\begin{align}
    a_{\ensuremath{K}} &= 4^{1/3}(b_\ensuremath{K} / a_\ensuremath{K} + 1)^{-1/3}
    \\b_\ensuremath{K}  &= 4 a_{\ensuremath{K}}^{-2} - a_{\ensuremath{K}}\,.
\end{align}
The ratio $b_\ensuremath{K} / a_\ensuremath{K}$ grows approximately as $\ensuremath{K} \log \ensuremath{K}$. Hence, we choose to parameterize
\begin{align}
    b_\ensuremath{K} / a_\ensuremath{K} = c_\ensuremath{K} \ensuremath{K} \log \ensuremath{K}\,.
\end{align}
To satisfy the constraints, we get that $.25 < c_\ensuremath{K} < .75$ so long as $\ensuremath{K} > 24$ (the details are in \Cref{sec::minimax_const_analysis}), and \Cref{lem::c_k_half} in \Cref{sec::lim_c_k} shows that $c_\ensuremath{K} \to 1/2$ as $\ensuremath{K} \to \infty$.

We can express $a_\ensuremath{K}$, $b_\ensuremath{K}$ in terms of $c_\ensuremath{K}$:
\begin{align}
    a_{\ensuremath{K}} &= 4^{1/3}(c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K} + 1)^{-1/3}
    \\b_\ensuremath{K}  &= 4 a_{\ensuremath{K}}^{-2} - a_{\ensuremath{K}}
    \\ &= 4^{1/3} (c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K} + 1)^{2/3} - 4^{1/3}(c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K} + 1)^{-1/3}
    \label{eq::value_c_2_worst+p}
    \\& = 4^{1/3}(c_\ensuremath{K} \ensuremath{K} \log \ensuremath{K})^{2/3} (1+o(1))\,.
\end{align}
We use here that for large $\ensuremath{K}$, the second term in \eqref{eq::value_c_2_worst+p} is negligible compared to the first.

Thus we get the {minimax compander} and {approximate minimax compander}, respectively:
\begin{align}
    \ensuremath{f}^*_{\ensuremath{K}}(x) &= \frac{\ensuremath{\mathrm{ArcSinh}} \left(\sqrt{(c_{\ensuremath{K}} \ensuremath{K} \log \ensuremath{K}) x} \right)}{\ensuremath{\mathrm{ArcSinh}} \left(\sqrt{c_{\ensuremath{K}} \ensuremath{K} \log \ensuremath{K}} \right)}
    \\ \approx \ensuremath{f}^{**}_{\ensuremath{K}}(x) &= \frac{\ensuremath{\mathrm{ArcSinh}} (\sqrt{((1/2) \ensuremath{K} \log \ensuremath{K}) x} )}{\ensuremath{\mathrm{ArcSinh}} (\sqrt{(1/2) \ensuremath{K} \log \ensuremath{K}} )} \,.
\end{align}
The minimax compander minimizes the maximum (raw) loss against all densities in $\cP_{1/\ensuremath{K}}$, while the approximate minimax compander performs very similarly but is more applicable since it can be used without computing $c_\ensuremath{K}$.

To compute the loss of the minimax compander, we can use \eqref{eq::raw_overall_dist} to get
\begin{align}
    L^{\dagger}(p_{\ensuremath{K}}^*, \ensuremath{f}^*_{\ensuremath{K}})
    & = \frac{1}{24} \left(\frac{2 \ensuremath{\mathrm{ArcSinh}}\left(\sqrt{c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K}} \right)} {\sqrt{b_\ensuremath{K} }} \right)^3
\end{align}
Substituting we get
\begin{align}
      L^{\dagger}(p_{\ensuremath{K}}^*, \ensuremath{f}^*_{\ensuremath{K}})  &= \frac{1}{24} \frac{8 \left(\log \left(\sqrt{c_\ensuremath{K} \ensuremath{K} \log \ensuremath{K}} +  \sqrt{c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K} + 1} \right)\right)^3}{2 c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K} (1+o(1))}\\
     & = \frac{1}{24} \frac{(\log 4 (c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K}))^3}{2 c_\ensuremath{K}  \ensuremath{K} \log \ensuremath{K}} (1+o(1)) \\

     & =  \frac{1}{24} \frac{\log^2 \ensuremath{K}}{\ensuremath{K}} (1 + o(1)) \label{eq::L_dagger_value_minimax}\,.
\end{align}

\else

Hence, we add the constraint $p \in \cP_{1/\ensuremath{K}}$, maximizing \eqref{eq::loss-to-maximize} subject to $p(x) \geq 0$, $\int_0^1 p(x) \, dx = 1$, and $\int_0^1 p(x) x \, dx = 1/\ensuremath{K}$. Solving this modified problem yields the \emph{maximin density} $p^*_{\ensuremath{K}}$ \eqref{eq::maximin-density} from \Cref{thm::minimax_compander}; $\ensuremath{f}^*_{\ensuremath{K}}$ \eqref{eq::minimax-compander} is simply the optimal compander (given by \eqref{eq::best_f_raw}) against $p^*_{\ensuremath{K}}$.
\fi

In fact, not only is $f^*_\ensuremath{K}$ optimal against the maximin density $p^*_\ensuremath{K}$, but (as alluded to in the name `minimax compander') it minimizes the maximum asymptotic loss over all $p \in \cP_{1/\ensuremath{K}}$.
More formally we show that $(f^*_\ensuremath{K}, p^*_\ensuremath{K})$ is a saddle point of $L^\dagger$.

The function $L^\dagger(p,\ensuremath{f})$ is concave (actually linear) in $p$ and convex in $\ensuremath{f'}$, and we can show that the pair $(\ensuremath{f}^*_{\ensuremath{K}}, p^*_{\ensuremath{K}})$ form a saddle point, thus proving \eqref{eq::minmax}-\eqref{eq::maxmin} from \Cref{thm::minimax_compander}.

\iflong
We can compute that
\begin{align}
    (\ensuremath{f}_\ensuremath{K}^*)'(x)  \, &\propto \, (p_{\ensuremath{K}}^*(x) x^{-1})^{1/3} \\
    & = x^{-1/3} (a_\ensuremath{K}  x^{1/3} + b_\ensuremath{K}  x^{4/3})^{-1/2} \\

    & = \frac{1}{\sqrt{a_\ensuremath{K}  x + b_\ensuremath{K}  x^2}}\,.
\end{align}
Assume we set $a_\ensuremath{K} $ and $b_\ensuremath{K} $ to the appropriate values for $\ensuremath{K}$.
For any $p \in \cP_{1/\ensuremath{K}}$,
\begin{align}
     {L^{\dagger}}(p, \ensuremath{f}_\ensuremath{K}^*) & = \int_0^1 p(x) x^{-1} ((\ensuremath{f}_\ensuremath{K}^*)'(x))^{-2}  dx\\
    & = \int_0^1 p(x) x^{-1} (a_\ensuremath{K}  x + b_\ensuremath{K}  x^2) dx \\
    & = \int_0^1 p(x) (a_\ensuremath{K}  + b_\ensuremath{K}  x) dx \\
    & = a_\ensuremath{K}  + b_\ensuremath{K}  \frac{1}{\ensuremath{K}}
\end{align}
i.e. $L^\dagger(p,f^*_\ensuremath{K})$ doesn't depend on $p$. Since $f^*_\ensuremath{K}$ is the optimal compander against the maximin compander $p^*_\ensuremath{K}$ we can therefore conclude:
\begin{align}
\sup_{p \in \cP_{1/\ensuremath{K}}} {L^{\dagger}}(p, \ensuremath{f}_\ensuremath{K}^*)  &= {L^{\dagger}}(p_\ensuremath{K}^*, \ensuremath{f}_\ensuremath{K}^*)
\\ = \inf_{\ensuremath{f} \in \ensuremath{\cF}} {L^{\dagger}}(p_\ensuremath{K}^*, \ensuremath{f}) &= \sup_{p \in \cP_{1/\ensuremath{K}} }\inf_{\ensuremath{f} \in \ensuremath{\cF}} {L^{\dagger}}(p, \ensuremath{f})\,.
\end{align}
Since it is always true that
\begin{align}\label{eq::minimax_ineq}
      \sup_{p \in \cP_{1/\ensuremath{K}}} \inf_{\ensuremath{f} \in \ensuremath{\cF}}  {L^{\dagger}}(p, \ensuremath{f}) \leq   \inf_{\ensuremath{f} \in \ensuremath{\cF}} \sup_{p \in \cP_{1/\ensuremath{K}}}  {L^{\dagger}}(p, \ensuremath{f}) \,,
\end{align}
this completes showing that $(\ensuremath{f}_\ensuremath{K}^*, p^*_{\ensuremath{K}})$ is a saddle point.
\fi

Furthermore, $\ensuremath{f}_\ensuremath{K}^* \in \ensuremath{\cF}^\dagger$ (specifically it behaves as a multiple of $x^{1/2}$ near $0$), so $\ensuremath{\widetilde{L}}(p,\ensuremath{f}_\ensuremath{K}^*) = L^\dagger(p,\ensuremath{f}_\ensuremath{K}^*)$ for all $p$, thus showing that $\ensuremath{f}_\ensuremath{K}^*$ performs well against any $p \in \cP_{1/\ensuremath{K}}$. Using \eqref{eq::raw_loss} with the expressions for $p^*_{\ensuremath{K}}$ and $\ensuremath{f}_\ensuremath{K}^*$ and \eqref{eq::L_dagger_value_minimax} gives \eqref{eq::raw_loss_saddle}. This completes the proof of \Cref{thm::minimax_compander}.

\begin{remark}
While the power compander $\ensuremath{f}(x) = x^{1/\log \ensuremath{K}}$ is not minimax optimal, it has similar properties to the minimax compander and differs in loss by at most a constant factor. We analyze the power compander in \Cref{sec::power_compander_analysis}.
\end{remark}

\iflong
\subsection{Existence of Priors with Given Marginals}
\fi

While $p^*_{\ensuremath{K}}$ is the worst density in $\cP_{1/\ensuremath{K}}$, it is unclear whether a prior $P^*$ on $\triangle_{\ensuremath{K}-1}$ exists with marginals $p^*_{\ensuremath{K}}$ -- even though $p^*_{\ensuremath{K}}$ has the correct expectation so that $\ensuremath{K}$ copies will sum to $1$ in expectation, it may not be possible to correlate them to guarantee they sum to $1$. However, it is possible to construct a prior $P^*$ whose marginals are as hard to quantize, up to a constant, as $p^*_{\ensuremath{K}}$, by use of clever correlation between the letters. We start with a lemma:

\begin{lemma}\label{lem::cramming2}
Let $p \in \cP_{1/\ensuremath{K}}$. Then there exists a joint distribution of $(X_1, \dots, X_{\ensuremath{K}})$ such that (i) $X_i \sim p$ for all $i \in [\ensuremath{K}]$ and (ii) $\sum_{i \in [\ensuremath{K}]} X_i \leq 2$, guaranteed.
\end{lemma}

\iflong
\begin{proof}
Let $F$ be the cumulative distribution function of $p$. Define the quantile function $F^{-1}$ as
\begin{align}
    F^{-1}(u) = \inf \{x : F(x) \geq u\}.
\end{align}

We break $[0,1]$ into $\ensuremath{K}$ uniform sub-intervals $I_i = ((i-1)/\ensuremath{K}, i/\ensuremath{K}]$ (let $I_1 = [0, 1/\ensuremath{K}]$). We then generate $X_1, X_2, \dots, X_{\ensuremath{K}}$ jointly by the following procedure:
\begin{enumerate}
    \item Choose a permutation $\sigma : [\ensuremath{K}] \to [\ensuremath{K}]$ uniformly at random (from $\ensuremath{K}!$ possibilities).
    \item Let $U_k \sim \unif_{I_{\sigma(k)}}$ independently for all $k$.
    \item Let $X_k = F^{-1}(U_k)$.
\end{enumerate}

Now we consider $\sum_k X_k$. Let $b_i = F^{-1}(i/k)$ for $i = 0, 1, \dots, \ensuremath{K}$. Note that if $\sigma(k) = i$ then $U_k \in ((i-1)/\ensuremath{K}, i/\ensuremath{K}]$ and hence $X_k = F^{-1}(U_k) \in [b_{i-1}, b_i]$. Therefore $X_{\sigma^{-1}(i)} \in [b_{i-1}, b_i]$ and thus for any permutation $\sigma$,
\begin{align}
    \sum_{i=1}^\ensuremath{K} b_{i-1} &\leq \sum_{i=1}^\ensuremath{K} X_{\sigma^{-1}(i)} \leq \sum_{i=1}^\ensuremath{K} b_i \\
    &= \Big(\sum_{i=1}^\ensuremath{K} b_{i-1}\Big) + b_{\ensuremath{K}} - b_0\\
    &\leq \Big(\sum_{i=1}^\ensuremath{K} b_{i-1}\Big) + 1  \leq 2
\end{align}

Note that we used
\begin{align}
    \sum_{i=1}^\ensuremath{K} b_{i-1} \leq \sum_{i=1}^\ensuremath{K} \bbE[X_{\sigma^{-1}(i)}]
     = \ensuremath{K} \bbE_{X \sim p}[X] = 1\,.
\end{align}

\end{proof}
\fi

\Cref{lem::cramming2} shows a joint distribution of $Z_1, \dots, Z_{\ensuremath{K}-1}$ such that $Z_i \sim p^*_{\ensuremath{K}}$ for all $i$ and $\sum_{i=1}^{\ensuremath{K}-1} Z_i \leq 2$ (guaranteed) exists.
\iflong
Then, if $X_i = Z_i/2$ for all $i \in [\ensuremath{K}-1]$, we have $\sum_{i=1}^{\ensuremath{K}-1} X_i \leq 1$. Then setting $X_{\ensuremath{K}} = 1 - \sum_{i = 1}^{\ensuremath{K} - 1} X_i \geq 0$ ensures that $(X_1, \dots, X_{\ensuremath{K}})$ is a probability vector.
\else
We scale by $1/2$ and add $X_{\ensuremath{K}}$ so that all $\ensuremath{K}$ variables sum to one.
\fi
Denoting this prior $P^*_{\text{hard}}$ and letting $p^{**}_{\ensuremath{K}}(x) = 2p^*_{\ensuremath{K}}(2x)$ (so $Z_i \sim p^*_{\ensuremath{K}} \implies X_i \sim p^{**}_{\ensuremath{K}}$) we get that
\begin{align}
    \inf_{\ensuremath{f} \in \ensuremath{\cF}} \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}&(P^*_{\text{hard}}, \ensuremath{f}) \geq (\ensuremath{K} - 1) \inf_{\ensuremath{f} \in \ensuremath{\cF}} \ensuremath{\widetilde{L}}(p^{**}_{\ensuremath{K}}, \ensuremath{f}) \label{eq::scale2_pstart}
    \\ = (\ensuremath{K} - 1)& \frac{1}{2} {L^{\dagger}}(p^*_{\ensuremath{K}}, \ensuremath{f}_\ensuremath{K}^*) \label{eq::after_scale2_pstart}
    \geq  \frac{1}{2} \frac{\ensuremath{K} - 1}{\ensuremath{K}}  \sup_{P \in \cP^\triangle_{\ensuremath{K}}} \ensuremath{\widetilde{\cL}}_{\ensuremath{K}}(P, \ensuremath{f}_\ensuremath{K}^*) \,.
\end{align}
The last inequality holds because $p^*_{\ensuremath{K}}$ is the maximin density (under expectation constraints). To make $P^*_{\text{hard}}$ symmetric, we permute the letter indices randomly without affecting the raw loss; thus we get \Cref{cor::worstcase_prior}.
To show how to get \eqref{eq::after_scale2_pstart} from \eqref{eq::scale2_pstart}, we have
\begin{align}
    \inf_{\ensuremath{f} \in \ensuremath{\cF}} \ensuremath{\widetilde{L}}(2p^*_{\ensuremath{K}}(2x), \ensuremath{f})   & = \frac{1}{24} \left(\int_0^1 (2p^*_{\ensuremath{K}}(2x) x^{-1})^{1/3} dx\right)^3\\
    & = \frac{1}{24} \left(\int_0^1 (2p^*_{\ensuremath{K}}(u) 2 u^{-1})^{1/3} \frac{1}{2}du\right)^3\\

    & = \frac{1}{2} {L^{\dagger}}(p^*_{\ensuremath{K}}, \ensuremath{f}_*)
\end{align}

\iflong
In \Cref{fig:cramming_plot}, we validate the distribution $P^*_{\text{hard}}$ by showing the performance of each compander when quantizing  random distributions drawn from $P^*_{\text{hard}}$. For the minimax compander, the KL divergence loss on the worst-case prior looks to be within a constant of that for the other datasets.

\begin{figure}
    \centering
    \includegraphics[scale = .6]{figures/kl_ucsc_dna_books_worst-case-cramming__truncate_exp_sinh_bfloat16_65536_fig.pdf}
    \caption{Each compander (or quantization method) is used on random distributions drawn from the prior $P^*_{\text{hard}}$. Comparison is given to when each compander is used on the books and DNA datasets.}
    \label{fig:cramming_plot}
\end{figure}
\fi

\section{Companders for Other Distances and Spaces}

\label{sec::other_losses}

While our primary focus has been KL divergence over the simplex, for context we compare our results to what the same compander analysis would give for other loss functions like squared Euclidean distance ($L_2^2$) and absolute distance ($L_1$). For a vector $\bx$ and its representation $\by$ let
\begin{align}
    L_2^2(\bx, \by) &= \sum_i (x_i - y_i)^2\\
    L_1(\bx, \by) &= \sum_i |x_i - y_i|
\end{align}

For squared Euclidean distance, asymptotic loss was already given by \eqref{eq::bennett_compander} in \cite{bennett1948}. Note that the formula scales as $1/N^2$. It turns out that the maximin single-letter distribution over a bounded interval is the uniform distribution. Thus, the minimax compander for $L_2^2$ is simply the identity function. Uniform quantization is indeed the `right' choice for quantizing a hypercube in high-dimensional space. (For unbounded spaces, $L_2^2$ loss does not scale with $1/N^2$.)

If we add the expected value constraint to the $L_2^2$ compander optimization problem, we can derive the best square distance compander for the probability simplex. For alphabet size $\ensuremath{K}$, we get that the minimax compander for $L_2^2$ is given by
\begin{align}
    f_{L_2^2, \ensuremath{K}}(x) = \frac{\sqrt{1 + \ensuremath{K}(\ensuremath{K}-2)x} - 1}{\ensuremath{K}-2}
\end{align}
and the total $L_2^2$ loss for probability vector $\bx$ and its quantization $\by$ has the relation
\begin{align}
    \lim_{N \to \infty} N^2 L_2^2 (\bx, \by) \leq \frac{1}{3}\,.
\end{align}

For $L_1$, unlike KL divergence and $L_2^2$, the loss scales as $1/N$. Like $L_2^2$, the minimax single-letter compander for $L_1$ loss in the hypercube $[0,1]^\ensuremath{K}$ is the identity function, i.e. uniform quantization. In general, the derivative of the optimal compander for single-letter density $p(x)$ has the form
\begin{align}
    \ensuremath{f}_{L_1, \ensuremath{K}} ' (x) \, \propto \, \sqrt{p(x)}\,.
\end{align}

On the probability simplex for alphabet size $\ensuremath{K}$, the worst case prior $p(x)$ has the form
\begin{align}
    p(x) = (\alpha_\ensuremath{K} x + \beta_\ensuremath{K} )^{-2}
\end{align}
where $\alpha_\ensuremath{K}, \beta_\ensuremath{K}$ are constants scaling to allow $\int_{[0,1]} \, dp = 1$ (i.e. $p$ is a valid probability density) and $\int_{[0,1]} x \, dp = 1/\ensuremath{K}$ (i.e. $\bbE_{X \sim p}[X] = 1/\ensuremath{K}$ so $\ensuremath{K}$ copies of it are expected to sum to $1$).

Thus, the minimax compander on the simplex for $L_1$ loss (and letting $\gamma_\ensuremath{K} = \alpha_\ensuremath{K}/\beta_\ensuremath{K}$) satisfies
\begin{align}
    \ensuremath{f}_{L_1, \ensuremath{K}} ' (x) \, &\propto \, (\alpha_\ensuremath{K} x + \beta_\ensuremath{K})^{-1}
    \\ \implies \ensuremath{f}_{L_1, \ensuremath{K}} (x) \, &\propto \, \log((\alpha_\ensuremath{K}/\beta_\ensuremath{K}) x + 1)
    \\ \implies \ensuremath{f}_{L_1, \ensuremath{K}} (x) &= \frac{\log(\gamma_\ensuremath{K} x + 1)}{\log(\gamma_\ensuremath{K} + 1)}
\end{align}
since $\ensuremath{f}_{L_1, \ensuremath{K}} (x)$ has to be scaled to go from $0$ to $1$.

The asymptotic $L_1$ loss for probability vector $\bx$ and its quantization $\by$ is bounded by
\begin{align}
    \lim_{N \to \infty} N L_1 (\bx, \by) \leq \frac{(\log \ensuremath{K})^3}{4}\,.
\end{align}

\renewcommand{\arraystretch}{1}
\begin{figure*}[]
    \centering
    \begin{tabular}{|c|c|l|c|}
        \hline
         Loss & Space & Optimal Compander & Asymp. UB   \\
         \hline
         KL & Simplex & $ \ensuremath{f}^*_{\ensuremath{K}}(x) = \frac{\ensuremath{\mathrm{ArcSinh}}(\sqrt{c_{\ensuremath{K}}  (\ensuremath{K} \log \ensuremath{K}) \, x})}{\ensuremath{\mathrm{ArcSinh}}(\sqrt{c_{\ensuremath{K}}  \ensuremath{K} \log \ensuremath{K}})}$  & $\frac{(\log \ensuremath{K})^2}{N^2} $ \\
         $L_2^2$ & Simplex & $ f_{L_2^2, \ensuremath{K}}(x) = \frac{\sqrt{1 + \ensuremath{K}(\ensuremath{K}-2)x} - 1}{\ensuremath{K}-2}$ & $\frac{1}{N^2}$ \\
         $L_2^2$ & Hypercube & $ f_{L_2^2}(x) = x$ (uniform quantizer) & $\frac{\ensuremath{K}}{N^2}$ \\
          $L_1$  & Simplex & $f_{L_1, \ensuremath{K}}(x) = \frac{\log(\gamma_\ensuremath{K} x + 1)}{\log(\gamma_\ensuremath{K} + 1)}$ & $\frac{(\log \ensuremath{K})^3}{N}$ \\
           $L_1$  & Hypercube & $f_{L_1}(x) = x$ (uniform quantizer)  & $\frac{\ensuremath{K}}{N}$ \\
          \hline
    \end{tabular}
    \caption{Summary of results for various losses and spaces. Asymp. UB is the asymptotic upper bound, i.e an upper bound on how we expect the loss of the optimal compander to scale with $N$ and $\ensuremath{K}$ (constant terms are neglected). }
    \label{tab::other_losses_spaces}
\end{figure*}

\section{Acknowledgements}

We would like to thank Anthony Philippakis for his guidance on the DNA $k$-mer experiments.

\bibliographystyle{resources/IEEEbib}