Billiards in ellipses revisited

Abstract: We prove some recent experimental observations of D. Reznik concerning periodic billiard orbits in ellipses. For example, the sum of cosines of the angles of a periodic billiard polygon remains constant in the one-parameter family of such polygons (that exist due to the Poncelet porism). In our proofs, we use geometric and complex analytic methods.

\section{Introduction} \label{sec:intro}

The billiard in an ellipse is a thoroughly studied completely integrable dynamical system, see, e.g., \cite{tabachnikov2005geometry}.
In particular, a billiard trajectory that is tangent to a confocal ellipse will remain tangent to it after each reflection.
That is, the confocal ellipses are the caustics of this billiard.

One of the properties of this system, a consequence of its complete integrability, is that a periodic
billiard trajectory tangent to a confocal ellipse includes into a 1-parameter family of periodic
trajectories tangent to the same confocal ellipse and having the same period and the same rotation number.
This is the assertion of the Poncelet porism for confocal ellipses.
The Poncelet porism concerns the same kind of $1$-parameter family of
polygons that are simultaneously inscribed and circumscribed in the
same pair of conics, but in general the conics need not be confocal.

A classic result about a continuous $1$-parameter family of billiard paths
is that their perimeters remain constant.
(See  \cite{tabachnikov2005geometry}, and also Lemma \ref{perimeter} below.)
Recently Dan Reznik conducted a large series of computer experiments with periodic orbits in
elliptic billiards and discovered numerous new properties of these polygons
that are similar in spirit to the constant-perimeter result.  See
\cite{reznik2019applet, reznik2019playlist, reznik2019media, reznik2019elliptic, reznik2020new, reznik2020loci}.
In this paper we give proofs which verify some of these observations.
Essentially, we prove $3$ main results.  In the body of the
paper, we will also prove a number of variants and generalizations.

We would also like to mention that another proof of the first two results is
presented in \cite{bialy2020danreznik}; it is based on a non-standard generating
functions for convex billiard discovered by Misha Bialy.
\newline
\newline
\noindent
{\bf First Result:\/}
Let $\alpha_1,...,\alpha_n$ be the angles associated to a periodic billiard path.
Figure \ref{fig:angles} shows the case $n=5$.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-3.mps}
\caption{Angles of a billiard trajectory.}
\label{fig:angles}
\end{figure}

\begin{theorem} \label{thm:sumcos2}
The sum
\[
\sum_{i=1}^n \cos\alpha_i
\]
remains constant as $P$ varies a 1-parameter family of periodic billiard paths on an ellipse.
\end{theorem}
This result corresponds exactly to one of Reznik's observations.
We will give two proofs.  The first proof is based on the invariance of the perimeter,
mentioned above, and on the invariance of a quantity called the
{\it Joachimsthal integral\/}.  The second proof is based on Liouville's
Theorem from complex analysis.
\newline
\newline
\noindent
{\bf Second Result:\/}
Let $\beta_1,...,\beta_n$ be the angles of the
polygon formed by tangents to the ellipse at the vertices of a periodic billiard trajectory,
as shown in Figure \ref{fig:prod} for the case $n=5$.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-4.mps}
\caption{The angles involved in Theorem \ref{thm:prodcos}.}
\label{fig:prod}
\end{figure}

\begin{theorem} \label{thm:prodcos}
The the product
\[
\prod_{i=1}^n \cos\beta_i
\]
remains constant as $P$ varies in a 1-parameter family of periodic billiard paths on an ellipse.
\end{theorem}
This result also corresponds exactly to one of Reznik's observations.
Our proof is the same kind of complex analysis argument that we
use for Theorem \ref{thm:sumcos2}.  We don't know a proof along the
lines of our first proof of Theorem \ref{thm:sumcos2} but see \cite{bialy2020danreznik}.
\newline
\newline
\noindent
{\bf Third Result:\/}
Figure~\ref{fig:areas} shows how we construct a polygon $Q$ starting from a Poncelet polygon $P$.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-6}
\caption{To Theorem \ref{thm:areas}.}
\label{fig:areas}
\end{figure}

\begin{theorem} \label{thm:areas}
Let $P$ be a Poncelet $n$-gon with odd $n$, inscribed into an ellipse and circumscribed about
a concentric ellipse.  Let $Q$ be the polygon formed by the tangent lines to the outer ellipse at the vertices of $P$.
The ratio of the areas of the two polygons remains constant within the
Poncelet family containing $P$.
\end{theorem}

This result is a mild generalization of an
observation of Reznik.
We will explain the exact relationship after giving
the proof.  The proof itself is short. It follows
readily from a result called the
Poncelet Grid Theorem \cite{Schwartz2007poncelet,levi2007poncelet}.

\section{Proof of the First Result} \label{genfacts}

Consider an ellipse in ${\mathbb R}^2$ given by the equation
\begin{equation} \label{eq:ellipse}
\frac{x_1^2}{a_1^2} + \frac{x_2^2}{a_2^2} =1,
\end{equation}
or, in the matrix form, $\langle Ax,x\rangle=1$, where $A = {\rm diag} (1/a_1^2,1/a_2^2)$.

Recall the notion of polar duality: given a smooth convex closed planar curve $\gamma$, to a
vector $x \in \gamma$ there corresponds a unique covector $x^*$ satisfying the conditions
\[
{\rm Ker}\ x^*=T_x\gamma\ {\rm and}\ (x,x^*)=1;
\]
here the parentheses denote the pairing of vectors and covectors. The points $x^*$ comprise the dual curve $\gamma^*$ in the dual plane.

Identifying vectors and covectors via Euclidean metric, we see that, for the
ellipse (\ref{eq:ellipse}), $x^* = Ax$. The polar dual curve is the ellipse given by the matrix $A^{-1}$.

The phase space of the billiard is $2$-dimensional; it consists of the inward
unit vectors $u$ with the foot point $x$ on the ellipse. The billiard transformation is shown in Figure \ref{fig:transf}.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-1.mps}
\caption{The billiard transformation: $(x,u) \mapsto (y,v)$.}
\label{fig:transf}
\end{figure}

Here is the key fact needed for our first result.  The quantity
$J$ in this result is known as the  Joachimsthal integral.

\begin{proposition} \label{pr:int}
  The function $J(x,u):=-(u, x^*)$ is invariant under the billiard transformation: $J(x,u)=J(y,v)$.
  \end{proposition}

\begin{proof}
	We claim that $(u, x^*)=-(u, y^*) = (v, y^*)$.
	For the first equality, it is enough to show that $(y-x, x^*)=-(y-x, y^*)$, because $y-x$ is parallel to $u$.
	Since $A$ is self-adjoint operator one has $(y, x^*)=(y,Ax)=(Ay,x)=(x,y^*)$.
	Subtracting from it the equality $(x, x^*)=(y,y^*)=1$ we obtain the required one.
        Since $u$ is collinear with $y-x$, one has $\langle A(x+y),u\rangle=0$, as needed.
	The second equality holds for billiards of every shape. By the law of billiard reflection,
        the vector $u+v$ is tangent to the ellipse at point $y$, hence  $(y^*, u+v)=0$. This  completes the proof.
\end{proof}

To each billiard trajectory $P$ we can associate the quantity $J(P)=J(x,u)$, where $(x,u)$ is the
first point-vector pair associated to $P$.

\begin{corollary}
  $J$ is constant as $P$ varies in a $1$-parameter family of periodic billiard paths in an ellipse.
\end{corollary}

\begin{proof}
  Let $E$ be a given ellipse.
  For the generic choice of caustic $E'$ -- i.e., a choice of ellipse confocal with $E$ -- the billiard
map is aperiodic and has dense orbits.  By the preceding result, $J$ is constant
on a dense set of billiard paths tangent to $E'$.   By continuity,
$J$ is constant on the space of all billiard paths tangent to almost any caustic $E'$.
By continuity, the same result holds for every caustic.
Finally, note that the
polygons in a $1$-parameter family of periodic billiard paths are
tangent to the same caustic.
\end{proof}

Here is the perimeter-invariance result mentioned in the introduction.

\begin{lemma}
\label{perimeter}
  The perimiter of $P$ is
  constant as $P$ varies in a $1$-parameter family of periodic billiard paths in an ellipse.
\end{lemma}

\begin{proof}
A $n$-periodic billiard path
extremizes the perimeter among the $n$-gons inscribed in the billiard table.
The 1-parameter family of periodic polygons is a curve in the space of inscribed polygons consisting of
critical points of the perimeter function. A function has a constant value on a curve of its critical points.
See \cite{tabachnikov2005geometry} for more details.
\end{proof}

The following identity, also noticed experimentally by Reznik,
immediately implies Theorem \ref{thm:sumcos2}.
The reason is that both the quantities $J$ and $L$ are
constant for  $1$-parameter families of periodic
billiard paths in ellipses.

\begin{theorem}
	\label{thm:sumcos1}
For an $n$-periodic billiard trajectory $P$ in an ellipse, one has
\[
\sum_{i=1}^n \cos \alpha_i =  JL -n,
\]
where $L$ and $J$ are the perimeter and the value of the Joachimsthal integral of $P$.
\end{theorem}

\begin{proof}
We compute
\[
L = \sum_i \langle p_{i+1}-p_i,u_i\rangle = \sum_i \langle p_i,u_{i-1}\rangle - \langle p_i,u_i\rangle = \sum_i \langle p_i,u_{i-1}-u_i\rangle
\]
(this is a discrete version of integration by parts). By the law of billiard reflection,
\[
u_{i-1}-u_i = 2 \sin \left(\frac{\pi-\alpha_i}{2}\right) \frac{p_i^*}{|p_i^*|} = 2 \cos \left(\frac{\alpha_i}{2}\right) \frac{p_i^*}{|p_i^*|}.
\]
Also one has
\[
J = -(u_i,p_i^*) = -|p_i^*| \cos \left(\pi-\frac{\alpha_i}{2}\right) =  |p_i^*| \cos \left(\frac{\alpha_i}{2}\right).
\]
Since $(p_i,p_i^*)=1$, it follows that
\[
JL=  \sum_i 2\cos^2 \left(\frac{\alpha_i}{2}\right) =  \sum_i (1+\cos \alpha_i) =
 n + \sum_i \cos \alpha_i,
\]
as needed.
\end{proof}

\begin{remark} \label{rmk:Bialy}
See [B] for a symplectic interpretation of these ideas.
\end{remark}

\section{Dual Minkowski billiards} \label{sect:dual}

Before we get to our complex analysis-based proofs, it is useful to discuss
billiards in Minkowski metrics.  This is used to construct dual billiards, which offer new generalizations and help proving the main results.

A~{\it Minkowski metric\/} is a norm on a vector space.
Let $U$ be an $n$-dimensional vector space and $V=U^*$ be its dual.
 Assume that $U$ and $V$ are equipped with Minkowski metrics, not necessarily centrally-symmetric
and dual to each other. Let $M \subset U$ be the unit co-ball of the metric in
$V$ and $N \subset V$ be he unit co-ball of the metric in $U$.
One has two billiards: $M$ in normed space $U$, and $N$ in normed space $V$.

\begin{lemma}
The two billiards systems are isomorphic.
\end{lemma}

\begin{proof}
This is proved in \cite{gutkin2002billiards}, section 7.  Here we sketch the ideas.
Abusing notation, denote by $\mathcal{D}$ the polar duality that identifies the unit
spheres and co-spheres of the metrics. The phase space of the billiard in $M$ consists of
the pairs $(q,u)$ where $q\in \partial M$ and $u$ is a (Minkowski) unit inward vector at $q$.
Assign to it the pair $(q,p) \in \partial M \times \partial N$ where $p=\mathcal{D}(u)$.
Likewise, the phase space of the billiard in $N$ consists of the pairs $(p,v)$ where
$p\in \partial N$ and $v$ is a (Minkowski) unit inward vector at $p$. Assign to it
the pair $(p,q) \in \partial N \times \partial M$ where $q=\mathcal{D}(v)$.

The assertion  is that a sequence $(\ldots,q_i,p_i,q_{i+1},\ldots)$ corresponds to a
billiard orbit in $M$ if and only if the sequence $(\ldots,p_i,q_{i+1},p_{i+1},\ldots)$ corresponds to a billiard orbit in $N$.
If $M$ is an ellipse with half-axes $a_1$ and $a_2$ in the Euclidean plane $U$, then $N$ is
the unit disc in the plane $V$ whose unit ball is polar to $M$, i.e., is an ellipse with half-axes $1/a_1$ and $1/a_2$.
\end{proof}

\begin{remark}
The affine map $(x_1,x_2)\rightarrow (a_1x_1, a_2x_2)$ isometrically maps the
Minkowski plane $V$ to the Euclidean plane, taking $N$ back to the ellipse with half-axes $a_1$ and $a_2$, that is, to $M$.
 One obtains a symmetry of the billiard in an ellipse, called the skew hodograph
transformation and discovered by A. Veselov  \cite{veselov1988integrable,veselov1991integrable}.
\end{remark}

\begin{remark}
It was noticed in \cite{arstein-avidan2014bounds} that minimal action of a billiard in
Minkowski metric is related with Ekeland--Hofer--Zehnder capacity, which made it possible
to apply symplectic geometry to convex geometry problems. In particular it was shown that
the famous Mahler conjecture followed from the Viterbo's
conjecture~\cite{arstein-avidan2014fromsympelctic, akopyan2016elementary}.
\end{remark}

\section{Another Proof of the First Result}

As before, let $u_1,\ldots,u_n$ be the unit vectors along the sides of $P$. These vectors are points of
the unit circle $\omega$ centered in the origin $O$, and they form an $n$-periodic trajectory
of the Minkowski billiard in $\omega$ with the Minkowski metric defined by the ellipse dual
to the original one. Therefore $u_1,\ldots,u_n$ are the vertices of a Poncelet $n$-gon,
inscribed in $\omega$ and circumscribed about some ellipse $\xi$ centered at $O$.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-7.mps}
\caption{To  Theorems ~\ref{thm:circle-ellipse1} and \ref{thm:circle-ellipse2}.}
\label{inter}
\end{figure}

The angles of $P$  satisfy  $\alpha_i = \pi - \angle u_{i-1} O u_{i}$,
thus
the next result is equivalent to Theorem \ref{thm:sumcos2}.

\begin{theorem}
	\label{thm:circle-ellipse1}
	Let $u_1,\ldots, u_n$ be a Poncelet polygon inscribed in a circle $\omega$ with center $O$ and circumscribed about an ellipse $\xi$ with the same center. Then

\[
\sum_{i=1}^n \cos \angle u_i O u_{i+1}
\]
is constant in the 1-parameter family of Poncelet $n$-gons.
\end{theorem}

\begin{proof}
Following the approach of \cite{schwartz2015pentagram}, we will complexify the situation, that is,
extend the setting to Poncelet polygons on the conics given by the same equations in the complex plane.
We show that the function in question is  bounded, and then the Liouville theorem  implies that this function is constant.
To extend our function to complex plane, we need to represent the function $\cos \angle u_i O u_{i+1}$ in a
more convenient way. Since $|Ou_i|=1$ for all $i$, we have:
\[\cos \angle u_i O u_{i+1}= \langle u_i, u_{i+1} \rangle .\]
In other words, for the proof of the first statement, we need to show that the sum $\langle u_i, u_{i+1} \rangle$ is constant.
Let us emphasize that here we consider the usual dot product, not the Hermitian one.

Consider the standard rational parametrization of the circle $\omega$:
\[
	p(t)=\left(\frac{2t}{t^2+1}, \frac{t^2-1}{t^2+1}\right).
\]
The points at infinity correspond to the value of the parameter $t= \pm I$.
Here $I=\sqrt{-1}$.
It is clear that the only possibility for the Poncelet polygon to have an
infinite $\sum\langle u_i, u_{i+1} \rangle$ is to have one of its vertex at infinity.

Let us show that when a vertex goes to infinity, the inner
products in which this vertex participates cancel each other.

Consider a point at infinity, say,  $p(I)$. We claim that its two neighboring vertices
 of the Poncelet polygon, denoted by $a(I)$ and $b(I)$, are opposite points of $\omega$.
Indeed, the lines $p(I)a(I)$ and $p(I)b(I)$ are tangent to $\xi$ and are parallel,
therefore the tangency points of these lines with $\xi$ are symmetric with respect to $O$,
and hence their intersection points with $\omega$ are also symmetric (the point $p(I)$ is invariant under the reflection in $O$, given by $t \mapsto -1/t$).
Thus, for any finite point $q$ on $\omega$, we have
$\langle q, a(I) \rangle + \langle q, b(I) \rangle=0.$

Now, consider point $p(t+I)$ with $t$ tending to zero and its neighboring
vertices $a(I+t)$ and $b(I+t)$ of the Poncelet polygon. Notice that $p(t+I)$ tends to
infinity as $O(1/t)$, while $a(t+I)$, $b(t+I)$ tend to their limit $a(I)$, $b(I)$ linearly.
Furthermore, due to the symmetry,
as $t$ goes to zero, the linear in $t$ terms are vectors with the same absolute value and the opposite directions:
\[
	a(t+I)=a(I)+\vec k  \cdot t +O(t^2), \ \
	b(t+I)=-a(I)-\vec k  \cdot t +O(t^2).
\]
Now we can bound above the sum for small $t$:
\begin{multline*}
	\label{eq:sum of neighbors}
	\langle
	p(t+I), a(t+I)
	\rangle
	+
	\langle
	p(t+I), b(t+I)
	\rangle
	=
	\langle
	p(t+I),a(t+I)+b(t+I)
	\rangle
	=\\
	\langle
	O(1/t), O(t^2)
	\rangle= O(t).
\end{multline*}
That is, the sum tends to zero as $t$ goes to $0$, and therefore it is bounded.
\end{proof}

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-9.mps}
\caption{The behavior of the polygon at infinity.}
\label{infin}
\end{figure}
\medskip

\section{Proof of the Second Result}

Referring to the construction in the previous section,
the angles $\beta_i$ in Figure \ref{fig:prod} are given by the formula
\[
\beta_i = \pi - \frac{\angle u_{i-1} O u_{i+1}}{2}=\angle u_{i-1}u_{i}u_{i+1}.
\]

\begin{theorem}
	\label{thm:circle-ellipse2}
	Let $u_1,\ldots, u_n$ be a Poncelet polygon inscribed in a circle $\omega$ with center $O$ and circumscribed about an ellipse $\xi$ with the same center. Then
\[
\ \prod_{i=1}^n \cos  \angle u_{i-1}u_{i}u_{i+1}
\]
is constant in the 1-parameter family of Poncelet $n$-gons.
\end{theorem}

\begin{proof}
Since the product of cosines changes continuously,
if the absolute value of this product is constant in the family, then its sign is fixed as well.
Therefore, instead of the product of cosines, we may consider the product of their squares:
\[
\cos^2 \angle u_{i-1} u_{i} u_{i+1}= \cos^2\frac{\angle u_{i-1}Ou_{i+1}}{2}=\frac{1+ \cos {\angle u_{i-1}Ou_{i+1}}}{2} = \frac{1+\langle u_{i-1}, u_{i+1} \rangle}{2}.
\]
Thus we need to prove that the product $\prod (1+\langle u_{i-1}, u_{i+1} \rangle)$ is bounded.
Again the only possibility for this product to be infinite is when one of the vertices goes to infinity.

Similarly to the previous proof, and using the same parameterization of the circle,
we assume that $p(t+I)$ is the vertex that goes to infinity, $a(t+I)$ and $b(t+I)$ are its neighboring vertices, and $a'(t+I)$ and $b'(t+I)$ are its second removed neighbors (which are also centrally symmetric, but it plays no role here).

Let us show that the corresponding product is bounded:
\begin{multline*}
	\label{eq:sum of neighbors}
(1+ \langle 	p(t+I), a'(t+I) \rangle)
(1+ \langle 	p(t+I), b'(t+I) \rangle)
(1+ \langle 	a(t+I), b(t+I) \rangle)=\\
O(t^{-1})\cdot O(t^{-1})\cdot (1+\langle a(I)+\vec k
\cdot t +O(t^2), -a(I)-\vec k  \cdot t +O(t^2) \rangle)=\\
O(t^{-2})\cdot(1+\langle a(I), -a(I)\rangle -2 \langle a(I), \vec k \cdot t \rangle + O(t^2))=\\
O(t^{-2})\cdot(-2 \langle a(I), \vec k \cdot t \rangle + O(t^2))=
O(t^{-2})\cdot(-2 \langle a(I), \vec k \cdot t \rangle )+ O(1).
\end{multline*}
It is left to notice that $\vec k$ is tangent to $\omega$ at $a(I)$,
therefore $\langle a(I), \vec k \cdot t \rangle =0.$
Thus  the product is bounded.

If $n$ is  odd, then only one vertex can go to infinity: if $p(\pm I)$ is a vertex of the Poncelet polygon, then $p(\mp I)$ is not its vertex.

For even $n$, a vertex $u_i$ goes to infinity simultaneously with $u_{i+n/2}$.
A simple combinatorial analysis of the configurations shows that the only case when
the above considered factors  coincide is when $n=4$.
In that case the polygon is always a rectangle, and the statement is obvious.
\end{proof}

\section{Variants and Generalizations} \label{var1}

Here we list some variants and generalizations of the results we
have proved so far.
\newline
\newline
{\bf Multi-dimensional version:\/}
{\rm The billiard inside an ellipsoid in ${\mathbb R}^{n+1}$ is also completely integrable: the phase space is $2n$-dimensional,
the trajectories are confined to $n$-dimensional tori, and the motion on these tori is quasi-periodic,
see \cite{dragovic2011poncelet}. In particular, if a point is periodic, then all points of
the torus are periodic with the same period, and the respective polygons have the same perimeters.
The billiard map still has the Joachimsthal integral, constant on the orbits confined to an
invariant torus, and the above arguments  go through, proving a multi-dimensional version of Theorem \ref{thm:sumcos2}.
\newline
\newline
\noindent
{\bf Sizes of the Angles:\/}
Concerning Theorem \ref{thm:prodcos},
it was pointed out to us by M.~Bialy that this theorem implies
that the sign of the quantity $\beta_i - \pi/2$ remains fixed during the rotation of the polygon.
 This  is consistent with the fact that all vertices of $Q$ lie on a ellipse polar dual to
the inner ellipse (the caustic) with respect to the outer one. If we fix this outer ellipse
and vary the caustic, then these polar dual ellipses, into which the polygons $Q$ are inscribed,
form a pencil of conics. This pencil contains the \emph{orthoptic circle}, the locus of points
from which an ellipse is seen under the right angle. This orthoptic circle separates
the two cases: when all angles $\beta_i$ are obtuse and when they are all acute.
\newline
\newline
{\bf Additional Invariants:\/}
The following theorem is a generalization of Theorem \ref{thm:sumcos2}, which is its $k=1$ case. We explain how to deduce this theorem from Theorem \ref{thm:sumcos2}.

\begin{theorem} \label{thm:gensumcos}
For each $k=1,...,n$, the quantity
\[
C_k = \sum_{i=1}^n \cos (\alpha_i + \alpha_{i+1} + \ldots + \alpha_{i+k-1})
\]
remains constant as $P$ varies a 1-parameter family of periodic billiard paths on an ellipse.
\end{theorem}

\begin{figure}[hbtp]
\centering
\includegraphics[height=3in]{cosines-fig-2.mps}
\caption{Four $13$-periodic billiard orbits in confocal ellipses tangent to the same caustic.}
\label{pentagons}
\end{figure}

\begin{proof}
To line up our proof with a previously published result that we use, we
state things in terms of Poncelet polygons and the Poncelet porism.
Label the lines containing the sides of a Poncelet $n$-gon cyclically.
Fix $k$ and consider
the intersections of $i$th and $(i+k)$th lines, where $i=1,2,\ldots,n$. This set of points
lies on a confocal ellipse and comprises several polygons, each a periodic billiard trajectory
(the number of polygons equals gcd $(n,k)$).  This statement is a part of the Poncelet
Grid theorem \cite{Schwartz2007poncelet,levi2007poncelet}. Figure~\ref{pentagons} illustrates the case of $n=13$.
The angles of these new polygons are expressed via the angles of the original one. Namely, the new angles are equal to
\[
\alpha_i + \alpha_{i+1} + \ldots + \alpha_{i+k-1} - (k-1)\pi.
\]
Therefore Theorem \ref{thm:sumcos2}, applied to the new polygons, implies that $C_k$ remains constant in their Poncelet family.
\end{proof}

Theorem \ref{thm:gensumcos} has a reformulation, generalizing Theorem \ref{thm:circle-ellipse1}, which is the $k=1$ case.

\begin{theorem}
	\label{thm:gensumcos2}
	Let $u_1,\ldots, u_n$ be a Poncelet polygon inscribed in a circle $\omega$ with center $O$ and circumscribed about an ellipse $\xi$ with the same center. Then, for each $k$,
\[
\sum_{i=1}^n \cos \angle u_i O u_{i+k}
\]
are constant in the 1-parameter family of Poncelet $n$-gons.
\end{theorem}

\noindent
{\bf Sums of Squared Lengths:\/}
Here is a  corollary of Theorem \ref{thm:gensumcos2}.
Note that \[|u_{i+k}-u_i|^2= 2-2 \cos \angle u_i O u_{i+k}.\] This implies

\begin{corollary}
	\label{cor:sides}
Consider a Poncelet polygon inscribed in a circle and circumscribed about a concentric ellipse.
Then the sum of the squared  lengths of its $k$-diagonals remains constant in the 1-parameter family of Poncelet polygons.
\end{corollary}

\noindent
{\bf Product of sines of half-angles:\/}
Notice that the angles of polygons formed by the tangents can be
represented through angles of the billiard trajectory:
\[
\beta_i=\frac{\alpha_i+\alpha_{i+1}}{2}.
\]
Therefore, the statement of Theorem~\ref{thm:prodcos} can be formulated in terms of
the angles of the billiard trajectory and then can
be extended for the angles of lines in the corresponded
Poncelet grid as in Theorem~\ref{thm:gensumcos}.
Doing this for odd $n$ and the polygon formed by sides of our trajectory
 with step $k=(n-1)/2$, we find that the corresponding angle of
the tangential polygon $\beta'_i$ equals
\[
\beta'_i=\frac{\sum_{j\ne i} \alpha_j}{2}=\frac{\pi(n-2)-\alpha_i}{2}.
\]
Since $n$ is odd we get
\[
\cos \beta'_i=\cos\frac{\pi(n-2)-\alpha_i}{2}= \pm \cos \frac{\pi-\alpha_i}{2} =\pm\sin \frac{\alpha_i}{2}.
\]

This gives us the following result, also noticed by D.~Reznik:
\begin{corollary}
	\label{cor:product of halfsines}
	For odd $n$, the quantity
	\[
	\prod_{i=1}^n \sin \frac{\alpha_i}{2}
	\]
	remains constant as $P$ varies in a 1-parameter
        family of periodic billiard paths on an ellipse.
\end{corollary}

\noindent
{\bf Hyperbolic Interpretation:\/}
One can interpret Theorems \ref{thm:gensumcos} and \ref{thm:circle-ellipse2} in terms of hyperbolic geometry.
Consider $\omega$ as the absolute of the Klein model of the hyperbolic plane, and $\xi$ as
an ellipse in it. The  hyperbolic and the Euclidean measures of the angles $u_i O u_k$
coincide. We obtain the following corollary.

\begin{corollary}
	\label{cor:poncelet}
	Let $u_1,\ldots, u_n$ be an ideal $n$-gon in the hyperbolic plane whose sides are tangent to an ellipse with center $O$. Then, for each $k$,
\[
\sum_{i=1}^n \cos \angle u_i O u_{i+k}\ \ {\rm and} \ \ \prod_{i=1}^n \cos  \left(\frac{\angle u_{i-1} O u_{i+1}}{2}\right)
\]
are constant in the 1-parameter family of ideal Poncelet $n$-gons.
\end{corollary}

\noindent
{\bf A Dual Version:\/}
Under the  duality transform with respect to the circle $\omega$, the inner ellipse $\xi$ goes to a
concentric ellipse $\xi^*$, and we again obtain a Poncelet polygon, this time inscribed in $\xi^*$ and
circumscribed about $\omega$. The angles between the unit vectors $u_i$ become the angles between
the sides of the Poncelet polygon, and we obtain the following corollary of Theorems \ref{thm:circle-ellipse1}
and \ref{thm:circle-ellipse2}.

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-8.mps}
\caption{To Corollary~\ref{cor:ellipse-circle}. }
\label{exter}
\end{figure}

\begin{corollary}[Ellipse-Circle version]
	\label{cor:ellipse-circle}
	Let $v_1,\ldots, v_n$ be a Poncelet polygon circumscribed about a circle and inscribed in a concentric ellipse. Denote its angles by $\alpha_i$. Then
\[
\sum_{i=1}^n \cos \alpha_i\ \ {\rm and} \ \ \prod_{i=1}^n \cos  \angle v_{i}Ov_{i+1}
\]
are constant in the 1-parameter family of  Poncelet $n$-gons.
\end{corollary}

\section{Proof of the Third Result}

Theorem \ref{thm:areas} is closely related to the
experimental observation of D. Reznik illustrated in Figure \ref{grid}.
Consider an $n$-periodic billiard trajectory in an ellipse with odd $n$ (the pentagon $Q$ in Figure~\ref{grid}).
The tangent lines to the ellipse at these  $n$ points form a new $n$-gon (the pentagon $P$ in Figure~\ref{grid}).
The observation is that {the ratio of the areas of these two polygons remains
constant as the $n$-periodic billiard trajectory varies in its 1-parameter family.}

\begin{figure}[hbtp]
\centering
\includegraphics{cosines-fig-5}
\caption{The ratio of the areas of the polygons $P$ and $Q$ remains constant in the 1-parameter family of Poncelet polygons.}
\label{grid}
\end{figure}

There exists an affine transformation that makes the two ellipses confocal: first turn one of
the ellipses into a circle by an affine transformation, and then stretch along the axes of the other ellipse to make the two confocal.
Since an affine transformation does not affect the ratio of the areas, it remains
to prove the claim for a Poncelet polygon on confocal ellipses, see Figure \ref{grid}.
This is the case observed by Reznik.

Now we deal with the confocal case.
The Poncelet grid theorem \cite{levi2007poncelet} implies that the affine transformation
that takes the inner ellipse to the outer one by scaling its main axes and reflecting
in the origin takes the inner polygon to the outer one as well (it maps each vertex
to the ``opposite one", see the labelling in Figure \ref{grid}). Since the ratio of the areas is invariant under an affine transformation, the result follows.

\bigskip
{\bf Acknowledgements}. This paper would not be written if not for Dan Reznik's curiosity and
persistence; we are very grateful to him. We also thank R. Garcia and J. Koiller for interesting discussions.
It is a pleasure to thank  the Mathematical Institute
of the University of Heidelberg for its stimulating atmosphere. ST thanks M.~Bialy
for interesting discussions and the Tel Aviv University for its invariable hospitality.

AA was supported by European Research Council (ERC) under the European Union's
Horizon 2020 research and innovation programme (grant agreement No 78818 Alpha).
RS is supported by NSF Grant DMS-1807320.
ST was supported by NSF grant DMS-1510055 and SFB/TRR 191.

\bibliographystyle{abbrvurl}

Metric Geometry (math.MG); Differential Geometry (math.DG); Dynamical Systems (math.DS)

Billiards in ellipses revisited


An Equivalence between Bayesian Priors and Penalties in Variational Inference

Abstract: In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by an ad hoc regularization term that penalizes some values of the parameters. Regularization terms appear naturally in Variational Inference (VI), a tractable way to approximate Bayesian posteriors: the loss to optimize contains a Kullback--Leibler divergence term between the approximate posterior and a Bayesian prior. We fully characterize which regularizers can arise this way, and provide a systematic way to compute the corresponding prior. This viewpoint also provides a prediction for useful values of the regularization factor in neural networks. We apply this framework to regularizers such as L2, L1 or group-Lasso.

\section{Introduction}

Adding a penalty term to a loss, in order to make the trained model fit some user-defined property, is very
common in machine learning. For instance, penalties are used to improve generalization, prune neurons
or reduce the rank of tensors of weights.
Therefore, usual penalties are mostly empirical and user-defined, and integrated to the loss as follows:
\begin{align*}
L(\mathbf{w}) &= \ell(\mathbf{w}) + r(\mathbf{w}) ,
\end{align*}
with $\mathbf{w}$ the vector of all parameters in the network, $\ell(\mathbf{w})$ the error term and
$r(\mathbf{w})$ the penalty term.

From a Bayesian point of view, optimizing such a loss $L$ is equivalent to finding the Maximum A Posteriori
(MAP) of the parameters $\mathbf{w}$ given the training data and a prior $\alpha \propto \exp (- r)$.
Indeed, assuming that the loss $\ell$ is a log-likelihood loss, namely,
$\ell(\mathbf{w})=-\ln p_\mathbf{w}(\mathcal{D})$ with dataset
$\mathcal{D}$, then
minimizing $L$ is equivalent to minimizing $
L_{\mathrm{MAP}}(\mathbf{w}) = -\ln p_\mathbf{w}(\mathcal{D}) -
\ln(\alpha(\mathbf{w}))$.
Thus, within the MAP framework, we can interpret the penalty term $r$ as
the influence of a prior $\alpha$~\citep{mackay1992practical}.

However,
the MAP approximates the Bayesian posterior very roughly, by taking its maximum. Variational
Inference (VI) provides a \emph{variational posterior} distribution rather than a
single value, hopefully representing the Bayesian posterior much better.
VI looks for the best posterior approximation within a family
$(\beta_{\mathbf{u}}(\mathbf{w}))_{\mathbf{u}}$ of approximate posteriors over
$\mathbf{w}$, parameterized
by a vector $\mathbf{u}$. For instance, the weights $\mathbf{w}$ may be
drawn from a Gaussian distribution with mean $\mathbf{u}$ and fixed
variance. The loss to be minimized over $\mathbf{u}$ is then:
\begin{align}
\label{eq:introVI}
L_{\mathrm{VI}}(\beta_{\mathbf{u}}) & \;=\; -\mathbb{E}_{\mathbf{w} \sim
		\beta_{\mathbf{u}}} \ln
		p_{\mathbf{w}}(\mathcal{D}) \,+\,
\mathrm{KL}(\beta_{\mathbf{u}} \| \alpha) \nonumber \\
&\;=\; \underbrace{\mathbb{E}_{\mathbf{w} \sim
		\beta_{\mathbf{u}}} \ell(\mathbf{w})
		}_{\text{data fit term}} \,+\,
\underbrace{ \mathrm{KL}(\beta_{\mathbf{u}} \| \alpha)}_{\text{penalty term}} ,
\end{align}
and is also an upper bound on
 the Bayesian negative log-likelihood of the data~\citep{jordan1999introduction}.
Here $p_{\mathbf{w}}(\mathcal{D})$ is the likelihood of the full dataset
$\mathcal{D}$ given $\mathbf{w}$, $\ell(\mathbf{w})=-\ln
p_{\mathbf{w}}(\mathcal{D})$ is the log-likelihood
loss,
and $\alpha$ is the Bayesian prior. The posteriors
$\beta_{\mathbf{u}}$ that minimize this loss will be concentrated around
values of $\mathbf{w}$ that assign high probability to the data, while
not diverging too much from the prior $\alpha$.
Thus, the KL divergence term can be seen as a penalty $r(\cdot)$ over the
vector $\mathbf{u}$.

\paragraph{Contributions and outline.} We start from the following
question:
given some arbitrary penalty $r(\cdot)$, does it admit such an
interpretation? Does there exist a prior $\alpha$
such that for all $\mathbf{u}$, $r(\mathbf{u}) =
\mathrm{KL}(\beta_{\mathbf{u}} \| \alpha)$ (up to an
additive constant)?	If so, is there a systematic way to compute such $\alpha$?

First, we provide a necessary and sufficient condition
(Theorem~\ref{thm:main}) over the penalty $r$,
that
ensures the existence of a prior $\alpha$ such that VI with prior $\alpha$
reproduces the penalty $r$. The theorem comes with an explicit
formula for $\alpha$. We recover the MAP case as a degenerate case
(Section~\ref{sec:appl_2}).

Thus, we are able to determine whether a penalty $r$ makes sense in a
Bayesian framework and can be interpreted as the influence of a prior. We
find this to be a strong constraint on $r$ (Section~\ref{sec:appl_1}).
Here the regularizer $r$ operates on the variational posterior
$\beta_\mathbf{u}$; for deterministic $\beta$ this reduces to $r$ directly acting on $\mathbf{w}$, and we
recover the traditional MAP correspondence in this case (Section~\ref{sec:appl_2}).

Second (Section~\ref{sec:NN_appl}), we propose a heuristic to predict a priori useful values of the penalty
factor $\lambda$ to be put in front of a penalty $r$ for neural networks, potentially
bypassing the usual hyperparameter search on $\lambda$. Namely, we posit
that the Bayesian prior $\alpha(\mathbf{w})$ corresponding to $\lambda r$ should reasonably
match what is known for good a priori values of weights $\mathbf{w}$ in neural
networks, namely, that the variance of weights under the prior $\alpha$
should match the Glorot initialization procedure
\citep{glorot2010understanding}. This usually provides a specific value of
$\lambda$.
Moreover, the
penalty size gets automatically adjusted depending on the width
of the various layers in the network.

We test this prediction for various penalties
(Section~\ref{sec:penkl_expes}), including
group-Lasso~\citep{yuan2006model}, for which per-layer adjustment of the
penalty is needed~\citep{alvarez2016learning}. Experimentally, the predicted value
of the regularization factors leads to reasonably good results without
extensive hyperparameter search. Still, the optimal penalization factor
is found to be systematically about $0.01$ to $0.1$ times our predicted value,
showing that our heuristic provides a usable order of magnitude but not
a perfect value, and suggesting that the Bayesian VI viewpoint may
over-regularize.

\section{Related Work}

Interpretations of existing empirical deep learning methods
in a Bayesian variational inference framework include, for instance,
\emph{variational drop-out}
\citep{kingma2015variational},
 a version of
drop-out which fits the Bayesian framework. Further developments of variational drop-out have been made
by \cite{molchanov2017variational} for weight pruning and by \cite{louizos2017bayesian} for neuron pruning.

Closer to our present work, the links between a penalized loss and the Bayesian point of view have previously
been mentioned by \cite{olshausen1997sparse} with a few approximations.
\cite{mackay1992practical} noted the equivalence  of the penalized loss
$L(\mathbf{w}) = \ell(\mathbf{w}) + r(\mathbf{w})$  and the MAP loss
$L_{\mathrm{MAP}}(\mathbf{w}) = \ell(\mathbf{w}) -
\ln(\alpha(\mathbf{w}))$
when $\ell(\mathbf{w})$ is the negative log-likelihood of the training dataset given the weights $\mathbf{w}$,
and with a prior $\alpha(\mathbf{w}) \propto \exp(- r(\mathbf{w}))$.
That is, finding the vector $\mathbf{\hat{w}}$ minimizing a loss $L$ can be equivalent to finding the MAP
estimator $\mathbf{\hat{w}}_{\mathrm{MAP}}$ by minimizing the loss $L_{\mathrm{MAP}}$ with a well-chosen prior
distribution $\alpha$.

However, the MAP framework is not completely satisfying from a Bayesian point of view: instead of returning a
distribution over the weights, which contains information about their uncertainty, it returns the most
reasonable  value. In order to evaluate this uncertainty, \cite{mackay1992bayesian} proposed a second-order
approximation of the
Bayesian posterior. In the process, \cite{mackay1995probable} also proposed a complete Bayesian
framework and interpretation of neural networks. Still, this approximation of the
Bayesian posterior is quite limited.

In the same period, \cite{hinton1993keeping} applied the Minimum Description Length
(MDL) principle to neural networks. Then, \cite{mackay2003information} made the
link between the MDL principle and variational inference, and \cite{graves2011practical} applied it to
neural networks, allowing for variational approximations of the Bayesian
posterior in a tractable way.

\paragraph{Motivation.} There exist several results about the Bayesian or the variational
posterior. For instance, a series of recent
papers~\cite{alquier2020concentration,cherief2020convergence,zhang2020convergence} provide theorems
about the convergence speed of the variational posterior, according to the size of the training set.
Therefore, translating user-specified penalties into Bayesian priors is a way to fit a framework in which
several results are already known. In particular, we recall in Appendix~\ref{sec:conv_rate} a theorem stated
in~\cite{zhang2020convergence}, which can be applied in our setup.

Moreover, using variational inference is a way of approximating the Bayesian posterior with several
advantages. First, as recalled in~\cite{knoblauch2019generalized}, the penalty used in variational inference
is optimal for approximating the true Bayesian posterior. That is, any other penalty performs worse. Second,
variational inference allows us to compute the variational posterior by gradient descent, which is a major
advantage when the model to train is a neural
network.

\section{Variational Inference} \label{sec:varinf}

We include here a reminder on variational inference for neural networks,
following \citep{graves2011practical}.

From a Bayesian viewpoint, we describe the vector of weights $\mathbf{w} \in \mathbb{R}^N$ of a neural
network as a random variable. Given a dataset $\mathcal{D}$, we denote by
$p_{\mathbf{w}}(\mathcal{D})$ the probability given to $\mathcal{D}$
 by the network with parameter
$\mathbf{w}$.
For instance, with a dataset $\mathcal{D} = \{(x_1, y_1),
\cdots , (x_n, y_n) \}$ of $n$ input-output pairs and a model that
outputs (log-)probabilities
$p_{\mathbf{w}}(y_i | x_i)$ for the outputs, then $\ln
p_{\mathbf{w}}(\mathcal{D})=\sum_{i=1}^n \ln p_{\mathbf{w}}(y_i | x_i)$
is the total log-likelihood of the data given the model.

Given the dataset $\mathcal{D}$, the posterior distribution
over weights $\mathbf{w}$ is:
\begin{align*}
\pi_{\mathcal{D}}(\mathbf{w}) = \frac{p_{\mathbf{w}}(\mathcal{D}) \alpha(\mathbf{w})}{\mathbb{P}(\mathcal{D})}
,\quad
\mathbb{P}(\mathcal{D})=\int_\mathbf{w}
\alpha(\mathbf{w})p_\mathbf{w}(\mathcal{D})
\end{align*}
which is analytically intractable for multi-layer nonlinear neural networks.
However, the posterior $\pi_D$ can be approximated by looking for
probability distributions $\beta$ that minimize the loss
\begin{align}
L_{\mathrm{VI}}(\beta) =
- \mathbb{E}_{\mathbf{w} \sim \beta} \ln
p_{\mathbf{w}} (\mathcal{D}) + \mathrm{KL}(\beta \| \alpha)  \label{eqn:varinf_base} ,
\end{align}
where
$\mathrm{KL}(\beta \| \alpha) = \int_{\mathbb{R}^N} \ln\left( \frac{\beta(\mathbf{w})}{\alpha(\mathbf{w})}
\right) \beta(\mathbf{w}) \, \mathrm{d} \mathbf{w}$ is the
Kullback--Leibler divergence.
Indeed, one has $L_{\mathrm{VI}}(\beta) = -\ln
\mathbb{P}(\mathcal{D})+\mathrm{KL}(\beta\|\pi_{\mathcal{D}})$, which is
minimal when $\beta=\pi_{\mathcal{D}}$.

The first term in the loss \eqref{eqn:varinf_base} represents the
error made over the dataset $\mathcal{D}$: it is small if $\beta$ is
concentrated around good parameters $\mathbf{w}$.
The second term
can be seen as a user-defined penalty over $\beta$ that keeps it
from diverging too much from the prior $\alpha$.
Moreover, for any distribution $\beta$, the
quantity $L_{\mathrm{VI}}(\beta)$ is a bound on the Bayesian
log-likelihood of the data: $L_{\mathrm{VI}}(\beta)\geq -\ln \int_\mathbf{w}
\alpha(\mathbf{w})p_\mathbf{w}(\mathcal{D})$
\citep{jordan1999introduction}.

In variational inference, a parametric family $\mathcal{B}$ of probability
distributions $\beta$ is fixed, and one looks for the best approximation
$\beta^*$ of the Bayesian posterior in $\mathcal{B}$ by minimizing
$L_{\mathrm{VI}}(\beta)$ in this family: the \emph{variational posterior}. Importantly, for some families
such as Gaussians with fixed variance, the gradient of
$L_{\mathrm{VI}}(\beta)$ can be computed if the gradients of $\ln
p_\mathbf{w}(\mathcal{D})$ can be computed \citep{graves2011practical}, so that
$L_{\mathrm{VI}}(\beta)$ can be optimized by stochastic gradient descent. Thus, this is well-suited for
models such as neural networks.

Thus, we consider a parametric family $\mathcal{B} = \{\beta_{\mathbf{u}}
: \mathbf{u} \in
\mathbb{R}^P\}$ with parameter $\mathbf{u}$, where each
$\beta_{\mathbf{u}}$ is a probability distribution over $\mathbf{w} \in
\mathbb{R}^N$.
Then we learn the parameters $\mathbf{u}$ instead of the weights $\mathbf{w}$. For instance,
we can choose one of the following families of variational posteriors.

\begin{example} \label{ex:gaussian}
	The family of products of Gaussian distributions over $\mathbf{w} = (w_1, \cdots,
	w_N)$:
	\begin{align*}
	\beta_{\mathbf{u}} &= \beta_{(\mu_1, \sigma_1^2, \cdots , \mu_N, \sigma_N^2)} \\
	&= \mathcal{N}(\mu_1, \sigma_1^2) \otimes \cdots \otimes \mathcal{N}(\mu_N, \sigma_N^2) .
	\end{align*}
	In this case, the weights of the neural network are random and independently sampled from different
	Gaussian distributions $\mathcal{N}(\mu_k, \sigma_k^2)$. Instead of learning them directly, the vector of
	parameters $\mathbf{u} = (\mu_1, \sigma_1^2, \cdots , \mu_N,
	\sigma_N^2)$ is learned to minimize
	$L_{\mathrm{VI}}(\beta_\mathbf{u})$.
\end{example}

\begin{example} \label{ex:dirac}
	The family of products of Dirac distributions over $\mathbf{w} = (w_1, \cdots,
	w_N)$:
	\begin{align*}
	\beta_{\mathbf{u}} = \beta_{(\mu_1, \cdots , \mu_N)} = \delta_{\mu_1} \otimes \cdots \otimes
	\delta_{\mu_N}.
	\end{align*}
	In this case, the weights are deterministic: $w_k$ and $\mu_k$ are identical for all $k$.
\end{example}

\section{Bayesian Interpretation of Penalties} \label{sec:penalty_KL}

\subsection{When Can a Penalty Be Interpreted as a Prior?}

In this section, we provide a necessary and sufficient condition that
ensures that a penalty $r(\mathbf{u})$ over the parameters of a
variational posterior can be
interpreted as a Kullback--Leibler divergence with respect to a prior
$\alpha$; namely, that
\begin{align*}
\exists \, K\in \mathbb{R},\, \forall \mathbf{u}, \quad r(\mathbf{u}) = \mathrm{KL}(\beta_{\mathbf{u}} \|
\alpha) + K
\end{align*}
(the constant $K$ does not affect optimization).
In the process, we give a
formula expressing $\alpha$ as a function of $r(\cdot)$.

In a nutshell, we place ourselves in the framework of variational inference: we assume the vector of weights
$\mathbf{w}$ of the probabilistic model to be a random variable, drawn from a learned distribution
$\beta_{\mathbf{u}}$ parameterized by a vector $\mathbf{u}$. For instance, the vector of weights $\mathbf{w}$
can be drawn from a multivariate normal distribution $\beta_{\mu, \mathbf{\Sigma}} \sim \mathcal{N}(\mu,
\mathbf{\Sigma})$.

We use some notions of distribution theory. We provide a reminder in
	Appendix~\ref{sec:distr}.

\paragraph{Notation.} In order to approximate the posterior distribution of a vector
$\mathbf{w} \in \mathbb{R}^N$, we denote by $(\beta_{\mu, \bm{\nu}})_{\mu, \bm{\nu}}$ the family of
variational posteriors over $\mathbf{w}$, parameterized by its mean $\mu$ and a vector of additional parameters
$\bm{\nu}$. The basic example is a multivariate Gaussian distribution $\beta_{\mu, \bm{\nu}}$
parameterized by its mean $\mu \in \mathbb{R}^N$ and its covariance matrix $\bm{\nu} = \mathbf{\Sigma} \in
\mathcal{M}_{N, N}({\mathbb{R}})$.

We say that the family $(\beta_{\mu, \bm{\nu}})_{\mu, \bm{\nu}}$ of
variational posteriors is \emph{translation-invariant} if
$\beta_{\mu, \bm{\nu}}(\theta) = \beta_{0, \bm{\nu}}(\theta - \mu)$ for
all $\mu, \bm{\nu}, \theta$.

We denote by $r(\mu, \bm{\nu}) = r_{\bm{\nu}}(\mu)$ some penalty, to be applied to the distribution
$\beta_{\mu, \bm{\nu}}$.

We denote by $\mathcal{F}$ the Fourier transform, given by
$(\mathcal{F}\varphi)(\xi) := \int_{\mathbb{R}^N}
\varphi(x) e^{-i \xi \cdot x} \, \mathrm{d} x$ for $\varphi \in
\mathcal{L}^1(\mathbb{R}^N)$.
This definition extends to the class of tempered
distributions $\mathcal{S}'(\mathbb{R}^N)$ (see
Appendix~\ref{sec:distr}).

In the sequel, we always restrict ourselves to priors
$\alpha \in \mathcal{T}(\mathbb{R}^N) = \{\alpha\; \text{ s.t. } \ln(\alpha) \in \mathcal{S}'(\mathbb{R}^N) \}$, i.e.\ log-tempered probability distributions,
hence the
condition $\ln(\alpha)\in \mathcal{S}'(\mathbb{R}^N)$ in the results
below. This provides a reasonable behavior of $\alpha$ at infinity.

\begin{definition} \label{def:A}
	We define the following distribution over $\mathbb{R}^N$:
	\begin{align}
	A_{\bm{\nu}} \; := \; - \mathrm{Ent}(\beta_{0, \bm{\nu}})\, \mathbbm{1} \, - \, \mathcal{F}^{-1}\left[
	\frac{\mathcal{F} r_{\bm{\nu}}}{\mathcal{F} \check{\beta}_{0, \bm{\nu}}} \right] , \label{eqn:main}
	\end{align}
	where $\check{\beta}_{0, \bm{\nu}}(\theta) := \beta_{0, \bm{\nu}}(-\theta)$ and $\mathbbm{1}$ is the
	constant function equal to $1$. We say that
	\begin{align*}
	r \text{ fulfills~}\condA{}

	\Leftrightarrow
	\left\lbrace
	\begin{array}{l}
	A_{\bm{\nu}} \text{ does not depend on } \bm{\nu}\text{, i.e.\ } A_{\bm{\nu}} = A \text{;} \\
	A \text{ is a function s.t.\ } \int \exp (A) < \infty \text{.}
	\end{array}
	\right.
	\end{align*}
\end{definition}

\begin{theorem}
	\label{thm:main}
	Let $(\beta_{\mu, \bm{\nu}})_{\mu, \bm{\nu}}$ be a
	translation-invariant family of
	variational posteriors.
	Let $r(\mu, \bm{\nu}) = r_{\bm{\nu}}(\mu)$ be a penalty over $\beta_{\mu,
		\bm{\nu}}$.

	We assume that $\forall \bm{\nu}$, the probability distribution
	$\beta_{0, \bm{\nu}}$ has finite entropy and lies in the Schwartz
	class $\mathcal{S}(\mathbb{R}^N)$;
	that $\mathcal{F} \beta_{0, \bm{\nu}}$ is nonzero everywhere;
	and that $\mathcal{F} r_{\bm{\nu}} \in
	\mathcal{E}'(\mathbb{R}^N)$, the class of distributions with
	compact support.

	We are looking for probability distributions $\alpha$ such that:
	\begin{align}
	\exists\, K \in \mathbb{R} : \forall (\mu, \bm{\nu}), \quad r_{\bm{\nu}}(\mu) =
	\mathrm{KL}(\beta_{\mu, \bm{\nu}} \| \alpha) + K . \label{eqn:main_eqn}
	\end{align}

	We have the following equivalence:
	\begin{align*}

	&\alpha \text{ is a solution to~\eqref{eqn:main_eqn}, with}
        \;\alpha \in \mathcal{T}(\mathbb{R}^N) \\

	\Leftrightarrow \;\;

	&r \text{ fulfills } \condA{} \text{ and }
	\alpha = \frac{1}{\kappa} \exp(A),

	\end{align*}
	where $A$ is defined in Equation~\eqref{eqn:main}.

\end{theorem}
The proof is given in Appendix~\ref{sec:proof}. It is based on the resolution of a classical integral equation
(\cite{polyanin1998handbook}, Section 10.3-1) adapted to the wider framework of distribution theory. This
extension is necessary, since the Fourier transform of the widely-used $\mathcal{L}^2$ penalty ($r(x) = x^2$)
cannot be expressed as a function.

The preceding result holds under some technical assumptions.
Even if some of the technical assumptions fail, the formula is still useful to compute a candidate prior $\alpha$
from a penalty $r(\cdot)$: apply
Equation~\eqref{eqn:main} on a penalty $r$ to compute $A_{\bm{\nu}}$,
then check that Condition~\condA{} holds, define $\alpha =
\frac{1}{\kappa} \exp (A)$, and finally compute
$\mathrm{KL}(\beta_{\mu, \bm{\nu}} \| \alpha)$ analytically and compare
it to $r$.

\begin{remark}[Condition $\alpha \in \mathcal{T}(\mathbb{R}^N)$] \label{rem:alpha}
	We can easily prove that most of common distributions belong to this set. For instance, for any $P$ and
	$Q$ polynomials or quotient of polynomials:
	\begin{align*}
	\alpha(\theta) = P(\theta) e^{Q(\theta)} \;\Rightarrow\; \alpha \in \mathcal{T}(\mathbb{R}^N) .
	\end{align*}
	Thus, $\mathcal{T}(\mathbb{R}^N)$ includes distribution families as Gaussian, Cauchy (which is
	heavy-tailed), Gamma, etc. On the contrary, if one takes $\alpha(\theta) \propto \exp(-\exp|\theta|)$,
	then $\alpha \notin
	\mathcal{T}(\mathbb{R}^N)$.
\end{remark}

\begin{remark} \label{rem:compact}
	The assumption $\mathcal{F} r_{\bm{\nu}} \in
	\mathcal{E}'(\mathbb{R}^N)$ includes the $\mathcal{L}^2$ penalty and all $\mathcal{L}^{2 p}$ penalties
	(for any positive integer $p$), but not the $\mathcal{L}^1$
	penalty. Indeed,
		For $r_2(x) = x^2$ one has $\mathcal{F} r_2 = - 2 \pi
		\delta'' \in \mathcal{E}'(\mathbb{R})$. For
		$r_{2 p}(x) = x^{2 p}$, one has $\mathcal{F} r_{2 p} = (-
		1)^p 2 \pi \delta^{(2 p)} \in  \mathcal{E}'(\mathbb{R})$,
		but for
		$r_1(x) = |x|$, $(\mathcal{F} r_1)(t) = 2 t^{-2} \notin
		\mathcal{E}'(\mathbb{R})$.
	Still, for any penalty $r \in \mathcal{S}'(\mathbb{R}^N)$, it is always possible to find a sequence $(r_n)_n \in
	\mathcal{S}'(\mathbb{R}^N)$ converging to $r$ in $\mathcal{S}'(\mathbb{R}^N)$ such that $\mathcal{F} r_n
	\in \mathcal{E}'(\mathbb{R}^N)$ for all $n$	(see Appendix~\ref{sec:compact}).
\end{remark}

\paragraph{Particular case: variational posteriors parameterized by their mean ($\bm{\nu} = \emptyset$).}
Theorem~\ref{thm:main} provides a method to compute a prior from a penalty and a family of variational
posteriors $(\beta_{\mu, \bm{\nu}})_{\mu, \bm{\nu}}$. Still, Eq.~\eqref{eqn:main} returns a distribution
$A_{\bm{\nu}}$ which may or may not satisfy the condition. Here we
present a corollary which guarantees that the condition is satisfied;
this holds under stricter conditions over the variational posteriors.

\begin{corollary}
	\label{cor:appl_3}
	Assume that the family of posterior distributions $(\beta_{\mu, \bm{\nu}})_{\mu, \bm{\nu}}$ is only
	parameterized by their means, that is $\beta_{\mu, \bm{\nu}} = \beta_{\mu}$ and $r_{\bm{\nu}}(\mu) =
	r(\mu)$.
	Assume that $\beta_{0} \in \mathcal{S}(\mathbb{R}^N)$, $\mathcal{F} r \in
	\mathcal{E}'(\mathbb{R}^N)$, $\mathcal{F} \beta_{0}$ is nonzero everywhere and $\beta_{0}$ has a finite
	entropy.

	Assume that $\mathcal{F}^{-1}[ \frac{\mathcal{F} r}{\mathcal{F}\check{\beta}_{0}}]$ is a function
	satisfying
	\begin{gather}
	\exists\, (a, b, k) \in \mathbb{R}^+_* \times \mathbb{R}^+_* \times \mathbb{R} : \nonumber \\
	\forall \theta \in
	\mathbb{R}^N, \quad \mathcal{F}^{-1}\left[
	\frac{\mathcal{F} r}{\mathcal{F}\check{\beta}_{0}} \right](\theta) \geq a \|\theta\|^b + k ,
	\label{eqn:cond_univar}
	\end{gather}
	where $\|\cdot\|$ is the $\mathcal{L}^2$ norm on $\mathbb{R}^N$.

	Then there exists $\kappa > 0$ such that:
	\begin{align*}
	\alpha(\theta) = \frac{1}{\kappa} \exp\left( -\mathrm{Ent}(\beta_{0}) - \mathcal{F}^{-1}\left[
	\frac{\mathcal{F}r}{\mathcal{F}\check{\beta}_{0}} \right](\theta) \right) = \frac{e^{A(\theta)}}{\kappa}
	\end{align*}
	is a probability density satisfying $r(\mu) = \mathrm{KL}(\beta_{\mu} \| \alpha)$ up to a constant, and
	is the unique such probability density in $\mathcal{T}(\mathbb{R}^N)$.
\end{corollary}

\begin{remark}
	In many cases, condition~\eqref{eqn:cond_univar} is not a
	limitation.

	For instance, with $\beta_{\mu} \sim \mathcal{N}(\mu, \sigma^2)$,
	where $\sigma^2$ is a constant, and with the
	penalty $r$ the $\mathcal{L}^2$ penalty,
	Condition~\eqref{eqn:cond_univar} reads: $
	\exists \, (a, b, k) \in \mathbb{R}^+_* \times \mathbb{R}^+_* \times
	\mathbb{R}\,$ such that $\,\theta^2 - \sigma^2 \geq a \|\theta\|^b + k$ \;for all
	$\theta\in\mathbb{R}^N$,
	which is satisfied.
\end{remark}

\begin{remark}[Application to convergence rates]
	Several theoretical results exist in a variational inference setup, as
	Theorem~2.4~\cite{zhang2020convergence}. This theorem indicates how to compute a convergence rate of the
	variational posterior according to the number of training points. We provide in
	Appendix~\ref{sec:conv_rate} a version of this theorem (Theorem~\ref{thm:conv_rate}), adapted to our
	notation and hypotheses (Corollary~\ref{cor:conv_rate}).
\end{remark}

\begin{remark}[R\'enyi divergence]
	We focused on an identification between penalties and a $\mathrm{KL}$-divergence, since using it is
	optimal to approach the Bayesian posterior~\cite{knoblauch2019generalized}. However, our method can be
	applied if we identify the penalty to a R\'enyi divergence (see Appendix~\ref{sec:renyi}).
\end{remark}

\subsection{Example~\ref{ex:gaussian}: Gaussian Distributions with $\mathcal{L}^2$ Penalty} \label{sec:appl_1}

Let us study the variational posteriors from  Example~\ref{ex:gaussian}: each vector
of weights $\mathbf{w} \in \mathbb{R}^N$ is drawn from a Gaussian
distribution
$
\beta_{\mathbf{u}} = \mathcal{N}(\mu_1, \sigma_1^2)
\otimes \cdots \otimes \mathcal{N}(\mu_N, \sigma_N^2)$ with parameter
$\mathbf{u}=(\mu_1, \sigma_1^2, \cdots , \mu_N, \sigma_N^2)$.
We assume that each pair of variational parameters $(\mu_k, \sigma_k)$ is penalized independently by some
penalty $r_{a, b}$ depending on two real-valued functions $a$ and $b$:
\begin{align}
r_{a, b}(\mu_k, \sigma_k^2) = a(\sigma_k^2) + b(\sigma_k^2) \mu_k^2 . \label{eqn:pen_ex_1}
\end{align}
The penalty over $\mathbf{u}$ is assumed to be the sum of the penalties $r_{a, b}(\mu_k, \sigma_k)$.
Therefore, we study each pair $(\mu_k, \sigma_k)$ independently and we omit the index $k$.

\begin{corollary}\label{cor:ex_1}
	If the penalty \eqref{eqn:pen_ex_1} above corresponds to a prior $\alpha$, then
	$\alpha$ is Gaussian.

	More precisely,
	let $\alpha$ be a probability distribution
        in $\mathcal{T}(\mathbb{R})$,
        and assume that $r_{a, b}(\mu,
	\sigma^2)=\mathrm{KL}(\beta_{\mu, \sigma^2} \| \alpha)$, up to a constant.
	Then there exists $\sigma_0^2>0$ such that
	$\alpha = \mathcal{N}(0, \sigma_0^2)$. Moreover, in that case, the penalty
	is, up to a constant:
	\begin{align*}
	r_{a, b}(\mu, \sigma^2) = \frac{1}{2} \left[ \frac{\sigma^2 +
		\mu^2}{\sigma_0^2} + \ln\left( \frac{\sigma_0^2}{\sigma^2}\right) - 1 \right] .
	\end{align*}
\end{corollary}

This corollary is an application of Theorem~\ref{thm:main}. We give the proof in Appendix~\ref{sec:details_1}.

Thus,
the assumption that a penalty $r$
arises from a variational interpretation
is a strong constraint over $r$. Here the
penalty $r$ was initially parameterized by a pair of real functions $(a, b)$, and is finally parameterized by
a single number $\sigma_0^2$.

\subsection{Example~\ref{ex:dirac}: Deterministic Posteriors and the MAP}
\label{sec:appl_2}

Another basic example
is to use Dirac functions as variational posteriors
(Example~\ref{ex:dirac}): $\beta_{\mu,
	\bm{\nu}} = \delta_\mu$. Since $\delta_0 \notin \mathcal{S}(\mathbb{R})$, the
technical conditions of Theorem~\ref{thm:main} are not satisfied. However, it is possible to apply
Formula~\eqref{eqn:main} and check that the resulting prior $\alpha$ is
consistent with a chosen penalty
$r$.

Applying Formula~\eqref{eqn:main} yields
\begin{align*}
A &= - \mathrm{Ent}(\delta_0) \mathbbm{1} - \mathcal{F}^{-1}\left[
\frac{\mathcal{F} r}{\mathcal{F} \delta_0} \right]
= - \mathcal{F}^{-1}\left[ \frac{\mathcal{F} r}{\mathbbm{1}} \right] = - r .
\end{align*}

Thus, if $\exp (-r)$ integrates to $0<\kappa<\infty$, then we can define $\alpha =
\frac{1}{\kappa} \exp (-r)$. Then, we can check that
indeed $\mathrm{KL}(\delta_{\mu} \| \alpha) = r(\mu)$ up to a constant:
\begin{align*}
\mathrm{KL}(\delta_{\mu} \| \alpha) &= -\WpEnt(\delta_0) - \left\langle
\delta_{\mu} , \ln (\alpha)
\right\rangle \\
&= - \ln \alpha(\mu)-\WpEnt(\delta_0) \\
&= r(\mu) + \ln \kappa -\WpEnt(\delta_0),
\end{align*}
which confirms that the proposed prior $\alpha$ is consistent with the
penalty $r(\cdot)$.

Thus, this formula recovers via variational inference the well-known
penalty--prior equivalence in the MAP approximation,
$\alpha_{\mathrm{VI}}(\theta) \propto \exp(-
r(\theta)) \propto \alpha_{\mathrm{MAP}}(\theta)$
\citep{mackay1992practical}.

However, this is somewhat formal:
the entropy $\WpEnt(\delta_0)$ of a Dirac function is technically undefined and is an ``infinite
constant''. In practice, though, with a finite machine precision
$\epsilon$,
a Dirac mass can be defined as a uniform distribution over
an interval of size $\epsilon$, and $\WpEnt(\delta_0)$ becomes the finite
constant $\ln \epsilon$.

\section{Application to Neural Networks: Choosing the Penalty Factor} \label{sec:NN_appl}

In this section, we compare the prior $\alpha$ arising from a penalty via
Theorem~\ref{thm:main}, to reasonable weight priors for neural networks.
In particular, we study how the prior varies when scaling the penalty by
a factor $\lambda$, namely, $r_{\lambda}(\cdot) = \lambda
\tilde{r}(\cdot)$ for a reference penalty $\tilde{r}$. Requiring that
$\alpha$ is comparable to standard priors for neural network
weights provides a specific value of the regularization constant
$\lambda$.
Specifically,
we compare the prior to the standard initialization of neuron weights
\citep{glorot2010understanding}: the variance of weights sampled from the
prior $\alpha$ should be approximately equal to the inverse of the number
of incoming weights to a neuron.
This constraint can be used to determine $\lambda$. In particular, this
suggests different values of $\lambda$ for different layers, depending on
their size.

Possible advantages of being able to predict a good value for $\lambda$
include avoiding a hyperparameter search, and better adjustment of the
relative penalties of different layers or groups of neurons. For instance, one
application of penalties in neural networks
is to push neurons or convolutional filters towards zero, allowing
for network pruning and reduced computational overhead.
Penalties have been developed to remove entire
neurons or filters, often based
on the Lasso penalty: for instance, group-Lasso \citep{scardapane2017group} and sparse
group-Lasso~\citep{alvarez2016learning}. In the latter work, different
penalties are used for different layers, with values of $\lambda$
determined empirically. We will compare our predicted values of $\lambda$
to the values used in these works.

\paragraph{Additive penalties and independent priors.}
Below, we focus on penalties that are expressed as sums over neurons or
groups of neurons, such as $\mathcal{L}^2$ or group-Lasso
penalties.
In the Bayesian setup, this corresponds to
the additivity property of Kullback--Leibler divergence over products of
distributions,
$
\mathrm{KL}(\beta_1 \otimes \beta_2 \| \alpha_1 \otimes \alpha_2) = \mathrm{KL}(\beta_1 \| \alpha_1) +
\mathrm{KL}(\beta_2 \| \alpha_2)$. Thus, sums of penalties over different
neurons or groups of neurons correspond to priors $\alpha$ and variational
posteriors $\beta$ that decompose as independent distributions over these
neurons or groups of neurons.

\subsection{A Reasonable Condition over the Prior $\alpha$}

Let us consider the variational inference framework applied to one weight
$w_{lij}$, the $j$-th weight of the $i$-th neuron in the $l$-th layer of
a neural network.
As a default
distribution, one could expect $\alpha$ to be usable to initialize the weight $w_{lij}$.

Therefore, we require $\alpha$ to satisfy the condition given by \cite{glorot2010understanding} over the initialization
procedure, which we call \condPrior{}:
\begin{align*}
\alpha \text{ fulfills~} \condPrior{} \;\;\;\; \Leftrightarrow \;\;\;\;
\left\lbrace
\begin{array}{ll}
\mathbb{E}_{w_{lij} \sim \alpha} [w_{lij}] = 0 \\
\mathbb{E}_{w_{lij} \sim \alpha} [w_{lij}^2] = 1/{P_l}
\end{array}
\right.  ,
\end{align*}
where $P_l$ is the number of incoming weights in one neuron of the layer $l$. More generally, this condition can be
written for the whole set of incoming weights to each neuron:
denoting by $\mathbf{w}_{l i}$ the vector of all incoming weights of neuron
$i$ in layer $l$, one can define
Condition~\condPriorP{}:
\begin{align*}
\alpha \text{ fulfills~} \condPriorP{} \;\;\;\; \Leftrightarrow \;\;\;\;
\left\lbrace
\begin{array}{ll}
\mathbb{E}_{\mathbf{w}_{l i} \sim \alpha} [\mathbf{w}_{l i}] = 0 \\
\mathbb{E}_{\mathbf{w}_{l i} \sim \alpha} [\|\mathbf{w}_{l i}\|^2] = 1
\end{array}
\right. ,
\end{align*}
which slightly extends \condPrior{}.

Thus, if the prior $\alpha$ depends on some variable $b$, then Condition~\condPrior{} is reflected on $b$.
For instance, in Example~\ref{ex:gaussian} (Section~\ref{sec:appl_1}), \condPrior{} is satisfied if
and only if
$\alpha_{\sigma_0^2}$ is $\mathcal{N}\left(0, 1/P_l\right)$.
In the end, our suggested recipe for finding reasonable values for parameters $b$ of a penalty $r_b$ is the following:
\begin{enumerate}
	\item Follow the formulas in Thm.~\ref{thm:main} to compute a
	prior $\alpha_b$ such that $r_b(\mu, \bm{\nu}) =
	        \mathrm{KL}(\beta_{\mu, \bm{\nu}} \| \alpha_b) + K$.
	\item write Condition~\condPrior{} or~\condPriorP{} (depending on
	the case) for $\alpha_b$: this provides a constraint on $b$.

\end{enumerate}

\subsection{Examples: $\mathcal{L}^2$, $\mathcal{L}^1$, and Group-Lasso
Penalties}
\label{sec:penkl_ex_heur}

We now review some standard penalties, and apply this criterion to compute
the penalty factor $\lambda$ for each penalty.

Here we work with Dirac posterior distributions
$\beta_{\mu} = \delta_{\mu}$ (Example~\ref{ex:dirac} and
Section~\ref{sec:appl_2}).
In that case,
since each weight $w_{lij}$ is deterministically set to $\mu_{lij}$, we use
$w_{lij}$ and $\mu_{lij}$ indifferently. Thus, the penalty with penalty
factor $\lambda$ is
\begin{align*}
r_{\lambda}(w) = r_{\lambda}(\mu) &= \lambda \tilde{r}(\mu) = \mathrm{KL}(\delta_{\mu} \|
\alpha_{\lambda}) + K
\end{align*}
for some reference penalty $\tilde{r}$. For each $\lambda$, the
corresponding prior is
$\alpha_{\lambda}(\theta) \propto e^{- \lambda \tilde{r}(\theta)}$.

We will apply Condition~\condPrior{} to find a value for $\lambda$, for
various penalties. In
Table~\ref{tab:penkl} we compare these values to usual ones.

We recall
that $P_l$ in the number of incoming weights of a neuron in layer $l$.

\begin{remark}
\label{rem:scaling}
We
recall that $r(\mathbf{w})$ is a full-dataset penalty (see
Eq.~\ref{eq:introVI}), so that the actual per-minibatch penalty for
stochastic gradient is
$\frac{1}{n}r(\mathbf{w})$ for individual samples or
$\frac{B}{n}r(\mathbf{w})$ for minibatches
of size~$B$.
\end{remark}

The results below apply equally to fully connected and to
convolutional networks; in the latter case, ``neuron'' should read as
``individual filter''.

\paragraph{$\mathcal{L}^2$ penalty.} We have $r_{\lambda}(w) = \lambda w^2$, thus
$\alpha_{\lambda}(\theta) \sim \mathcal{N}(0, 1/(2 \lambda))$.
Then Condition~\condPrior{} is equivalent to $\lambda =
\frac{P_l}{2}$. Thus, from a Bayesian viewpoint, when using a
$\mathcal{L}^2$-penalty,
each weight $w$ of a neuron with $P_l$ incoming weights should be
penalized by
\begin{align}
r_{\lambda}(w) = \frac{P_l}{2} w^2 .
\end{align}

\paragraph{$\mathcal{L}^1$ penalty.} We have $r_{\lambda}(w) = \lambda |w|$, thus
$\alpha_{\lambda}(\theta) = \frac{\lambda}{2}e^{-\lambda |\theta|}$.
Therefore, Condition~\condPrior{} is equivalent to $\lambda = \sqrt{2 P_l}$.
As a consequence, when penalizing weight $w$ in the $l$-th layer of a
neural network with a $\mathcal{L}^1$-penalty, this weight $w$ should be penalized with the term:
\begin{align}
r_{\lambda}(w) = \sqrt{2 P_l} |w| .
\end{align}

\paragraph{Standard Group-Lasso penalty.} The group-Lasso jointly
penalizes all the incoming weights of
each neuron $\mathbf{w}_{l, i} \in \mathbb{R}^{P_l}$. Thus, we consider a prior and a posterior that are
probability distributions on $\mathbb{R}^{P_l}$, and use
Condition~\condPriorP{}. Denoting by $\mathbf{w} \in
\mathbb{R}^{P_l}$ the incoming weights of a neuron, we have
\begin{align}
r_{\lambda}(w) &= \lambda \|\mathbf{w}\|_2 . \label{eqn:loss_group_lasso}
\end{align}
Then $\alpha_{\lambda}(\theta) = \frac{\lambda^{P_l}}{S_{P_l - 1} \Gamma(P_l)} e^{- \lambda
	\|\theta\|_2}$,
where $\Gamma$ is Euler's Gamma function and $S_{n - 1}$ is the surface of the $(n - 1)$-sphere. After
computation of the variance, Condition~\condPriorP{} is equivalent to $\lambda = \sqrt{P_l (P_l + 1)}$.
As a consequence, when penalizing neuron
$\mathbf{w}$ in the $l$-th layer of a neural network with a group-Lasso penalty, this neuron $\mathbf{w}$
should be penalized with the term:
\begin{align}
r_{\lambda}(\mathbf{w}) = \sqrt{P_l (P_l + 1)} \|\mathbf{w}\|_2 .
\end{align}

This choice differs from
\citep{alvarez2016learning
, who use $\lambda \propto
\sqrt{P_l}$, after an intuition proposed in \cite{yuan2006model}. Their whole-network penalty is
\begin{align}
\sum_{l = 1}^{L} \tilde{\lambda}_l \Bigg( (1 - \gamma) \sqrt{P_l} \sum_{i
	= 1}^{n_l} \|\mathbf{w}_{l, i}\|_2
 + \gamma \sum_{i = 1}^{n_l} \|\mathbf{w}_{l, i}\|_1 \Bigg) ,
\label{eqn:loss_sgl}
\end{align}
where $\gamma \in [0, 1]$ is a fixed constant, $L$ is the number of layers, $n_l$ is the number of neurons in
the $l$-th layer, $P_l$ is the number of parameters in each neuron in the
$l$-th layer, and $\mathbf{w}_{l, i}$ is
the set of weights of the $i$-th neuron of the $l$-th layer.
The per-layer regularization constants $\tilde{\lambda}_l$ are chosen
empirically in
\citep{alvarez2016learning}.
On the other hand, our
Bayesian reasoning yields a scaling $\sqrt{P_l(P_l+1)}$ instead of
$\sqrt{P_l}$
for the penalties $\|\mathbf{w}_{l,i}\|_2$ in \eqref{eqn:loss_sgl}.

\paragraph{Reversed Group-Lasso penalty.} We recall that the standard group-Lasso penalty groups the weights
of a layer by neurons, that is output features.
It is also possible to group them by input features, namely, to group
together the
outgoing weights of each neuron. For a
fully-connected neural network, the loss penalized by a ``reversed'' group-Lasso can be written:
\begin{align}
L(\mathbf{w}) &= \ell(\mathbf{w}) + \sum_{l = 1}^{L} \lambda_l \sum_{j = 1}^{P_l}
\|\mathbf{w}_{l, \cdot, j}\|_2 , \label{eqn:rev_group_lasso}
\end{align}
where $\mathbf{w}_{l, \cdot, j} \in \mathbb{R}^{n_l}$ is the vector of weights of the $l$-th layer linked to
the $j$-th input feature and $n_l$ is the number of neurons in the $l$-th
layer.
By computations similar to standard group-Lasso,
\condPriorP{} is equivalent to $\lambda = \sqrt{P_l (n_l +
1)}$ where $n_l$ is the number of neurons in layer $l$, namely,
\begin{align*}
r_{\lambda}(\mathbf{w}_{l, \cdot, j}) = \sqrt{P_l (n_l + 1)} \|\mathbf{w}_{l, \cdot, j}\|_2 .
\end{align*}

\begin{table}[t]
	\caption{
	How the regularization constant $\lambda$ depends on the number
	of neurons $n_l$ and the number of parameters per neuron $P_l$ in
	layer $l$, both for our heuristics
	($\lambda_{\mathrm{Bayesian}}$) and for standard settings
	($\lambda_{\mathrm{usual}}$).
 		For details on group-Lasso and reversed group-Lasso
		penalties, see Equations~\eqref{eqn:loss_group_lasso} and~\eqref{eqn:rev_group_lasso}.}
	\begin{center}
		\begin{small}

		\begin{sc}
			\begin{tabular}[h]{P{1cm}P{0.5cm}P{0.5cm}P{1.965cm}P{2.158cm}}
				\toprule
				penalty                                & $\mathcal{L}^2$ & $\mathcal{L}^1$ &
				group-Lasso                 & rev.\ gr.-Lasso        \\ \midrule
				$\lambda_{\mathrm{Bayesian}}$ & $P_l / 2$       & $\sqrt{2 P_l}$  & $\sqrt{P_l
					(P_l + 1)}$ & $\sqrt{P_l (n_l + 1)}$ \\
				$\lambda_{\mathrm{usual}}$     & $1$             & $1$             &
				$\sqrt{P_l}$                & $\sqrt{n_l}$           \\ \bottomrule
			\end{tabular}
		\end{sc}
		\end{small}
	\end{center}
	\label{tab:penkl}
	\vskip -0.1in
\end{table}

\section{Experiments} \label{sec:penkl_expes}

Penalty terms are often used in the literature \citep{scardapane2017group,alvarez2016learning,han2015learning}
in order to prune neural networks, that is, to make them
sparse. Indeed, penalties such as $\mathcal{L}^1$ or group-Lasso tend to push weights or entire groups of
weights towards $0$ \citep{Tibshirani}. The efficiency of such methods can be measured in terms of
final accuracy and number of remaining parameters.

We have chosen to compare within this context our heuristic to usual setups for the penalty
factor, for different penalties. Our main criterion remains the final accuracy of the pruned network.
According to the results given in Table~\ref{tab:penkl}, the global penalty should be
decomposed into a sum of penalties over each layer:
\begin{align*}
r(\mathbf{w}) = \lambda \sum_{l = 1}^{L} \lambda_l r(\mathbf{w}_l) ,
\end{align*}
where $\lambda$ is a global penalty factor, $L$ is the number of layers, $r(\mathbf{w}_l)$ is the penalty term
due to the $l$-th layer and $\lambda_l$ is its corresponding penalty factor. According to our heuristic, each
$\lambda_l$ should be set to $\lambda_{\mathrm{Bayesian}}$ given in Table~\ref{tab:penkl} and $\lambda =
\lambda_{\mathrm{Th}} = 1/n$, where $n$ is the size of the training dataset (by
Remark~\ref{rem:scaling}). In the usual setup, each $\lambda_l$ is set to
$\lambda_{\mathrm{usual}}$ given in Table~\ref{tab:penkl}, while several values for $\lambda$ are tested.

We test the quality of our heuristic in two steps. First, we only test the computed partial penalty factors
$\lambda_l$: they are fixed to $\lambda_{\mathrm{Bayesian}}$, while several values for $\lambda$ are tested.
Second, we test the full heuristic: each $\lambda_l$ is fixed to its corresponding
$\lambda_{\mathrm{Bayesian}}$ and $\lambda = \lambda_{\mathrm{Th}} = 1/n$.

\paragraph{Experimental setup.} We consider two neural networks: a version of VGG19~\citep{simonyan2014very}
with one fully connected layer of size $512$, and CVNN, which is a simple network with two convolutional
layers (of respective sizes $100$ and $200$ with $5 \times 5$ patches), each followed by a ReLU and a $2
\times 2$ max-pool layer, and two fully-connected layers (of sizes $1000$ and $200$).

The training and pruning procedure is detailed in
Appendix~\ref{sec:exp_setup} and the full experimental procedure is detailed in
Appendix~\ref{sec:exp_procedure}. The
CIFAR-10 dataset is decomposed
into three parts: a training set ($42000$ images), a validation set ($8000$
images), and a test set ($10000$ images). All reported accuracies are computed over the test set, which is
never used during each run.

\paragraph{Results.} We give the results in Figure~\ref{fig:penkl}.
Each point gives the final accuracy and
number of parameters of a neural network for a given setup, averaged over 3 runs. The red line and the blue
line correspond respectively to the usual setup and our setup for
per-layer penalty
$\lambda_l$, for various
global factors $\lambda$. To check the quality of the heuristics over
$\lambda_l$, we should compare the maxima of the two lines in each
graph. The theoretical value $\lambda = \lambda_{\mathrm{Th}}$ is marked by the grey vertical line,
so the blue point on the grey vertical line illustrates the
performance of the joint theoretical values on both $\lambda_l$ and $\lambda$.

We report in Table~\ref{tab:penkl_results} the values of relevant
quantities: $\mathrm{acc}_{\mathrm{usual}}^{*}$, the best (over
$\lambda$) accuracy of the usual per-layer scaling $\lambda_l$;
$\mathrm{acc}_{\mathrm{Bayesian}}^{*}$, the best (over
$\lambda$) accuracy of the Bayesian per-layer scaling $\lambda_l$ (we denote by $\lambda^*$ the optimal
$\lambda$); and
$\mathrm{acc}_{\mathrm{Bayesian}}$, the accuracy of the Bayesian
per-layer $\lambda_l$ together with the Bayesian predicted $\lambda = \lambda_{\mathrm{Th}}$.
This would allow us to
conclude:
\begin{enumerate}
	\item \label{it:penkl_1}
	if $\mathrm{acc}_{\mathrm{Bayesian}}^{*} > \mathrm{acc}_{\mathrm{usual}}^{*}$, then we can conclude
	that our theoretical estimation for $\lambda_l$ is better than usual
	ones;
	\item \label{it:penkl_2}
	moreover, we check whether $\mathrm{acc}_{\mathrm{Bayesian}} > \mathrm{acc}_{\mathrm{usual}}^{*}$. If it
	is, then we can conclude that our theoretical estimation for $\lambda_l$ \emph{and} the
	choice $\lambda = \lambda_{\mathrm{Th}}$ are better than the usual ones;
	\item \label{it:penkl_3}
	finally, the closer the ratio $\lambda_{\mathrm{Th}} / \lambda^{*}$ is to $1$, the closer we are to the
	Bayesian
	theoretical framework.
\end{enumerate}

Regarding Point~\ref{it:penkl_1}, our theoretical estimation for $\lambda_l$ leads to accuracies which remain
close to accuracies obtained in the usual setup.
Our setup leads to slightly better accuracies when training CVNN, and slightly worse for VGG19.

Regarding Points~\ref{it:penkl_2} and~\ref{it:penkl_3},
the usual setup with optimized $\lambda$ and our setup with Bayesian
estimation for $\lambda = \lambda_{\mathrm{Th}}$ perform similarly, though the second one usually performs
slightly worse, due to a mismatch between $\lambda_{\mathrm{Th}}$ and $\lambda^*$. Indeed,
$\lambda_{\mathrm{Th}}$ is always roughly $10$ times greater than~$\lambda^*$.\\

\newcommand{.12\linewidth}{.12\linewidth}
\begin{figure*}[h]

	\begin{tabular}{r@{\hskip 12mm}c}

		&  $\mathcal{L}^2$ penalty \hspace{0.6cm} \qquad \hspace{0.5cm} $\mathcal{L}^1$ penalty
		\hspace{1.4cm} Gr.-Lasso penalty \hspace{0.5cm} Rev.\
		gr.-Lasso penalty
		\hspace{4.5cm}\mbox{}\\

		CVNN & \hspace{-1.5cm}
		\begin{minipage}{\linewidth}
			\includegraphics[height=.12\linewidth]{Out_2019_11_29_KLpen_L2_CVNN_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_12_13_KLpen_L1_CVNN_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_12_05_KLpen_GLasso_CVNN_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_11_29_KLpen_GLassoOut_CVNN_final_new}
		\end{minipage} \\

		VGG19 & \hspace{-1.5cm}
		\begin{minipage}{\linewidth}
			\includegraphics[height=.12\linewidth]{Out_2019_11_29_KLpen_L2_VGG_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_12_13_KLpen_L1_VGG_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_12_05_KLpen_GLasso_VGG_final_new}
			\qquad
			\includegraphics[height=.12\linewidth]{Out_2019_11_29_KLpen_GLassoOut_VGG_final_new}
		\end{minipage} \\

	\end{tabular}

	\caption{\label{fig:penkl}Final performance and number of parameters for various penalties in function of
		the penalty
		factor $\lambda$. Red line: standard setup for the penalty; blue line: setup provided by the
		heuristic
		\condPrior{} or \condPriorP{}. Each bar corresponds to a final neural network: its abscissa is the
		penalty factor $\lambda$ used for training it, its ordinate is its final accuracy, and its width
		corresponds to its final number of parameters. The vertical grey line is the value of the
		heuristic
		for $\lambda$.}

\end{figure*}

\begin{table*}[t!]
	\caption{Comparison of the results. We show on the first row the best accuracy
		$\mathrm{acc}_{\mathrm{usual}}^{*}$ obtained with the usual setup. Then we show the
		best accuracy $\mathrm{acc}_{\mathrm{Bayesian}}^{*}$ obtained with our setup, the
		accuracy $\mathrm{acc}_{\mathrm{Bayesian}}$ obtained with our setup with the Bayesian theoretical
		value
		for $\lambda$, that is $\lambda_{\mathrm{Th}} = n^{-1}$, and the ratio between $\lambda_{\mathrm{Th}}$
		and the factor $\lambda =
		\lambda^{*}$, that is the optimal value for $\lambda$. Among the results obtained with our methods,
		those highlighted in blue are better, and those highlighted in red are worse than in the usual setup.}

	\begin{center}
		\begin{tiny}
		\begin{sc}

		\begin{tabular}[h]{ccccccccc}
			\toprule
			{}                                                                                &
			\multicolumn{2}{c}{$\mathcal{L}^2$ pen.}    &   \multicolumn{2}{c}{$\mathcal{L}^1$ pen.}
			&        \multicolumn{2}{c}{group-Lasso pen.}         &   \multicolumn{2}{c}{rev.\ gr.-Lasso
			pen.}   \\
			\cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9}
			{} & CVNN                        & VGG             & CVNN                  & VGG
			& CVNN                        & VGG                   & CVNN                 &
			VGG                   \\ \midrule
			$\mathrm{acc}_{\mathrm{usual}}^{*}$ ($\%$)                                      & $88.00 \pm
			.4$          & $93.35 \pm .15$ & $88.36 \pm .3$        & $93.17 \pm .3$        & $88.43 \pm
			.14$         & $92.78
			\pm .19$   & $88.04 \pm .4$       & $93.37 \pm .09$       \\ \midrule
			$\mathrm{acc}_{\mathrm{Bayesian}}^{*}$                                          & $\blue{88.69
				\pm .12}$ & $93.48 \pm .09$ & $88.41 \pm .3$        & $92.89 \pm .2$        & $\blue{88.67 \pm
				.09}$ & $\red{92.35 \pm .18}$ & $88.32 \pm .16$      & $\red{93.03 \pm .15}$ \vspace{0.4mm}\\
			$\mathrm{acc}_{\mathrm{Bayesian}}$                                                & $88.25 \pm
			.3$          & $93.28 \pm .17$ & $\red{87.48 \pm .08}$ & $\red{92.74 \pm .19}$ & $\red{87.45 \pm
				.17}$  & $\red{92.24 \pm .14}$ & $\red{85.49 \pm .3}$ & $\red{92.85 \pm .06}$ \vspace{0.4mm}\\
			$\lambda_{\mathrm{Th}} / \lambda^{*}$                                                          &
			$10^{0.5}$                  & $10^1$          & $10^1$                & $10^1$                &
			$10^2$                      & $10^2$                & $10^{1.5}$           & $10^1$
			\\ \bottomrule
		\end{tabular}
		\end{sc}
		\end{tiny}
	\end{center}

	\vskip -0.1in
	\label{tab:penkl_results}
\end{table*}

\FloatBarrier

\paragraph{Discussion.} This systematic overestimation of the penalty factor indicates that some phenomenon is
not yet understood. We have two mutually compatible explanations.

First, the choice $\lambda = 1/n$ could be
overestimated, as a strict Bayesian setup might be overcautious from the very
beginning.
Indeed, in other Bayesian approaches to neural networks, such observations have been made.
For instance, when using stochastic Langevin dynamics to
approximate the posterior, performance is better if the weight of the
posterior is arbitrarily decreased by an additional factor $n$ (see footnote 5 in
\cite{MarceauOllivier17}, and \cite{Li:2016}).

Second, our choice for $\lambda_l$, based on Glorot's initialization, could also be overestimated. Indeed,
$\lambda_l$ only depends on $P_l$ (the
number of parameters in each neuron in layer $l$),
and not on the location of layer $l$ in the network for instance, while the following observations
suggest that
the heuristic used to set $\lambda_l$ should take into account the architecture.
We noticed that in most of the pruning experiments we made, the reached level of sparsity is much higher in the second half of the network (layers closer to the output).
This is corroborated by \cite{zhang2019all} which shows that the layers may behave differently
from a training point of view, depending on the architecture of the network and their position in it.
Notably, they have proven that, in a trained VGG, some of the last convolutional layers can
be reset to their initial value without changing much the final accuracy,
while this cannot be done for the first layers.

\section{Conclusion}

We have provided a theorem that bridges the gap between empirical penalties and Bayesian priors when learning
the distribution of the parameters of a model.
This way, various regularization techniques can be studied in the same Bayesian framework, and be
seen as probability distributions.
This unification of points of view is valuable on the theoretical side, since it can be used to apply
well-know theorems in variational inference, such as results about convergence rates. Moreover,
it allowed us to take into account well known
heuristics (as Glorot's initialization) in order to find reasonable values for hyperparameters of the penalty.

We have checked experimentally that our theoretical framework leads to reasonable results. However, we noticed
a constant mismatch (about a factor $10$) between our predicted penalty factor $\lambda_{\mathrm{Th}}$ and the
best one $\lambda^*$.
This fact raises interesting questions about a possible overcautiousness of the Bayesian framework and the
possible impact of the architecture when penalizing layers, which we plan to investigate further.

\paragraph{Acknowledgements.} The project leading to this work has received funding from the European Research Council
(ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 834175).

Machine Learning (cs.LG); Statistics Theory (math.ST); Machine Learning (stat.ML)