Refined Shapley-Folkman Lemma and Its Application in Duality Gap Estimation

Abstract: Based on concepts like kth convex hull and finer characterization of nonconvexity of a function, we propose a refinement of the Shapley-Folkman lemma and derive a new estimate for the duality gap of nonconvex optimization problems with separable objective functions. We apply our result to a network flow problem and the dynamic spectrum management problem in communication systems as examples to demonstrate that the new bound can be qualitatively tighter than the existing ones. The idea is also applicable to cases with general nonconvex constraints.

\section{Introduction}

The Shapley-Folkman lemma (Theorem~\ref{thm:shapleyfolkman}) was stated and used to establish the existence of approximate equilibria in economy with nonconvex preferences \cite{Starr1969}. It roughly says that the sum of a large number of sets is close to a convex set and thus can be used to generalize results on convex objects to nonconvex ones.

\begin{theorem}\label{thm:shapleyfolkman}
Let $S_1,S_2,\dots,S_n$ be any subsets of $\mathbb{R}^m$. For each $z \in \conv\sum_{i=1}^nS_i=\sum_{i=1}^n\conv S_i$, there exist points $z^i \in \conv S_i$ such that $z=\sum_{i=1}^nz^i$ and $z^i \in S_i$ except for at most $m$ values of $i$.
\end{theorem}

\begin{remark}
In this paper, we use superscript to index vectors and use subscript to refer to a particular component of a vector. For instance, both $x^i$ and $x^{ij}$ are vectors, but $x^i_s$ is the $s$th component of the vector $x^i$. For two vectors $x$ and $y$, $x \leq y$ means $x_s \leq y_s$ holds for all components.
\end{remark}

The Shapley-Folkman lemma has found applications in many fields including economics and optimization theory. In particular, it has been used to study optimization problems with separable objectives and linear constraints such as
\begin{equation}\label{eq:primal}
\begin{split}
\min \quad & \sum_{i=1}^nf_i(x^i) \\
\st \quad & \sum_{i=1}^nA_ix^i \leq b.
\end{split}
\end{equation}
Here $x^i \in \mathbb{R}^{n_i}$ are the decision variables. The function $f_i:\mathbb{R}^{n_i} \to \bar\mathbb{R}$ is proper\footnote{This means that the function never takes $-\infty$ and its domain is nonempty.} and lower semi-continuous, and its domain is bounded. $A_i$ is a matrix of size $m \times n_i$, so there are $m$ linear constraints in total. The dual problem of \eqref{eq:primal} is
\begin{equation}\label{eq:dual}
\begin{split}
\max \quad & -\sum_{i=1}^nf^*_i(-A^T_iy)-b^Ty \\
\st \quad & y \geq 0,
\end{split}
\end{equation}
where $f^*_i$ is the conjugate function of $f_i$. Denote the optimal value of the primal problem \eqref{eq:primal} and dual problem \eqref{eq:dual} as $p$ and $d$, respectively. In general, there will be a positive duality gap $p-d>0$ if some function $f_i$ is not convex.

The authors of \cite{AE1976} presented the following upper bound for the duality gap based on the Shapley-Folkman lemma:
\begin{equation}\label{eq:originalbound}
p-d \leq \min\{m+1,n\}\max_{i=1,\dots,n}\rho(f_i).
\end{equation}
Here $\rho(f)$ is the nonconvexity of a proper function $f$ defined by
\begin{equation}\label{eq:nonconvexity}
\rho(f)=\sup\left\{f\left(\sum_j\alpha_jx^j\right)-\sum_j\alpha_jf(x^j)\right\}
\end{equation}
over all finite convex combinations of points $x^j \in \dom f$, i.e., $f(x^j)<+\infty$, $\alpha_j \geq 0$ with $\sum_j\alpha_j=1$.

In \cite{UB2016}, an improved bound for the duality gap\footnote{In fact, the bound \eqref{eq:udellbound} derived in \cite{UB2016} is for the difference $p-\hat p$, in which $\hat p$ is the optimal value of the convexified problem where the function $f_i$ is substituted by $f^{**}_i$. However, $\hat p=d$ because the refined Slater's condition holds for the convexified problem.} was given by
\begin{equation}\label{eq:udellbound}
p-d \leq \sum_{i=1}^{\min\{m,n\}}\rho(f_i),
\end{equation}
where we assume that $\rho(f_1) \geq \dots \geq \rho(f_n)$. Although the new bound \eqref{eq:udellbound} is only a slight improvement over the original bound \eqref{eq:originalbound} by a factor of $m/(m+1)$, it nevertheless demonstrates that \eqref{eq:originalbound} can never be tight except for some trivial situations. Thus, using Shapley-Folkman lemma as in \cite{AE1976} does not result in a tight bound for the duality gap, which suggests that the original Shapley-Folkman lemma itself can be improved in certain circumstances.

In this paper, we propose a refinement for the original Shapley-Folkman lemma and derive a new bound for the duality gap based on this new result. The refined Shapley-Folkman lemma is stated and proved in Section~\ref{sec:refined}. Unlike \eqref{eq:originalbound} and \eqref{eq:udellbound}, our new bound for the duality gap depends on some finer characterization of the nonconvexity of a function, which is introduced in Section~\ref{sec:nonconvexity}. The new bound itself is given in Section~\ref{sec:dualitygap}, which can easily recover existing ones like \eqref{eq:udellbound}. Next, we will apply it to two examples, a network flow problem and the dynamic spectrum management problem, in Section~\ref{sec:application} to demonstrate that the new bound can be qualitatively tighter than the bound \eqref{eq:udellbound}. Although we mainly focus on the case of linear constraints, Section~\ref{sec:nonlinear} shows how the major idea in this paper can be applied to the cases with general convex or even nonconvex constraints.

\section{Refined Shapley-Folkman Lemma}\label{sec:refined}

To write down our refined version of the Shapley-Folkman lemma, we need to first introduce the concept of $k$th convex hull.

\begin{definition}
The \emph{$k$th convex hull} of a set $S$, denoted by $\conv_k S$, is the set of convex combinations of $k$ points in $S$, i.e.,
\[
\conv_k S=\left\{\sum_{j=1}^k\alpha_jv^j \middle| v^j \in S, \alpha_j \geq 0, \forall j=1,\dots,k, \sum_{j=1}^k\alpha_j=1\right\}.
\]
\end{definition}

\begin{figure}
\centering
\tikzset{every node/.style={circle,fill=black,inner sep=0.03cm}}
\subfloat[$\conv_1 S$]{
\begin{tikzpicture}
\draw (0,0) node["$A$" left]{} (1,0) node["$B$" right]{} (0,1) node["$C$" above]{};
\end{tikzpicture}
}\qquad
\subfloat[$\conv_2 S$]{
\begin{tikzpicture}
\draw (0,0) node["$A$" left]{} -- (1,0) node["$B$" right]{} -- (0,1) node["$C$" above]{} -- cycle;
\end{tikzpicture}\label{fig:conv2s}
}\qquad
\subfloat[$\conv_3 S$]{
\begin{tikzpicture}
\draw[fill=gray] (0,0) node["$A$" left]{} -- (1,0) node["$B$" right]{} -- (0,1) node["$C$" above]{} -- cycle;
\end{tikzpicture}\label{fig:conv3s}
}
\caption{The $k$th convex hull of a three-point set $S=\{A,B,C\}$.}\label{fig:convks}
\end{figure}

Figure~\ref{fig:convks} gives a simple example to illustrate the definition of $k$th convex hull. In Figure~\ref{fig:convks}, the set $S=\{A,B,C\}$, $\conv_1 S=S$, $\conv_2 S$ are the segments $AB$, $BC$ and $CA$, while $\conv_3 S$ is the full triangle which is also the convex hull of set $S$. In general, Carath\'eodory's theorem implies that $\conv_{m+1}S=\conv S$ for any set $S \subseteq \mathbb{R}^m$. However, for a particular set, the minimum $k$ such that $\conv_k S=\conv S$ can be smaller than $m+1$, and this number intuitively reflects how the set is closer to being convex. For instance, if we start from $T=\conv_2 S$, the set in Figure~\ref{fig:conv2s}, then $\conv_k T=\conv T$ for $k=2$.

Next, we recall the concept of $k$-extreme points of a convex set, which is a generalization of extreme points.

\begin{definition}
A point $z$ in a convex set $S$ is called a \emph{$k$-extreme} point of $S$ if we cannot find $(k+1)$ independent vectors $d^1,d^2,\dots,d^{k+1}$ such that $z \pm d^i \in S$.
\end{definition}

According to our definition, if a point is $k$-extreme, then it is also $k'$-extreme for $k' \geq k$. For a convex set in $\mathbb{R}^m$, a point is an extreme point if and only if it is 0-extreme, a point is on the boundary if and only if it is $(m-1)$-extreme, and every point is $m$-extreme. For example, in Figure~\ref{fig:conv3s}, the vertices $A,B,C$ are 0-extreme, the points on segments $AB$, $BC$ and $CA$ are 1-extreme, and all the points are 2-extreme.

Now we can state our refined Shapley-Folkman lemma:

\begin{theorem}\label{thm:refined}
Let $S_1,S_2,\dots,S_n$ be any subsets of $\mathbb{R}^m$. Assume $z$ is a $k$-extreme point of $\conv\sum_{i=1}^nS_i$, then there exist integers $1 \leq k_i \leq k+1$ with $\sum_{i=1}^nk_i \leq k+n$ and points $z^i \in \conv_{k_i}S_i$ such that $z=\sum_{i=1}^nz^i$.
\end{theorem}

The original Shapley-Folkman lemma (Theorem~\ref{thm:shapleyfolkman}) now becomes a direct corollary of Theorem~\ref{thm:refined}. Since any point $z \in \conv\sum_{i=1}^nS_i$ is an $m$-extreme point. Applying Theorem~\ref{thm:refined} on this point gives a decomposition $z=\sum_{i=1}^nz^i$ with $z^i \in \conv_{k_i}S_i \subseteq \conv S_i$ and $\sum_{i=1}^nk_i \leq m+n$. Then the conclusion in Theorem~\ref{thm:shapleyfolkman} follows because $z^i \in S_i$ if $k_i=1$, while the number of indices $i$ with $k_i \geq 2$ is bounded by $m$.

\begin{remark}
Our Theorem~\ref{thm:refined} is similar to the refined version of Shapley-Folkman lemma proposed in \cite{LS2013}. However, the result in \cite{LS2013} does not take extremeness of the point into account. By Theorem~\ref{thm:refined}, stronger result can be obtained from the knowledge of extremeness.
\end{remark}

To prove Theorem~\ref{thm:refined}, we need the following property of $k$-extreme points in a polyhedron:

\begin{lemma}\label{lem:kexposed}
Let $P \subseteq \mathbb{R}^m$ be a polyhedron and $z$ be a $k$-extreme point of $P$, then there exists a vector $a \in \mathbb{R}^m$ such that the set $\{y \in P | a^Ty \leq a^Tz\}$ is in a $k$-dimensional affine subspace.
\end{lemma}
\begin{myproof}
Assume that the polyhedron $P$ is represented by $Ax \geq b$. Let $A_=$ be the submatrix of $A$ containing the rows of active constraints for the point $z$, and $b_=$ be the vector containing the corresponding constants in $b$. The dimension of the kernel of $A_=$ is at most $k$. Otherwise, we can find independent and sufficiently small vectors $d^1,\dots,d^{k+1}$ such that $A_=d^i=0$ and $A(z \pm d^i) \geq b$ for $i=1,\dots,k+1$. This implies $z \pm d^i \in P$, which contradicts with the $k$-extremeness of point $z$.

Let $a$ be the vector such that $a^T$ is the sum of all rows in $A_=$. Consider a point $y$ satisfying $Ay \geq b$ and $a^Ty \leq a^Tz$. Since adding all inequalities together in $A_=y \geq b_==A_=z$ gives $a^Ty \geq a^Tz$, we must have $A_=y=b_=$. Therefore, $y$ is in the affine subspace defined by $A_=x=b_=$ whose dimension is at most $k$.
\end{myproof}

\begin{remark}
In the literature, the point satisfying the conclusion of Lemma~\ref{lem:kexposed} is called a \emph{$k$-exposed point}. For a general convex set $S$, a $k$-extreme point may fail to be a $k$-exposed point, although it must be in the closure of the set of $k$-exposed points if $S$ is compact \cite{Asplund1963}. For the special case of polyhedra, these two concepts are equivalent, and Lemma~\ref{lem:kexposed} is a generalization of the well-known result that an extreme point of a polyhedron is the unique minimizer of some linear function.
\end{remark}

\begin{myproof}[Proof of Theorem~\ref{thm:refined}]
Since $z$ is in the convex hull of $\sum_{i=1}^nS_i$, there exists some integer $l$ such that $z$ can be written as
\begin{equation}\label{eq:zinconv}
z=\sum_{j=1}^l\alpha_j\sum_{i=1}^nv^{ij},
\end{equation}
in which $v^{ij} \in S_i$, $\alpha_j \geq 0$, $j=1,\dots,l$ and $\sum_{j=1}^l\alpha_j=1$.

Define $S'_i=\{v^{i1},\dots,v^{il}\} \subseteq S_i$, then \eqref{eq:zinconv} actually tells us that $z \in \conv\sum_{i=1}^nS'_i$, so $z$ must be $k$-extreme in this polytope that lies in $\conv\sum_{i=1}^nS_i$. By Lemma~\ref{lem:kexposed}, there exists a vector $a \in \mathbb{R}^m$ such that the set
\[
\left\{y \in \conv\sum_{i=1}^nS'_i \middle| a^Ty \leq a^Tz\right\}
\]
is in a $k$-dimensional affine subspace $L$ of $\mathbb{R}^m$. Without loss of generality, we assume that the subspace
\[
L=\{y \in \mathbb{R}^m | y_{k+1}=y_{k+2}=\dots=y_m=0\}.
\]

Next, consider the following linear program in which $\beta_{ij}$ are the decision variables:
\[
\begin{split}
\min \quad & \sum_{i=1}^n\sum_{j=1}^l\beta_{ij}a^Tv^{ij} \\
\st \quad & \sum_{i=1}^n\sum_{j=1}^l\beta_{ij}v^{ij}_s=z_s, \quad \forall s=1,\dots,k, \\
& \sum_{j=1}^l\beta_{ij}=1, \quad \forall i=1,\dots,n, \\
& \beta_{ij} \geq 0, \quad \forall i=1,\dots,n, \: \forall j=1,\dots,l.
\end{split}
\]
Setting $\beta_{ij}=\alpha_j$ gives a feasible solution to the above problem with objective value $a^Tz$. Among all the optimal solutions, pick up a particular vertex solution $\beta^*_{ij}$, which should have at least $nl$ active constraints. We already have $k+n$ active constraints, so the number of nonzero $\beta^*_{ij}$ entries is at most $k+n$. Define
\[
z^i=\sum_{j=1}^l\beta^*_{ij}v^{ij}, \quad z'=\sum_{i=1}^nz^i,
\]
and let $k_i$ be the number of nonzero entries in $\beta^*_{i1},\dots,\beta^*_{il}$. Since $\sum_{j=1}^l\beta^*_{ij}=1$, there must be a nonzero one and thus $k_i \geq 1$. Now we know that $z^i \in \conv_{k_i}S_i$, and $\sum_{i=1}^nk_i \leq k+n$ implies that each $k_i$ cannot exceed $k+1$. The remaining thing to show is $z_s=z'_s$ for $s=k+1,\dots,m$. Because
\[
z' \in \sum_{i=1}^n\conv S'_i=\conv\sum_{i=1}^nS'_i
\]
and
\[
a^Tz'=\sum_{i=1}^n\sum_{j=1}^l\beta^*_{ij}a^Tv^{ij} \leq a^Tz,
\]
$z' \in L$. Since $z \in L$, the last $m-k$ components of both $z$ and $z'$ are all zeros, so $z=z'$.
\end{myproof}

A useful consequence can be made from Theorem~\ref{thm:refined}:

\begin{corollary}\label{cor:refinedhalf}
Let $S_1,S_2,\dots,S_n$ be any subsets of $\mathbb{R}^m$. If $z \in \conv\sum_{i=1}^nS_i$, then there exist integers $1 \leq k_i \leq m$ with $\sum_{i=1}^nk_i \leq m-1+n$ and points $z^i \in \conv_{k_i}S_i$ such that $z_s=\sum_{i=1}^nz^i_s$ for $s=1,\dots,m-1$ and $z_m \geq \sum_{i=1}^nz^i_m$.
\end{corollary}
\begin{myproof}
Using the same argument in the proof of Theorem~\ref{thm:refined}, choose $S'_i \subseteq S_i$ containing finite points such that $z \in \conv\sum_{i=1}^nS'_i$. Since $\conv\sum_{i=1}^nS'_i$ is a compact set,
\[
\inf\left\{w_m \middle| w \in \conv\sum_{i=1}^nS'_i, w_1=z_1, \dots, w_{m-1}=z_{m-1}\right\}
\]
can be achieved by some point $w^*$. $w^*$ is an $(m-1)$-extreme point of $\conv\sum_{i=1}^nS'_i$, and applying Theorem~\ref{thm:refined} on the point $w^*$ gives the desired result.
\end{myproof}

In Section~\ref{sec:dualitygap}, when we prove the bound for the duality gap, Corollary~\ref{cor:refinedhalf} is applied to the epigraph of an $m$-variable function, where $m$ is the number of constraints. The bound in Corollary~\ref{cor:refinedhalf} will be $m+n$ instead of $m+1+n$ from Theorem~\ref{thm:refined} directly. Although the last component in the decomposition given by Corollary~\ref{cor:refinedhalf} is only an inequality, we will see in the following that the inequality is sufficient for our purpose. Therefore, Corollary~\ref{cor:refinedhalf} is the true source of the improvement in \cite{UB2016} from the original bound \eqref{eq:originalbound}.

\section{Characterization of Nonconvexity}\label{sec:nonconvexity}

To improve the bound \eqref{eq:udellbound}, some finer characterization of the nonconvexity of a function has to be introduced. In parallel to the definition of $k$th convex hull of a set, define the \emph{$k$th nonconvexity} $\rho^k(f)$ of a proper function $f$ to be the supremum in \eqref{eq:nonconvexity} taken over the convex combinations of $k$ points instead of arbitrary number of points. Obviously,
\[
0=\rho^1(f) \leq \rho^2(f) \leq \dots \leq \rho(f).
\]
In fact, we have the following property:

\begin{proposition}\label{prop:nonconvexity}
For any proper function $f:\mathbb{R}^n \to \bar\mathbb{R}$, $\rho^{n+1}(f)=\rho(f)$.
\end{proposition}
\begin{myproof}
We only need to show that $\rho(f) \leq \rho^{n+1}(f)$. Choose any convex combination $x=\sum_{j=1}^l \alpha_jx^j$ with all points $x^j \in \dom f$, $\alpha_j \geq 0$, and $\sum_{j=1}^l \alpha_j=1$. Since $(x^j,f(x^j)) \in \epi f$, the point
\[
\left(\sum_{j=1}^l \alpha_jx^j,\sum_{j=1}^l \alpha_jf(x^j)\right) \in \conv\epi f.
\]
Using Corollary~\ref{cor:refinedhalf}, we can find $(y^i,t_i) \in \epi f$, $\beta_i \geq 0$, $i=1,\dots,n+1$, and $\sum_{i=1}^{n+1} \beta_i=1$ such that
\[
x=\sum_{j=1}^l \alpha_jx^j=\sum_{i=1}^{n+1} \beta_iy^i, \quad \sum_{j=1}^l \alpha_jf(x^j) \geq \sum_{i=1}^{n+1} \beta_it_i.
\]
Now
\[
f(x)-\sum_{j=1}^l \alpha_jf(x^j) \leq f\left(\sum_{i=1}^{n+1} \beta_iy^i\right)-\sum_{i=1}^{n+1}\beta_it_i \leq f\left(\sum_{i=1}^{n+1} \beta_iy^i\right)-\sum_{i=1}^{n+1}\beta_if(y^i),
\]
which implies $\rho(f) \leq \rho^{n+1}(f)$.
\end{myproof}

For lower semi-continuous functions, the following proposition provides an equivalent definition for the $k$th nonconvexity, which sheds light on the connection between the concepts of $k$th nonconvexity and $k$th convex hull.

\begin{proposition}\label{prop:nonconvexityalt}
Assume a proper function $f$ is lower semi-continuous and bounded below by some affine function. Let $f^{(k)}$ be the function whose epigraph is the closure of the $k$th convex hull of the epigraph of $f$, i.e.,
\[
\epi f^{(k)}=\cl\conv_k\epi f.
\]
Then
\begin{equation}\label{eq:nonconvexityalt}
\rho^k(f)=\sup_x\{f(x)-f^{(k)}(x)\},
\end{equation}
where we interpret $(+\infty)-(+\infty)=0$.
\end{proposition}
\begin{myproof}
The assumption on the function $f$ implies that $f^{(k)}$ is also a proper function. Consider an arbitrary $k$-point convex combination of points $x^j \in \dom f$, for $j=1,\dots,k$. Following the first step in the proof of Proposition~\ref{prop:nonconvexity}, we have
\[
\left(\sum_{j=1}^k \alpha_jx^j,\sum_{j=1}^k \alpha_jf(x^j)\right) \in \conv_k\epi f \subseteq \cl\conv_k\epi f=\epi f^{(k)}.
\]
Therefore,
\[
f\left(\sum_{j=1}^k \alpha_jx^j\right)-\sum_{j=1}^k \alpha_jf(x^j) \leq f\left(\sum_{j=1}^k \alpha_jx^j\right)-f^{(k)}\left(\sum_{j=1}^k \alpha_jx^j\right),
\]
which implies
\[
\rho^k(f) \leq \sup_x\{f(x)-f^{(k)}(x)\}.
\]

On the other hand, for any $x \in \dom f^{(k)}$,
\[
(x,f^{(k)}(x)) \in \epi f^{(k)}=\cl\conv_k\epi f.
\]
In the case of $x \in \dom f$, by the lower semi-continuity of $f$, for every $\epsilon>0$, there exists $(\kappa,\eta) \in \conv_k\epi f$ which is sufficiently close to $(x,f^{(k)}(x))$ such that
\[
f(\kappa) \geq f(x)-\epsilon, \quad \eta \leq f^{(k)}(x)+\epsilon.
\]
Because $(\kappa,\eta) \in \conv_k\epi f$, there exists $\alpha_j \geq 0$ for $j=1,\dots,k$ such that $\sum_{j=1}^k\alpha_j=1$ and
\[
\kappa=\sum_{j=1}^k\alpha_jx^j, \quad \eta \geq \sum_{j=1}^k\alpha_jf(x^j)
\]
in which $x^j \in \dom f$. Thus
\[
f(x)-f^{(k)}(x) \leq f(\kappa)-\eta+2\epsilon \leq f\left(\sum_{j=1}^k\alpha_jx^j\right)-\sum_{j=1}^k\alpha_jf(x^j)+2\epsilon.
\]
The $f(x)=+\infty$ case can be dealt with similarly, and we can conclude
\[
\sup_x\{f(x)-f^{(k)}(x)\} \leq \rho^k(f)
\]
by letting $\epsilon \to 0$.
\end{myproof}

\begin{remark}
If a proper function $f$ is bounded below by some affine function, then
\[
\epi f^{**}=\cl\conv f
\]
(see \cite[Theorem X.1.3.5]{HL1993}). Therefore, \eqref{eq:nonconvexityalt} can be regarded as a generalization for the alternative definition of nonconvexity
\[
\rho(f)=\sup_x\{f(x)-f^{**}(x)\}
\]
used in \cite{UB2016}.
\end{remark}

In the remaining of this section, three examples will be given to illustrate how to calculate the $k$th nonconvexity of a particular function. The results will be used in Section~\ref{sec:application}.

\begin{example}\label{exmp:linear}
Consider the function
\[
f(x)=f(x_1,\dots,x_n)=\min_{s=1,\dots,n}x_s
\]
defined on the box $0 \leq x \leq 1$, $x \in \mathbb{R}^n$. It is already known that $\rho(f)=(n-1)/n$ (see \cite[Table 1]{UB2016}). By Proposition~\ref{prop:nonconvexity}, $\rho^k(f)=\rho(f)=(n-1)/n$ for $k \geq n+1$.

For $k=1,\dots,n$, as in the proof of Proposition~\ref{prop:nonconvexityalt}, pick up any $k$-point convex combination of points $0 \leq x^j \leq 1$, $j=1,\dots,k$. For a given $i \in \{1,\dots,k\}$, let $s(i)$ be the index such that $x^i_{s(i)}$ is the minimum among $x^i_1,\dots,x^i_n$, then
\[
\begin{split}
f(x)&=\min_{s=1,\dots,n}\left\{\sum_{j=1}^k \alpha_jx^j_s\right\} \leq \sum_{j=1}^k \alpha_jx^j_{s(i)} \\
    &\leq \alpha_ix^i_{s(i)}+1-\alpha^i=\alpha_if(x^i)+1-\alpha^i.
\end{split}
\]
Summing up among $i=1,\dots,k$, we have
\[
kf(x) \leq \sum_{i=1}^k \alpha_if(x^i)+k-1,
\]
which implies
\[
f(x)-\sum_{i=1}^k \alpha_if(x^i) \leq \frac{k-1}{k}\left(1-\sum_{i=1}^k \alpha_if(x^i)\right) \leq \frac{k-1}{k}.
\]
The above argument shows that $\rho^k(f) \leq (k-1)/k$. In fact, the equality holds, which can be easily seen by considering the average of first $k$ points of
\begin{gather*}
x^1=(0,1,\dots,1), \\
x^2=(1,0,\dots,1), \\
\dots, \\
x^n=(1,1,\dots,0).
\end{gather*}
In conclusion,
\[
\rho^k(f)=\begin{dcases*}
\frac{k-1}{k}, & if $k=1,\dots,n$, \\
\frac{n-1}{n}, & if $k \geq n+1$.
\end{dcases*}
\]
\end{example}

\begin{example}\label{exmp:log}
Consider the function
\[
g(x)=g(x_1,\dots,x_n)=-\log\max_{s=1,\dots,n}x_s
\]
defined on the region $x \geq 0$ except $x=0$.

For $k=1,\dots,n$, pick up any $k$-point convex combination. Without loss of generality, assume the coefficients $\alpha_j>0$ for $j=1,\dots,k$. For a given $i \in \{1,\dots,k\}$, let $s(i)$ be the index such that $x^i_{s(i)}$ is the maximum among $x^i_1,\dots,x^i_n$, then
\[
\begin{split}
g(x)&=-\log\max_{s=1,\dots,n}\left\{\sum_{j=1}^k \alpha_jx^j_s\right\} \leq -\log\sum_{j=1}^k \alpha_jx^j_{s(i)} \\
    &\leq -\log(\alpha_ix^i_{s(i)})=-\log\alpha_i+g(x^i).
\end{split}
\]
Summing up among $i=1,\dots,k$ with weight $\alpha_i$, we have
\[
\begin{split}
g(x) &\leq -\sum_{i=1}^k \alpha_i\log\alpha_i+\sum_{i=1}^k \alpha_ig(x^i) \\
     &\leq \log k+\sum_{i=1}^k \alpha_ig(x^i).
\end{split}
\]
The above argument shows that $\rho^k(g) \leq \log k$. In fact, the equality holds, which can be easily seen by considering the average of first $k$ points of
\begin{gather*}
x^1=(1,0,\dots,0), \\
x^2=(0,1,\dots,0), \\
\dots, \\
x^n=(0,0,\dots,1).
\end{gather*}

To calculate $\rho^{n+1}(g)$, define $h(x)=-\log\sum_{s=1}^n x_s$. Then $h(x)$ is convex and $g(x)-\log n \leq h(x) \leq g(x)$. Thus, for any $(n+1)$-point convex combination,
\[
\begin{split}
g\left(\sum_{j=1}^{n+1} \alpha_jx^j\right) &\leq h\left(\sum_{j=1}^{n+1} \alpha_jx^j\right)+\log n \\
                                           &\leq \sum_{j=1}^{n+1} \alpha_jh(x^j)+\log n \\
                                           &\leq \sum_{j=1}^{n+1} \alpha_jg(x^j)+\log n.
\end{split}
\]
Therefore, $\rho^{n+1}(g) \leq \log n$. On the other hand, $\rho^{n+1}(g) \geq \rho^n(g)=\log n$.

In conclusion,
\[
\rho^k(g)=\begin{dcases*}
\log k, & if $k=1,\dots,n$, \\
\log n, & if $k \geq n+1$.
\end{dcases*}
\]
\end{example}

For more complex functions, it is usually hard to compute its $k$th nonconvexity exactly. However, sometimes we can approximate the $k$th nonconvexity of a function by reducing it to another function whose nonconvexity is already known. This technique is demonstrated by the following example:

\begin{example}\label{exmp:capacity}
Consider the function
\[
h_\sigma(x)=h_\sigma(x_1,\dots,x_n)=\sum_{s=1}^n\log\frac{\norm{x}_1-x_s+\sigma}{\norm{x}_1+\sigma}
\]
defined on the box $0 \leq x \leq 1$, $x \in \mathbb{R}^n$. Here $\sigma$ is a parameter in the range $0<\sigma \leq 1$.

Define an auxiliary function
\[
H(x;\sigma)=\prod_{s=1}^n\frac{\norm{x}_1-x_s+\sigma}{\norm{x}_1+\sigma},
\]
then $h_\sigma(x)=\log H(x;\sigma)$. To compute the $k$th nonconvexity for the function $h_\sigma$, we first prove some elementary properties for the function $H(x;\sigma)$.

\begin{lemma}\label{lem:hprop}
The function $H(x;\sigma)$ has the following properties:
\begin{enumerate}[label=(\alph*)]
\item For any vectors $x$ and $y$ in the region $0 \leq x,y \leq \sigma$, if $y \leq x$, then $H(y;\sigma) \geq H(x;\sigma)$.
\item $\sigma H(x;1) \leq H(x;\sigma) \leq H(x;1)$.
\end{enumerate}
\end{lemma}
\begin{myproof}
For any $x$ in the region $0 \leq x \leq \sigma$, the partial derivatives
\begin{align*}
\frac{\partial H(x;\sigma)}{\partial x_i}&=H(x;\sigma)\left(\sum_{s=1}^n\frac{1}{\norm{x}_1-x_s+\sigma}-\frac{1}{\norm{x}_1-x_i+\sigma}-\frac{n}{\norm{x}_1+\sigma}\right) \\
&=H(x;\sigma)\left(\sum_{s=1}^n\frac{x_s}{(\norm{x}_1-x_s+\sigma)(\norm{x}_1+\sigma)}-\frac{1}{\norm{x}_1-x_i+\sigma}\right) \\
&\leq H(x;\sigma)\left(\sum_{s=1}^n\frac{x_s}{\norm{x}_1(\norm{x}_1+\sigma)}-\frac{1}{\norm{x}_1+\sigma}\right)=0,
\end{align*}
which gives the first property.

For the second property, it is obvious to see that $H(x;\sigma) \leq H(x;1)$. The other inequality is equivalent to
\[
p(\sigma)=\frac{1}{\sigma}H(x;\sigma)-H(x;1) \geq 0.
\]
The partial derivative
\begin{align*}
\frac{\partial H(x;\sigma)}{\partial\sigma}&=H(x;\sigma)\left(\sum_{s=1}^n\frac{1}{\norm{x}_1-x_s+\sigma}-\frac{n}{\norm{x}_1+\sigma}\right) \\
&=H(x;\sigma)\sum_{s=1}^n\frac{x_s}{(\norm{x}_1-x_s+\sigma)(\norm{x}_1+\sigma)} \\
&\leq H(x;\sigma)\sum_{s=1}^n\frac{x_s}{\sigma(\norm{x}_1+\sigma)}=\frac{1}{\sigma}H(x;\sigma)\frac{\norm{x}_1}{\norm{x}_1+\sigma} \leq \frac{1}{\sigma}H(x;\sigma)
\end{align*}
implies that
\[
p'(\sigma)=-\frac{1}{\sigma^2}H(x;\sigma)+\frac{1}{\sigma}\frac{\partial H(x;\sigma)}{\partial\sigma} \leq 0.
\]
Therefore, the function $p(\sigma)$ is nonincreasing. Together with $p(1)=0$, we have proved the nonnegativity of $p(\sigma)$.
\end{myproof}

To upper bound the $k$th nonconvexity of the function $h_\sigma$, consider arbitrary points $x^j$ for $j=1,\dots,k$ with corresponding combination weights $\alpha_j>0$. Define $k$ vectors $y^1,\dots,y^k$ in $\mathbb{R}^k$ by
\begin{gather*}
y^1=(1/H(x^1;1),0,\dots,0), \\
y^2=(0,1/H(x^2;1),\dots,0), \\
\dots, \\
y^k=(0,0,\dots,1/H(x^k;1)).
\end{gather*}
Using the result of nonconvexity for the function $g$ given in Example~\ref{exmp:log} and the properties proved in Lemma~\ref{lem:hprop}, we have
\begin{align*}
h_\sigma&\left(\sum_{j=1}^k\alpha_jx^j\right)=\log H\left(\sum_{j=1}^k\alpha_jx^j;\sigma\right) \leq \log H\left(\sum_{j=1}^k\alpha_jx^j;1\right) && \text{by property (b)} \\
&\leq \log\max_{j=1,\dots,k}H(\alpha_jx^j;1) && \text{by property (a)} \\
&\leq \log\max_{j=1,\dots,k}\frac{1}{\alpha_j}H(\alpha_jx^j;\alpha_j)=\log\max_{j=1,\dots,k}\frac{1}{\alpha_j}H(x^j;1) && \text{by property (b)} \\
&=g\left(\sum_{j=1}^k\alpha_jy^j\right) \leq \sum_{j=1}^k\alpha_jg(y^j)+\log k && \text{by the nonconvexity of $g$} \\
&=\sum_{j=1}^k\alpha_j\log H(x^j;1)+\log k \leq \sum_{j=1}^k\alpha_j\log\frac{1}{\sigma}H(x^j;\sigma)+\log k && \text{by property (b)} \\
&=\sum_{j=1}^k\alpha_jh_\sigma(x^j)+\log\frac{k}{\sigma}.
\end{align*}
The above argument shows that the $k$th nonconvexity $\rho^k(h_\sigma) \leq \log(k/\sigma)$.
\end{example}

\begin{remark}
In Example~\ref{exmp:capacity} above, an upper bound for the $k$th nonconvexity of function $h_\sigma$ is obtained by a reduction from the nonconvexity of $g$ in Example~\ref{exmp:log}. Along this line of thoughts, it is conceivable to find the exact value for the $k$th nonconvexity of $h_\sigma$ if we are able to reduce $h_\sigma$ to itself (but with just $k$ variables).
\end{remark}

\section{Bounding Duality Gap}\label{sec:dualitygap}

Now we can state the main result on the duality gap between the primal problem \eqref{eq:primal} and the dual problem \eqref{eq:dual}.

\begin{theorem}\label{thm:dualitygap}
Assume that the primal problem \eqref{eq:primal} is feasible, i.e., $p<+\infty$. Then there exist integers $1 \leq k_i \leq m+1$ such that $\sum_{i=1}^n k_i \leq m+n$ and the duality gap
\[
p-d \leq \sum_{i=1}^n \rho^{k_i}_i.
\]
Here $\rho^k_i=\rho^k(f_i)$ is the $k$th nonconvexity of function $f_i$.
\end{theorem}

First, let us define the perturbation function $v: \mathbb{R}^m \to \bar\mathbb{R}$ by letting $v(z)$ be the optimal value of the perturbed problem
\[
\begin{split}
\min \quad & \sum_{i=1}^n f_i(x^i) \\
\st \quad  & \sum_{i=1}^n A_ix^i \leq b+z.
\end{split}
\]
As is the case with convex optimization, $p=v(0)$ and $d=v^{**}(0)$ (see \cite[Lemma~2.2]{ET1999}).

\begin{lemma}\label{lem:perturbfunc}
The perturbation function $v$ is lower semi-continuous.
\end{lemma}

\begin{myproof}
Pick any $z \in \mathbb{R}^m$. We want to show that if $z^k \to z$ as $k \to \infty$,
\[
l=\liminf_{k \to \infty}v(z^k) \geq v(z).
\]
The above inequality clearly holds when $l=+\infty$. If $l<+\infty$, by considering a subsequence of $\{v(z^k)\}_{k=1}^\infty$, without loss of generality we can assume $v(z^k)<+\infty$ for each $k$ and
\[
\lim_{k \to \infty}v(z^k)=l.
\]
For each $k$, find $(\hat x^{1k},\dots,\hat x^{nk})$ attaining the optimal value of the perturbed problem related to $v(z^k)$, i.e.,
\[
v(z^k)=\sum_{i=1}^n f_i(\hat x^{ik}), \quad \sum_{i=1}^n A_i\hat x^{ik} \leq b+z^k.
\]
By extracting a convergent subsequence for each $\{\hat x^{ik}\}_{k=1}^\infty$, we can assume $\{\hat x^{ik}\}_{k=1}^\infty$ has a limit $x^i$. Then
\[
\sum_{i=1}^n A_ix^i \leq b+z,
\]
which implies that $(x^1,\dots,x^n)$ is feasible to the perturbed problem related to $v(z)$, so
\[
\sum_{i=1}^n f_i(x^i) \geq v(z).
\]
Now
\[
l=\lim_{k \to \infty}\sum_{i=1}^n f_i(\hat x^{ik}) \geq \sum_{i=1}^n \liminf_{k \to \infty}f_i(\hat x^{ik}) \geq \sum_{i=1}^n f_i(x^i) \geq v(z),
\]
because $f_i$ is lower semi-continuous.
\end{myproof}

\begin{myproof}[Proof of Theorem~\ref{thm:dualitygap}]
Since \eqref{eq:primal} is feasible, $v(0)=p<+\infty$. Let
\[
\xi=\inf\sum_{i=1}^n f_i(x^i),
\]
then by our assumption of $f_i$, $\xi$ is finite. $v(z) \geq \xi$ for all $z \in \mathbb{R}^m$.  As a consequence, $v(z)$ is bounded below by some affine function, so
\[
-\infty<v^{**}(0) \leq v(0)<+\infty, \quad \epi v^{**}=\cl\conv\epi v.
\]

By Lemma~\ref{lem:perturbfunc}, $v$ is lower semi-continuous. Since $(0,v^{**}(0)) \in \epi v^{**}=\cl\conv\epi v$, for every $\epsilon>0$, there exists $(\kappa,\eta) \in \conv\epi v$ which is sufficiently close to $(0,v^{**}(0))$ such that
\begin{equation}\label{eq:approx}
v(\kappa) \geq v(0)-\epsilon, \quad \eta \leq v^{**}(0)+\epsilon.
\end{equation}
Because $(\kappa,\eta) \in \conv\epi v$, there exists some integer $l$ and $\alpha_j \geq 0$ for $j=1,\dots,l$ such that $\sum_{j=1}^l \alpha_j=1$ and
\[
\kappa=\sum_{j=1}^l\alpha_jz^j, \quad \eta \geq \sum_{j=1}^l\alpha_jv(z^j)
\]
in which $z^j \in \dom v$.

For each $j=1,\dots,l$, find $(\hat x^{1j},\dots,\hat x^{nj})$ attaining the optimal value of the perturbed problem related to $v(z^j)$, i.e.,
\[
v(z^j)=\sum_{i=1}^n f_i(\hat x^{ij}), \quad \sum_{i=1}^n A_i\hat x^{ij} \leq b+z^j,
\]
which means there exists some vector $w^j \in \mathbb{R}^m_+$ such that
\[
(b+z^j-w^j,v(z^j)) \in \sum_{i=1}^n C_i,
\]
where
\[
C_i=\{(A_ix^i,f_i(x^i)) | f_i(x^i)<+\infty, x^i \in \mathbb{R}^{n_i}\}.
\]
Taking convex combination of the points above, we have
\[
\left(b+\kappa-\sum_{j=1}^l\alpha_jw^j,\sum_{j=1}^l\alpha_jv(z^j)\right) \in \conv\sum_{i=1}^n C_i.
\]

Now we can apply Corollary~\ref{cor:refinedhalf}, which gives points $(r^i,s_i) \in \conv_{k_i}C_i$ with $1 \leq k_i \leq m+1$ such that
\[
b+\kappa \geq b+\kappa-\sum_{j=1}^l\alpha_jw^j=\sum_{i=1}^n r^i, \quad \eta \geq \sum_{j=1}^l\alpha_jv(z^j) \geq \sum_{i=1}^n s_i
\]
and $\sum_{i=1}^n k_i \leq m+n$. Since $(r^i,s_i) \in \conv_{k_i}C_i$, there exists $\tilde x^{ij} \in \mathbb{R}^{n_i}$, $\beta_{ij} \geq 0$ for $j=1,\dots,k_i$ such that $f_i(\tilde x^{ij})<+\infty$, $\sum_{j=1}^{k_i} \beta_{ij}=1$ and
\[
r^i=\sum_{j=1}^{k_i} \beta_{ij}A_i\tilde x^{ij}, \quad s_i=\sum_{j=1}^{k_i} \beta_{ij}f_i(\tilde x^{ij}).
\]
Thus,
\begin{equation}\label{eq:feasibilityproof}
\kappa \geq \sum_{i=1}^n\sum_{j=1}^{k_i} \beta_{ij}A_i\tilde x^{ij}-b=\sum_{i=1}^n A_i \sum_{j=1}^{k_i} \beta_{ij}\tilde x^{ij}-b,
\end{equation}
and
\begin{equation}\label{eq:dualitygapproof}
\sum_{i=1}^n \rho^{k_i}_i+\eta \geq \sum_{i=1}^n\left(\rho^{k_i}_i+\sum_{j=1}^{k_i} \beta_{ij}f_i(\tilde x^{ij})\right) \geq \sum_{i=1}^n f_i\left(\sum_{j=1}^{k_i} \beta_{ij}\tilde x^{ij}\right).
\end{equation}

From \eqref{eq:feasibilityproof} we know that
\[
\left(\sum_{j=1}^{k_1} \beta_{1j}\tilde x^{1j},\dots,\sum_{j=1}^{k_n} \beta_{nj}\tilde x^{nj}\right)
\]
is feasible to the perturbed problem related to $v(\kappa)$, so the corresponding objective value
\[
\sum_{i=1}^n f_i\left(\sum_{j=1}^{k_i} \beta_{ij}\tilde x^{ij}\right) \geq v(\kappa).
\]
The above inequality, \eqref{eq:dualitygapproof} and \eqref{eq:approx} imply
\[
v^{**}(0)+\epsilon+\sum_{i=1}^n \rho^{k_i}_i \geq v(0)-\epsilon.
\]

We finish the proof by letting $\epsilon \to 0$ and choose the worst case of $\sum_{i=1}^n \rho^{k_i}_i$ encountered in this process.
\end{myproof}

From a computational viewpoint, since we do not know the $k_i$ that appeared in Theorem~\ref{thm:dualitygap}, in order to find a number for the bound, we have to find the worst case $k_i$ by solving the following optimization problem
\begin{equation}\label{eq:findbound}
\begin{split}
\max \quad & \sum_{i=1}^n \rho^{k_i}_i \\
\st \quad  & 1 \leq k_i \leq m+1, \: k_i \in \mathbb{Z}, \quad \forall i=1,\dots,n, \\
           & \sum_{i=1}^n k_i \leq m+n.
\end{split}
\end{equation}

Let $B$ be the optimal value of \eqref{eq:findbound}, then
\[
B \leq \sum_{i=1}^n \rho(f_i).
\]
On the other hand, since for any feasible solution of \eqref{eq:findbound}, the number of $k_i$ with $k_i \geq 2$ is bounded by $m$, so
\[
B=\sum_{i:k_i \geq 2} \rho^{k_i}_i \leq \sum_{i=1}^m \rho(f_i)
\]
if $\rho(f_1) \geq \dots \geq \rho(f_n)$. The above argument shows that the bound $B$ given by the optimization problem \eqref{eq:findbound} is at least as tight as the bound \eqref{eq:udellbound} in \cite{UB2016}.

To illustrate the procedure to calculate the bound $B$, consider the simple case where all the $x^i$ in the primal problem \eqref{eq:primal} are one-dimensional and all the functions $f_i$ equal to the same function $f$. In this case, $\rho^{k_i}_i=\rho(f)$ if $k_i \geq 2$. The optimal value to \eqref{eq:findbound} is attained when the number of $k_i$ that equal to 2 is maximized, so the optimal value is $\min\{m,n\}\rho(f)$, which is the same as the result given by \eqref{eq:udellbound}. The Example~1 used in \cite{UB2016} belongs to this category. It hence explains why the bound \eqref{eq:udellbound} is tight for that example. However, if the dimension of $x^i$ in the primal problem can be arbitrarily large, the bound \eqref{eq:udellbound} can be very loose. As will be shown in Section~\ref{sec:application}, the difference between the bound \eqref{eq:udellbound} and the exact duality gap tends to infinity for a series of problems.

\section{Applications}\label{sec:application}

\subsection{Joint Routing and Congestion Control in Networking}

In this part, we will first apply the previous result to the network utility maximization problem. Consider a network with $N$ users and $L$ links. Let a strictly positive vector $c \in \mathbb{R}^L$ contain the capacity of each link. Each user $i$ has $K^i$ available paths to send its commodity. We assume that the users are sorted such that $K^1 \geq \dots \geq K^N$. The routing matrix of user $i$, denoted by $R^i$, is a $L \times K^i$ matrix defined by
\[
R^i_{lk}=\begin{dcases*}
1, & if the $k$th path of user $i$ passes through link $l$, \\
0, & otherwise.
\end{dcases*}
\]

Let $x^i \in \mathbb{R}^{K^i}$ be the vector in which $x^i_k$ is the amount of commodity sent by user $i$ on its $k$th path. Assume that each user $i$ has a utility function $U_i(\cdot)$ depending on the vector $x^i$, then the network utility maximization problem can be written as
\begin{equation}\label{eq:utilitymax}
\begin{split}
\max \quad & \sum_{i=1}^N U_i(x^i) \\
\st \quad  & \sum_{i=1}^N R^ix^i \leq c, \\
           & x^i \geq 0, \quad \forall i=1,\dots,N.
\end{split}
\end{equation}

If all the utility functions $U^i(\cdot)$ are concave, then the above problem \eqref{eq:utilitymax} can be solved by standard convex optimization techniques. Difficulty arises when $U^i(\cdot)$ is not concave. For example, if we restrict each user to choose only one path (single-path routing) and want to maximize the total throughput of the network, then the corresponding utility function is
\[
U_i(x^i)=\max_{s=1,\dots,K^i}x^i_s.
\]
Define
\[
f_i(x^i)=\begin{dcases*}
\min_{s=1,\dots,K^i}(-x^i_s), & if $0 \leq x^i \leq \norm{c}_\infty$, \\
+\infty,                      & otherwise.
\end{dcases*}
\]
Here $\norm{c}_\infty$ is the maximum link capacity in the network. Now the original network utility maximization problem \eqref{eq:utilitymax} is equivalent to the following problem:
\begin{equation}\label{eq:utilitymaxalt}
\begin{split}
\min \quad & \sum_{i=1}^Nf_i(x^i) \\
\st \quad  & \sum_{i=1}^NR^ix^i \leq c.
\end{split}
\end{equation}
The above problem is a particular case of the general optimization problem with separable objectives \eqref{eq:primal} studied in this paper. Using the same technique as shown in Example~\ref{exmp:linear}, we can prove that
\[
\rho^k(f_i) \leq \frac{k-1}{k}\norm{c}_\infty, \quad \rho(f_i)=\frac{K^i-1}{K^i}\norm{c}_\infty.
\]

In the following, suppose each user has a large number of paths to select. More explicitly, $K^i \geq L+1$ is assumed for user $i$. Based on the bound \eqref{eq:udellbound}, the duality gap is bounded by
\[
\sum_{i=1}^{\min\{N,L\}}\frac{K^i-1}{K^i}\norm{c}_\infty,
\]
which is at least
\[
\min\{N,L\}\frac{L}{L+1}\norm{c}_\infty.
\]
In contrast, by Theorem~\ref{thm:dualitygap}, the duality gap is bounded by the optimal value of the following optimization problem:
\[
\begin{split}
\max \quad & \sum_{i=1}^N \frac{k_i-1}{k_i}\norm{c}_\infty \\
\st \quad  & 1 \leq k_i \leq L+1, \: k_i \in \mathbb{Z}, \quad \forall i=1,\dots,N, \\
           & \sum_{i=1}^N k_i \leq N+L.
\end{split}
\]
Let $N'$ be the number of users whose $k_i \geq 2$, then $0 \leq N' \leq \min\{N,L\}$. If $N'>0$, using the inequality between arithmetic mean and harmonic mean,
\[
\begin{split}
\sum_{i=1}^N \frac{k_i-1}{k_i}&=\sum_{i:k_i \geq 2} \frac{k_i-1}{k_i}=N'-\sum_{i:k_i \geq 2} \frac{1}{k_i} \\
&\leq N'-\frac{N'^2}{\sum_{i:k_i \geq 2} k_i} \leq N'-\frac{N'^2}{N'+L} \\
&=\frac{L}{1+L/N'} \leq \min\{N,L\}\frac{L}{L+\min\{N,L\}}.
\end{split}
\]
Taking the $N \geq L$ case as an example, by the above inequality, we can bound the duality gap by $L\norm{c}_\infty/2$, essentially half of the bound given by \eqref{eq:udellbound}. The same result was obtained by a specialized technique in \cite{BTT2016}.

Next, we consider another case in which each user has logarithmic utility but still must choose only one path. The utility function of user $i$ can be written as
\[
U_i(x^i)=\log\max_{s=1,\dots,K^i}x^i_s.
\]
Define
\[
g_i(x^i)=\begin{dcases*}
-\log\max_{s=1,\dots,K^i}x^i_s, & if $0 \leq x^i \leq \norm{c}_\infty$, $x^i \neq 0$, \\
+\infty,                       & otherwise.
\end{dcases*}
\]
Then the network utility maximization problem \eqref{eq:utilitymax} is equivalent to the problem obtained by replacing $f_i$ with $g_i$ in \eqref{eq:utilitymaxalt}. Using the result in Example~\ref{exmp:log},
\[
\rho^k(g_i) \leq \log k, \quad \rho(g_i)=\log K^i.
\]

Applying the bound \eqref{eq:udellbound} to this case, we can bound the duality gap by
\begin{equation}\label{eq:oldboundlog}
\sum_{i=1}^{\min\{N,L\}} \log K^i,
\end{equation}
which is at least $\min\{N,L\}\log(L+1)$. On the other hand, by Theorem~\ref{thm:dualitygap}, the duality gap is bounded by the optimal value of the following optimization problem
\begin{equation}\label{eq:loggapmax}
\begin{split}
\max \quad & \sum_{i=1}^N \log{k_i} \\
\st \quad  & 1 \leq k_i \leq L+1, \: k_i \in \mathbb{Z}, \quad \forall i=1,\dots,N, \\
           & \sum_{i=1}^N k_i \leq N+L.
\end{split}
\end{equation}
If we still let $N'$ be the number of users whose $k_i \geq 2$, then $0 \leq N' \leq \min\{N,L\}$ and the above bound
\[
\begin{split}
\sum_{i=1}^N \log{k_i}&=\sum_{i:k_i \geq 2} \log{k_i}=\log\prod_{i:k_i \geq 2} k_i \\
                     &\leq \log\left(\frac{\sum_{i:k_i \geq 2} k_i}{N'}\right)^{N'} \leq \log\left(\frac{N'+L}{N'}\right)^{N'} \leq \min\{N,L\}\log\left(1+\frac{L}{\min\{N,L\}}\right),
\end{split}
\]
where in the last step the monotonicity of the function $(1+1/x)^x$ is used. Note that the new bound is qualitatively tighter than the bound \eqref{eq:oldboundlog} provided by \eqref{eq:udellbound}.

\subsection{Dynamic Spectrum Management in Communication}

Consider a communication system consisting of $L$ users sharing a common band. The band is divided equally into $N$ tones. Each user $l$ has a power budget $p_l$ which can be allocated across all the tones. Let $x^i_l$ be the power of user $l$ allocated on tone $i$. Due to the crosstalk interference between users, the total noise for a user on tone $i$ is the sum of a background noise $\sigma_i$ and the power of all other users on the same tone. Therefore, the achievable transmission rate of user $l$ on tone $i$ is given by
\[
u^i_l=\frac{1}{N}\log\left(1+\frac{x^i_l}{\norm{x^i}_1-x^i_l+\sigma_i}\right).
\]

The dynamic spectrum management problem is to maximize the total throughput of all users under the power budget constraints, which can be formulated as the following nonconcave optimization problem:
\begin{equation}\label{eq:dsm}
\begin{split}
\max \quad & \sum_{l=1}^L\sum_{i=1}^Nu^i_l \\
\st \quad  & \sum_{i=1}^N x^i_l \leq p_l, \quad \forall l=1,\dots,L, \\
           & x^i_l \geq 0, \quad \forall i=1,\dots,N,\;\forall l=1,\dots,L.
\end{split}
\end{equation}
For simplicity, we assume that the noises $\sigma_i \leq 1$ and the power budgets $p_l \leq 1$ (if not, then scale all the $\sigma_i$ and $p_l$ simultaneously). The latter requires all the variables $x^i_l \leq 1$. Using the function $h_\sigma$ introduced in Example~\ref{exmp:capacity}, the objective function of \eqref{eq:dsm} can be rewritten as a sum of separable objectives:
\[
\sum_{l=1}^L\sum_{i=1}^Nu^i_l=-\frac{1}{N}\sum_{i=1}^Nh_{\sigma_i}(x^i).
\]

For the purpose of designing dual algorithms, it is of great interest to estimate the duality gap for the problem \eqref{eq:dsm}. In \cite{YL2006}, the authors showed that the duality gap will tend to zero if the number of users $L$ is fixed and the number of tones $N$ goes to infinity. \cite{LZ2009} further determined the convergence rate of the duality gap to be $O(1/\sqrt N)$. Using the bound \eqref{eq:udellbound}, we now demonstrate how to improve the convergence rate estimation to $O(1/N)$, which can be only achieved by the method in \cite{LZ2009} in the special case where all the noises $\sigma_i$ are the same.\footnote{The paper \cite{LZ2009} actually studied the generalization of problem \eqref{eq:dsm} under the existence of path loss coefficient between different users. However, the argument for $O(1/N)$ provided here can also be adapted to the general problem.}

Example~\ref{exmp:capacity} proves that the nonconvexity
\[
\rho^k(h_{\sigma_i}) \leq \log\frac{k}{\sigma_i} \leq \log\frac{k}{\sigma}, \quad \rho(h_{\sigma_i})=\rho^{L+1}(h_{\sigma_i}) \leq \log\frac{L+1}{\sigma},
\]
where $\sigma$ is the minimum among all the noises $\sigma_i$, so \eqref{eq:udellbound} implies that the duality gap is upper bounded by
\begin{equation}\label{eq:oldboundcapacity}
\frac{\min\{N,L\}}{N}\log\frac{L+1}{\sigma},
\end{equation}
which is in the order of $O(1/N)$ if $L$ is fixed and $N$ increases.

In order to further improve the estimation \eqref{eq:oldboundcapacity} for the duality gap, we can resort to Theorem~\ref{thm:dualitygap} and follow the exact same steps for solving \eqref{eq:loggapmax}, which shows that the duality gap is upper bounded by
\[
\frac{\min\{N,L\}}{N}\log\frac{1+L/\min\{N,L\}}{\sigma}.
\]
Like the previous example, our bound is still tighter than the one \eqref{eq:oldboundcapacity} from \eqref{eq:udellbound}.

\section{Generalization with Nonlinear Constraints}\label{sec:nonlinear}

The idea in this paper can also be applied to separable problems with nonlinear constraints such as
\begin{equation}\label{eq:primalnonlinear}
\begin{split}
\min \quad & \sum_{i=1}^n f_i(x^i) \\
\st \quad  & \sum_{i=1}^n g_i(x^i) \leq b.
\end{split}
\end{equation}
Here each $g_i: \mathbb{R}^{n_i} \to \mathbb{R}^m$ is a lower semi-continuous function. Note that the problem \eqref{eq:primal} we studied above is a special case of the above optimization problem \eqref{eq:primalnonlinear} if we choose $g_i(x^i)=A_ix^i$. Let $y \in \mathbb{R}^m_+$ be the dual variables, then the Lagrangian is
\[
L(x,y)=\sum_{i=1}^n (f_i(x^i)+y^Tg_i(x^i))-y^Tb
\]
and the Lagrange dual problem of \eqref{eq:primalnonlinear} is
\[
d=\sup_{y \geq 0}\:\inf_x L(x,y).
\]

If the functions $g_i$ are not convex, the duality gap should not only depend on the nonconvexity of functions $f_i$ but also somehow relate to the functions $g_i$. Like \cite{BS1982}, we want to define the $k$th order nonconvexity of a proper function $f:\mathbb{R}^n \to \bar\mathbb{R}$ with respect to another function $g: \mathbb{R}^n \to \mathbb{R}^m$, denoted by $\rho^k(f,g)$. To do this, we introduce the following auxiliary function
\[\label{eq:auxfunc}
h^k(x)=\inf_{z \in \mathbb{R}^n} \left\{f(z) \middle| g(z) \leq \sum_{j=1}^k \beta_jg(x^j), \: \forall \beta_j \geq 0, x^j \in \mathbb{R}^n \text{ s.t. } \sum_{j=1}^k \beta_j=1 \text{ and } x=\sum_{j=1}^k \beta_jx^j \right\}.
\]
Then $\rho^k(f,g)$ is defined by
\[
\rho^k(f,g)=\sup\left\{h^k\left(\sum_{j=1}^k \alpha_jx^j\right)-\sum_{j=1}^k \alpha_jf(x^j)\right\}.
\]
over all possible convex combinations $\alpha_j \geq 0$, $j=1,\dots,k$, with $\sum_{j=1}^k \alpha_j=1$ of points $x^j$ satisfying $f(x^j)<+\infty$. If the function $g$ is convex, then in the infimum of \eqref{eq:auxfunc} we can choose $z=x$, which gives $h^k(x) \leq f(x)$ and $\rho^k(f,g) \leq \rho^k(f)$. However, the last inequality may not hold if $g$ is not convex.

The proof of Theorem~\ref{thm:dualitygap} can be modified accordingly to the case with nonlinear constraints by replacing $\rho^{k_i}(f_i)$ with $\rho^{k_i}(f_i,g_i)$. In the case when all $g_i$ are convex, $\rho^{k_i}(f_i,g_i) \leq \rho^{k_i}(f_i)$, which implies that the original conclusion in Theorem~\ref{thm:dualitygap} remains true even for convex but nonlinear constraints.

\section{Conclusion}

The improvements obtained in this paper are attributed to two sources. First, instead of using a single number measurement, a series of numbers are introduced to characterize the nonconvexity of a function in a potentially much finer manner. This is based on the concept of $k$th convex hull of a set, which allows us to differentiate different levels of nonconvexity for nonconvex sets. Second, for a separable nonconvex problem, we do not approximate each subproblem individually as people had done before. Instead, by considering all subproblems jointly and noticing that the total deviation of every subproblem to a convex problem is bounded, we reach a much tighter estimation.


Betti Curves of Rank One Symmetric Matrices

Abstract: Betti curves of symmetric matrices were introduced in (Giusti et. al., 2015) as a new class of matrix invariants that depend only on the relative ordering of matrix entries. These invariants are computed using persistent homology, and can be used to detect underlying structure in biological data that may otherwise be obscured by monotone nonlinearities. Here we prove three theorems that fully characterize the Betti curves of rank 1 symmetric matrices. We then illustrate how these Betti curve signatures arise in natural data obtained from calcium imaging of neural activity in zebrafish.

\section{Introduction}

Measurements in biology are often related to the underlying variables in a nonlinear fashion. For example, a brighter calcium imaging signal indicates higher neural activity, but a neuron with twice the activity of another does not produce twice the brightness. This is because the measurement is a monotone nonlinear function of the desired quantity. How can one detect meaningful structure in matrices derived from such data? One solution is to try to estimate the monotone nonlinearity, and invert it. A different approach, introduced in \cite{Giusti_et_al}, is to compute new matrix invariants that depend only on the relative ordering of matrix entries, and are thus invariant to the effects of monotone nonlinearities.

Figure~\ref{fig:fig1}a illustrates the pitfalls of trying to use traditional linear algebra methods to estimate the underlying rank of a matrix in the presence of a monotone nonlinearity. The original $100\times100$ matrix $A$ is symmetric of rank 5 (top left cartoon), and this is reflected in the singular values (bottom left). In contrast, the matrix $B$ with entries $B_{ij} = f(A_{ij})$ appears to be full rank, despite having exactly the same {\it ordering} of matrix entries: $B_{ij} > B_{k\ell}$ if and only if $A_{ij} > A_{k\ell}$. The apparently high rank of $B$ is purely an artifact of the monotone nonlinearity $f$. This motivates the need for matrix invariants, like Betti curves, that will give the same answer for $A$ and $B$. Such invariants depend only on the ordering of matrix entries and do not ``see'' the nonlinearity \cite{Giusti_et_al}.

\begin{figure}[!ht]
    \centering
    \includegraphics[width = \textwidth]{rank_5_distortion.png}
   \caption{{\bf Low rank structure is obscured by monotone nonlinearities.} The matrix $A$ is a $100 \times 100$ symmetric matrix of rank $5$. $B$ is obtained from $A$ by applying the monotone nonlinearity $f(x) = 1 - e^{-5x}$ entrywise. This alters the singular values so that $B$ appears to be full rank, but this is purely an artifact of the nonlinearity. (b) An axis simplex in $\mathbb{R}^{10}$ generates a distance matrix that is full rank, and this is evident in the singular values. The Betti curves, however, are consistent with an underlying rank of 1.}
   \label{fig:fig1}
   \vspace{-.15in}
\end{figure}

In this paper we will fully characterize all Betti curves that can arise from rank 1 matrices. This provides necessary conditions that must be satisfied by any matrix whose underlying rank is 1 -- that is, whose ordering is the same as that of a rank 1 matrix.  We then apply these results to calcium imaging data of neural activity in zebrafish and find that correlation matrices for cell assemblies have Betti curve signatures of rank 1.

It turns out that many interesting matrices have an underlying rank of 1. For example, consider the distance matrix induced by an \textit{axis simplex}, meaning a simplex in $\mathbb{R}^n$ whose vertices are $\alpha_1 \mathbf{e_1}, \alpha_2 \mathbf{e_2}, \dots, \alpha_n \mathbf{e_n}$, where $\mathbf{e_1}, \mathbf{e_2}, \dots, \mathbf{e_n}$ are the standard basis vectors. The Euclidean distance matrix $D$ for these points has off-diagonal entries $D_{ij} = \sqrt{\alpha_i^2 + \alpha_j^2},$ and is typically full rank (Figure~\ref{fig:fig1}b, top). But this matrix has underlying rank 1. To see this, observe that the matrix with entries $\sqrt{\alpha_i^2 + \alpha_j^2}$ has the same ordering as the matrix with entries $\alpha_i^2 + \alpha_j^2$, and this in turn has the same ordering as the matrix with entries $e^{\alpha_i^2 + \alpha_j^2} = e^{\alpha_i^2} e^{\alpha_j^2}$, which is clearly rank 1.  Although the singular values do not reflect this rank 1 structure, the Betti curves do (Figure~\ref{fig:fig1}b, bottom).

Adding another vertex at the origin to an axis simplex yields a {\it simplex with an orthogonal corner}. The distance matrix is now $(n+1)\times(n+1)$, and is again given by $D_{ij} = \sqrt{\alpha_i^2 + \alpha_j^2}$ but with $i,j = 0,\ldots,n$ and $\alpha_0 = 0$. The same argument as above shows that this matrix also has underlying rank 1.

The organization of this paper is as follows. In Section 2 we provide background on Betti curves and prove three theorems characterizing the Betti curves of symmetric rank 1 matrices. In Section 3 we illustrate our results by computing Betti curves for pairwise correlation matrices obtained from calcium imaging data of neural activity in zebrafish. All Betti curves were computed using the well-known persistent homology package Ripser \cite{ripser}.

\section{Betti curves of symmetric rank 1 matrices}

{\bf Betti curves.} Given a symmetric $n \times n$ matrix $M$, the ${n \choose 2}$ off-diagonal entries $M_{ij}$ for $i<j$ can be sorted in increasing order. We denote by $\widehat{M}$ be the corresponding ordering matrix, where $\widehat{M}_{ij} = k$ if $M_{ij}$ is the $k$-th smallest entry. From the ordering, we can construct an increasing sequence of graphs $\{G_t(M)\}$ for $t \in [0,1]$ as follows: for each graph, the vertex set is $1,\ldots,n$ and the the edge set is
$$E(G_t(M)) = \left\{(i,j) \mid \widehat{M}_{ij} \leq t {n \choose 2}\right\}.$$
Note that for $t = 0$, $G_t(M)$ has no edges, while at $t = 1$ it is the complete graph with all edges. Although $t$ is a continuous parameter, it is clear that there are only a finite number of distinct graphs in the family $\{G_t(M)\}_{t \in [0,1]}$. Each new graph differs from the previous one by the addition of an edge. When it is clear from the context, we will denote $G_t(M)$ as simply $G_t$.

\begin{figure}[!h]
\begin{center}
\includegraphics[width = 4.25in]{clique_complex_example.png}
\end{center}
\vspace{-.2in}
\caption{{\bf Betti curves of real symmetric matrices.} (a) $\widehat{M}$ is the ordering matrix for a symmetric $5 \times 5 $ matrix $M$ of rank $4$. (b) We plot the Betti curves $\beta_0, \ldots, \beta_3$ induced by $M$. (c) The filtration of clique complexes $X(G_t)$ for $\widehat{M}$ from which the Betti curves are computed. Note that at times $t = 4/10$ and $t = 5/10$, a one-dimensional hole appears, contributing to nonzero $\beta_1$ values in (b). At time $t = 6/10$, this hole is filled by the cliques created after adding edge $(2,5)$, and thus $\beta_1(t)$ goes back to zero.}
\label{fig2}
\vspace{-.2in}
\end{figure}

For each graph $G_t$, we build a \textit{clique complex} $X(G_t)$, where every $k$-clique in $G_t$ is filled in by a $k$-simplex. We thus obtain a filtration of clique complexes $\{X(G_t) \mid t \in [0,1] \}$. The $i$-th Betti curve of $M$ is defined as
$$\beta_i(t) = \beta_i(X(G_t)) = \mathrm{rank} H_i(X(G_t),{\bf k}),$$
where $H_i$ is the $i$-th homology group with coefficients in the field ${\bf k}$. The Betti curves clearly depend only on the ordering of the entries of $M$, and are thus invariant to (increasing) monotone nonlinearities like the one shown in Figure~\ref{fig:fig1}a. They are also naturally invariant to permutations of the indices $1,\ldots,n$, provided rows and columns are permuted in the same way. See \cite{Giusti_et_al,kahle} for more details.

Figure~\ref{fig2}a shows the ordering matrix $\widehat{M}$ of a small matrix $M$. The complete sequence of clique complexes is depicted in panel c, and the corresponding Betti curves in panel b. Note that $\beta_2 = \beta_3 = 0$ for all values of the edge density $t$. On the other hand, $\beta_1(t) = 1$ for some intermediate values of $t$ where a $1$-dimensional hole arises in the clique complex $X(G_t)$. Note that $\beta_0(t)$ counts the number of connected components for each clique complex, and is thus monotonically decreasing for any matrix.

Our main results characterize Betti curves of symmetric rank 1 matrices.
We say that a vector $\mathbf{x} = (x_1, \dots, x_n)$  \textit{generates} a rank 1 matrix $M$ if $M = \mathbf{x^T}\mathbf{x}.$ Perhaps surprisingly, there are significant differences in the ordering matrices $\widehat{M}$ depending on the sign pattern of the $x_i$. The simplest case is when $M$ is generated by a vector $\mathbf{x}$ with all positive or all negative entries. When $\mathbf{x}$ has a mix of positive and negative entries, then $M$ has a block structure with two diagonal blocks of positive entries and two off-diagonal blocks of negative entries. This produces qualitatively distinct $\widehat{M}$. Even more surprising, taking $M = -\mathbf{x^T}\mathbf{x}$ qualitatively changes the structure of the ordering in a way that Betti curves can detect. This corresponds to building clique complexes from $M$ by adding edges in reverse order, from largest to smallest. Here we consider all four cases: $M = \mathbf{x^T}\mathbf{x}$ and $M = -\mathbf{x^T}\mathbf{x}$ for $\mathbf{x}$ a vector whose entries are either (i) all the same sign or (ii) have a mix of positive and negative signs.

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 4.25in]{BettiCurveResults.png}
    \caption{\textbf{Betti curves for different classes of symmetric rank 1 matrices.} For each of the six figures, we generate Betti curves $\beta_0, \beta_1, \beta_2,$ and $\beta_3$ of twenty five randomly generated $80 \times 80$ symmetric matrices. (a) The Betti curve of a positive rank one matrix $M = \mathbf{x}^T\mathbf{x}$ where all entries of row vector $\mathbf{x}$ are chosen uniformly from $[0,1]$. (d) The Betti curve of a negative rank one matrix, where the only difference from the positive case is that $M =-\textbf{x}^T\textbf{x}$. (b,e) follow similar suit, except the entries of $\textbf{x}$ are chosen uniformly from the interval $[-1,1]$  and thus has mixed signs. (c,f) are random symmetric matrices constructed by making a matrix $X$ with i.i.d. entries in $[0,1]$, and setting $M_{\pm} = \pm (X + X^T)/2$. For random matrices, there is no difference in the Betti curves between the positive and negative versions.}
    \label{fig3}
    \vspace{-.15in}
\end{figure}

When $\mathbf{x}$ has all entries the same sign, we say the matrix $M = \mathbf{x^T}\mathbf{x}$ is \textit{positive rank one} and $M = -\mathbf{x^T}\mathbf{x}$ is \textit{negative rank one}.
Observe in Figure~\ref{fig3}a that for a positive rank one matrix, $\beta_k(t)$ for $k > 0$ is identically zero. In Figure~\ref{fig3}d we see that the same is true for negative rank one matrices, though the $\beta_0(t)$ curve decreases more slowly than in the positive rank one case. The vanishing of the higher Betti curves in both cases can be proven.

\begin{theorem}\label{theorem_positive_rank_one}
Let $M$ be a positive rank one matrix or a negative rank one matrix. The $k$-th Betti curve $\beta_k(t)$ is identically zero for all $k >0$ and $t \in [0,1]$.
\end{theorem}

\begin{proof}
Without loss of generality, let $\textbf{x} = (x_1, x_2, \dots , x_n)$ be a vector such that $0 \leq x_1 \leq x_2 \leq \dots \leq x_n.$ Let $M = \textbf{x}^T \textbf{x}$ be a rank one matrix generated by $\textbf{x}$ and for a given $t \in [0,1]$, consider the graph $G_t$. It is clear that, because $x_1$ is minimal, if $(i,j)$ is an edge of $G_t$, then $(1,i)$ and $(1,j)$ are also edges of $G_t$, since $x_1x_i \leq x_ix_j$ and  $x_1x_j \leq x_ix_j$.
It follows that $X(G_t)$ is the union of a cone and a collection of isolated vertices, and hence is homotopy equivalent to a set of points. It follows that $\beta_k(t)=0$ for all $k>0$ and all $t \in [0,1]$, while $\beta_0(t)$ is the number of isolated vertices of $G_t$ plus one (for the cone component). See Figure~\ref{fig_pos_rank_one} for an example. The same argument works for $-M$, where the cone vertex corresponds to $x_n$ instead of $x_1$. $\square$
\end{proof}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{Positive_rank1_example.png}
    \caption{Illustration for the proof of the positive rank one case. Note that vertex 1 has minimal value in the generating vector ${\bf x}$, and hence all other nodes are connected to 1 before any other node.}
    \label{fig_pos_rank_one}
        \vspace{-.2in}
\end{figure}

The axis simplices we described in the Introduction are all positive rank one. We thus have the following corollary.

\begin{corollary}
The k-th Betti curve of a distance matrix induced by an axis simplex or by a simplex with an orthogonal corner is identically zero for $k > 0$.
\end{corollary}

In the cases where $\textbf{x}$ has mixed signs, the situation is a bit more complicated. Figure~\ref{fig3}b shows that for {\it positive mixed sign} matrices $M = {\bf x}^T {\bf x}$, the first Betti curve $\beta_1(t)$ ramps up linearly to a high value and then quickly crashes down to 1. In contrast, Figure~\ref{fig3}e shows that the {\it negative mixed sign} matrices $M = - {\bf x}^T {\bf x}$ have vanishing $\beta_1(t)$ and have a similar profile to the positive and negative rank one Betti curves in Figure~\ref{fig3}a,d. Note, however, that $\beta_0$ decreases more quickly than the negative rank one case, but more slowly than positive rank one.

Figure~\ref{fig:fig5} provides some intuition for the mixed sign cases. The matrix $M = {\bf x}^T {\bf x}$ splits into blocks, with the green edges added first. Because these edges belong to a bipartite graph, the number of 1-cycles increases until $G_t$ is a complete bipartite graph with all the green edges (see Figure~\ref{fig:fig5}b). This is what allows $\beta_1(t)$ to increase approximately linearly. Once the edges corresponding to the diagonal blocks are added, we obtain coning behavior on each side similar to what we saw in the positive rank one case. The 1-cycles created by the bipartite graph quickly disappear and the higher-order Betti curves all vanish. On the other hand, for the negative matrix $-M = -{\bf x}^T {\bf x}$, the green edges will be added last. This changes the $\beta_1(t)$ behavior dramatically, as all 1-cycles in the bipartite graph are automatically filled in with cliques because both sides of the bipartite graph are complete graphs by the time the first green edge is added.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{fig5.png}
    \caption{(a) A mixed sign rank one matrix $M = {\bf x}^T {\bf x}$ with negative entries in green. (b) The graph $G_t$ induced by $M$ at $t = 1$, with edge colors corresponding to matrix colorings. The negative entries (green) get added first and form a complete bipartite graph. In the negative version, $- {\bf x}^T {\bf x}$, the green edges are added last. This leads to qualitative differences in the Betti curves.}
    \label{fig:fig5}
    \vspace{-.2in}
\end{figure}

\begin{lemma} \label{lemma:bipartite}
Let $M$ be an $n \times n$ positive mixed sign rank one matrix, generated by a vector $\mathbf{x}$ with precisely $\ell$ negative entries. Then $G_t(M)$ is bipartite for $t \leq t_0 = \ell(n-\ell)/\binom{n}{2}$ and is the complete bipartite graph, $K_{\ell,n-\ell}$, at $t = t_0$.
\end{lemma}

The next two theorems characterize the Betti curves for the positive and negative mixed sign cases. We assume $\mathbf{x} = (x_1, x_2, \dots , x_\ell, x_{\ell+1}, \dots , x_n),$ where $x_1 \leq x_2 \leq \dots \leq x_{\ell} < 0 \leq x_{\ell+1} \leq \cdots \leq x_n$.

\begin{theorem}\label{thm:positive-mixed-signs}
Let $M = \mathbf{x}^T\mathbf{x}$. Then $\beta_1(t) \leq (\ell-1)(n-\ell-1),$ with equality when $G_t$ is the complete bipartite graph $K_{\ell, n-\ell}.$ The higher Betti curves all vanish: $\beta_k(t) = 0$ for $k > 1$ and all $t \in [0,1]$.
\end{theorem}

\begin{theorem}\label{thm:negative-mixed-sign}
Let $M = -\mathbf{x}^T\mathbf{x}$. Then $\beta_k(t) = 0$ for $k > 0$ and all $t \in [0,1]$.
\end{theorem}

To prove these theorems, we need a bit more algebraic topology. Let $A$ and $B$ be simplicial complexes with disjoint vertex sets. Then \emph{the join} of $A$ and $B$, denoted $A\ast B$, is defined as
$$A \ast B = \{ \sigma_A \cup \sigma_B \mid \sigma_A \in A \text{ and }  \sigma_B \in B\}.$$

The homology of the join of two simplicial complexes, was computed by
J. W. Milnor \cite[Lemma 2.1]{Milnor_lemma}. We give the simpler version with field coefficients:
$$\widetilde{H}_{r+1}(A\ast B) = \bigoplus_{i+j=r} \widetilde{H}_i(A) \otimes \widetilde{H}_j(B),$$
where $\widetilde{H}$ denotes the reduced homology groups.\footnote{Note that these are the same as the usual homology groups for $i>0$.} Recall that a simplicial complex $A$ is called {\it acyclic} if $\beta_i(A) = \mathrm{rank}(H_i(A)) = 0$ for all $i>0$.

\begin{corollary}\label{cor:acyclic-join}
If $A$ and $B$ are acyclic, then $H_i(A\ast B)=0$ for all $i>1$. Also $\beta_1(A \ast B) = \rk H_1(A\ast B) = \rk(\widetilde{H}_0(A) \ast \widetilde{H}_0(B)) = (\beta_0(A)-1)(\beta_0(B)-1)$.
\end{corollary}

\begin{example}
The complete bipartite graph, $K_{\ell, m}$, is the join of two zero dimensional complexes, $S_\ell$ and $S_m$. We see that $\beta_1(K_{\ell, m})=(\ell-1)(m-1).$ This immediately gives us the upper bound on $\beta_1(t)$ from Theorem~\ref{thm:positive-mixed-signs}.
\end{example}

\begin{example}
Let $G$ be a graph such that $V(G) = B \cup R$. Let $G_B$ be the subgraph induced by $B$ and $G_R$ the subgraph induced by $R.$ Assume that every vertex in $B$ is connected by an edge to every vertex in $R$.
Then $X(G) = X(G_B) \ast X(G_R).$
\end{example}

We are now ready to prove Theorems~\ref{thm:positive-mixed-signs} and~\ref{thm:negative-mixed-sign}. Recall that in both theorems, $M$ is generated by a vector ${\bf x}$ with positive and negative entries. We will use the notation $B = \{i \mid x_i < 0\}$ and $R = \{i \mid x_i \geq 0\}$, and refer to these as the ``negative'' and ``positive'' vertices of the graphs $G_t(M)$. Note that $|B| = \ell$ and $|R| = n - \ell$. In Figure~\ref{fig:fig5}b, $B = \{1,2,3\}$ and $R = \{4,5,6,7\}$. When $M = {\bf x}^T{\bf x}$, as in Theorem~\ref{thm:positive-mixed-signs}, the ``crossing'' edges between $B$ and $R$ are added first. (These are the green edges in Figure~\ref{fig:fig5}b.) When $M = -{\bf x}^T{\bf x}$, as in Theorem~\ref{thm:negative-mixed-sign}, the edges within the $B$ and $R$ components are added first, and the crossing edges are added last.

\begin{proof}[of Theorem~\ref{thm:positive-mixed-signs}]
We've already explained where the Betti 1 bound $\beta_1(t) \leq (\ell-1)(n-\ell-1)$ comes from (see Example 1). It remains to show that the higher Betti curves all vanish for $t \in [0,1]$. Let $t_0$ be the value at which $G_{t_0}(M)$ is the complete bipartite graph with parts $B$ and $R$. For $t \leq t_0$, when edges corresponding to negative entries of $M$ are being added, $G_t(M)$ is always a bipartite graph (see Lemma~\ref{lemma:bipartite}). It follows that $G_t(M)$ has no cliques of size greater than two, and so $X(G_t(M)) = G_t(M)$ is a one-dimensional simplicial complex. Thus, $\beta_k(t) = 0$ for all $k>1$.

For $t > t_0$, denote by $M_B$ and $M_R$ the principal submatrices induced by the indices in $B$ and $R$, respectively. Clearly, $M_B$ and $M_R$ are positive rank one matrices, and hence by Theorem 1 the clique complexes $X(G_t(M_B))$ and $X(G_t(M_R))$ are acyclic for all $t$. Moreover, for $t > t_0$ we have that $X(G_t(M)) = X(G_t(M_B)) \ast X(G_t(M_R))$, where $\ast$ is the join (see Example 2). It follows from Corollary~\ref{cor:acyclic-join} that $\beta_k(t) = 0$ for all $k>1$. $\square$
\end{proof}

The proof of Theorem~\ref{thm:negative-mixed-sign} is a bit more subtle. In this case, the edges within each part $B$ and $R$ are added first, and the ``crossing'' edges come last. The conclusion is also different, as we prove that the $X(G_t(M))$ are all acyclic.

\begin{proof}[of Theorem~\ref{thm:negative-mixed-sign}]
Recall that $M = -{\bf x}^T{\bf x}$, and let $t_0$ be the value at which all edges for the complete graphs  $K_B$ and $K_R$ have been added, so that $G_{t_0}(M) = K_B \cup K_R$ (note that this is a disjoint union). Let $M_B$ and $M_R$ denote the principal submatrices induced by the indices in $B$ and $R$, respectively. Note that for $t \leq t_0$, $X(G_t(M)) = X(G_t(M_B)) \cup X(G_t(M_R))$. Since $M_B$ and $M_R$ are both positive rank one matrices, and the clique complexes $X(G_t(M_B))$ and $X(G_t(M_R))$ are disjoint. It follows that $X(G_t(M))$ is acyclic for $t \leq t_0$.

To see that $X(G_t(M))$ is acyclic for $t > t_0,$ we will show that $X(G_t(M))$ is the union of two contractible simplicial complexes whose intersection is also contractible.
Let $M^*$ be the positive rank one matrix generated by the vector of absolute values, $y = (|x_1|, \ldots, |x_n|)$, so that $M^* = y^T y$. Now observe that for $t > t_0$, the added edges in $G_t(M)$ correspond to entries of the matrix where $M_{ij} = - x_i x_j = |x_i| |x_j| = M^*_{ij}$. This means that the ``crossing'' edges have the same relative ordering as those of the analogous graph filtration $F_t(M^*)$. Let $s$ be the value at which the last edge added to $F_s(M^*)$ is the same as the last edge added to $G_t(M)$. (In general, $s \neq t$ and $F_s(M^*) \neq G_t(M)$.)

Since $M^*$ is positive rank one, the clique complex $X(F_s(M^*))$ is the union of a cone and some isolated vertices (see the proof of Theorem~\ref{theorem_positive_rank_one}). Let $\widetilde{F}_s$ be the largest connected component of $F_s(M^*)$ -- that is, the graph obtained by removing isolated vertices from $F_s(M^*)$. The clique complex $X(\widetilde{F}_s)$ is a cone, and hence contractible (not merely acyclic). Now define two simplicial complexes, $\Delta_1$ and $\Delta_2$, as follows:
$$\Delta_1 = X(\widetilde{F}_s \cup K_B) \;\; \text{and} \;\;
 \Delta_2 = X(\widetilde{F}_s \cup K_R).$$
It is easy to see that both $\Delta_1$ and $\Delta_2$ are contractible. Moreover, $X(G_t) = \Delta_1 \cup \Delta_2$. Since the intersection $\Delta_1 \cap \Delta_2 = X(\widetilde{F}_s)$ is contractible, using the Mayer-Vietoris sequence we can conclude that $X(G_t)$ is contractible for all  $t > t_0.$ $\square$
\end{proof}

\section{Application to calcium imaging data in zebrafish larvae}

Calcium imaging data for neural activity in the optic tectum of zebrafish larvae was collected by the Sumbre lab at École Normale Superieure. The individual time series of calcium activation for each neuron were preprocessed to produce neural activity rasters. This in turn was used to compute cross correlograms (CCGs) that capture the time-lagged pairwise correlations between neurons. We then integrated the CCGs over a time window of $\pm \tau_{\max}$ for $\tau_{\max} = 1$ second, to obtain pairwise correlation values:
$$C_{ij} = \frac{1}{2{\tau}_{\max}r_i r_j}\dfrac{1}{T} \int \limits_{-\tau_{\max}}^{\tau_{\max}} \int \limits_{-T}^{T} f_i(t)f_j(t + \tau) dt d\tau.$$
Here $T$ is the total time of the recording, $f_i(t)$ is the raster data for neuron $i$ taking values of $0$ or $1$ at each (discrete) time point $t$. The ``firing rate'' $r_i$ is the proportion of $1$s for neuron $i$ over the entire recording.

\begin{figure}[!ht]
    \centering
    \includegraphics[width = \textwidth]{zebrafish_examples.png}
    \caption{We plot Betti curves and singular values of two submatrices induced by preidentified cell assemblies 83 and 122 (data courtesy of the Sumbre lab at Ecole Normale Superieure). The Betti curves are consistent with a rank one matrix. In contrast, Betti curves for random assembly whose pairwise correlations come from the same CCG matrix of the same order. Below each Betti curve, we plot the normalized singular values and visually see that they are full rank and appear close to the singular values of a random assembly of the same size.}
    \label{fig6}
\end{figure}

The Sumbre lab independently identified cell assemblies: subsets of neurons that were co-active spontaneously and in response to visual stimuli, and which are believed to underlie functional units of neural computation.
Figure~\ref{fig6} shows the Betti curves of principal submatrices of $C$ for two pre-identified cell assemblies. The Betti curves for the assemblies display the same signature we saw for rank 1 matrices, and are thus consistent with an underlying low rank. In contrast, the singular values of these same correlation matrices (centered to have zero mean) appear random and full rank, so the structure detected by the Betti curves could not have been seen with singular values. To check that this low rank Betti curve signature is not an artifact of the way the correlation matrix $C$ was computed, we compared the Betti curves to a randomly chosen assembly of the same size that includes the highest firing neuron of the original assembly. (We included the highest firing neuron because this could potentially act as a coning vertex, yielding the rank 1 Betti signature for trivial reasons.) In summary, we found that the real assemblies exhibit rank 1 Betti curves, while those of randomly-selected assemblies do not.

\section{Conclusion}

In this paper, we proved three theorems that characterized the Betti curves of rank 1 symmetric matrices. We also showed these rank 1 signatures are present in some cell assembly correlation matrices for zebrafish.
A limitation of these theorems, however, is that the converse is not in general true. Identically zero Betti curves do not imply rank 1, though experiments suggest they are indicative of low rank. So while we can use these results to rule out an underlying rank 1 structure, as in the random assemblies, we can not conclude from Betti curves alone that a matrix has underlying rank 1.

\subsubsection{Code}

All code used for the construction of the figures except, for Figure~\ref{fig6}, is available on Github. Code and data to produce Figure~\ref{fig6} is available upon request.
\url{https://github.com/joshp112358/rank-one-betti-curves}.


On the number of cyclic subgroups of a finite abelian group

Abstract: We prove by using simple number-theoretic arguments formulae concerning the number of elements of a fixed order and the number of cyclic subgroups of a direct product of several finite cyclic groups. We point out that certain multiplicative properties of related counting functions for finite Abelian groups are immediate consequences of these formulae.

\section{Introduction}

Consider an arbitrary finite Abelian group $G$ of order $\# G>1$.
Let $c_{\delta}(G)$ and $c(G)$ denote the number of cyclic subgroups
of order $\delta$ ($\delta \mid \# G$) and the number of all cyclic
subgroups of $G$, respectively.

Let $\# G= p_1^{a_1}\cdots p_r^{a_r}$ be the prime power
factorization of $\# G$ and let $G\simeq G_1\times \cdots \times G_r$ be
the primary decomposition of the group $G$, where $\# G_i =
p_i^{a_i}$ ($1\le i\le r$). Furthermore, let $L(G)$ and $L(G_i)$
denote the subgroup lattices of $G$ and $G_i$ ($1\le i\le r$),
respectively. It is known the lattice isomorphism $L(G)\simeq L(G_1)
\times \cdots \times L(G_r)$. See R.~Schmidt \cite{Sch1994},
M.~Suzuki \cite{Suz1951}. It follows that
\begin{equation} \label{c_d_multipl}
c_{\delta}(G)=c_{\delta_1}(G_1)\cdots c_{\delta_r}(G_r),
\end{equation}
where $\delta=\delta_1\cdots \delta_r$ and $\delta_i=p_i^{b_i}$ with
some exponents $b_i\le a_i$ ($1\le i\le r$). Also,
\begin{equation} \label{c_multipl}
c(G)=c(G_1)\cdots c(G_r).
\end{equation}

Consider now a $p$-group $G_{(p)}$ of type
$(\lambda_1,\ldots,\lambda_k)$ with $1\le \lambda_1\le \ldots \le
\lambda_k$. It is known that the number of cyclic subgroups of order
$p^{\nu}$ of $G_{(p)}$ is
\begin{equation} \label{c_p_group}
c_{p^{\nu}}(G_{(p)})= \frac{p^{k-j_{\nu}}-1}{p-1}
p^{\lambda_0+\lambda_1+\ldots+\lambda_{j_{\nu}}+ (k-j_{\nu}-1)(\nu
-1)},
\end{equation}
where $\lambda_{j_{\nu}} <\nu \le \lambda_{j_{\nu}+1}$ and
$\lambda_0=0$. See G.~A.~Miller \cite{Mil1939}, Y.~Yeh
\cite{Yeh1948}.

For every finite Abelian group $G$ the value of $c_{\delta}(G)$ is
therefore completely determined by formulae \eqref{c_d_multipl} and
\eqref{c_p_group}. Obviously, $c(G)=\sum_{\delta \mid \# G}
c_{\delta}(G)$, leading to a formula for $c(G)$.

Recently, formula \eqref{c_p_group} was recovered by
M.~T\u{a}rn\u{a}uceanu \cite[Th.\ 4.3]{Tar2010}, in a slightly
different form, by a method based on properties of certain matrices
attached to the invariant factor decomposition of an Abelian group
$G$, i.e., $G\simeq C_{i_1}\times \cdots \times C_{i_s}$, where
$i_1\cdots i_s=\# G$, $i_1\mid i_2\mid \ldots \mid i_s$, $C_k$
denoting the cyclic group of order $k$.

For arbitrary positive integers $r,n_1,\ldots, n_r$ consider in what
follows the direct product $C_{n_1}\times \cdots \times C_{n_r}$ and
denote by $c(n_1,\ldots,n_r)$ the number of its cyclic subgroups.
Let $\phi$ denote Euler's function.

\begin{theorem} \label{Th_1} For any $n_1,\ldots, n_r\ge 1$, $c(n_1,\ldots,n_r)$
is given by the formula
\begin{equation} \label{total_number_cyclic_subgroups}
c(n_1,\ldots,n_r)= \sum_{d_1\mid n_1,\ldots,d_r\mid n_r}
\frac{\phi(d_1) \cdots \phi(d_r)}{\phi(\operatorname{lcm}(d_1,\ldots,d_r))}.
\end{equation}
\end{theorem}

Formula \eqref{total_number_cyclic_subgroups} was derived in
\cite{Tot2011} using the orbit counting lemma (Burnside's lemma).
Consequently, the number of all cyclic subgroups of a finite Abelian
group can be given by formula \eqref{total_number_cyclic_subgroups}
in a more compact form. Namely, consider the invariant factor
decomposition of $G$ given above and apply
\eqref{total_number_cyclic_subgroups} by selecting $r=s$ and
$n_1=i_1,\ldots,n_s=i_s$. Note that
\eqref{total_number_cyclic_subgroups} can be used also starting with
the primary decomposition of $G$.

In the case $r=2$, by using the identity $\phi(d_1)\phi(d_2)=
\phi(\gcd(d_1,d_2)) \phi(\operatorname{lcm}(d_1,d_2))$ ($d_1,d_2\ge 1$) we deduce
from \eqref{total_number_cyclic_subgroups} the next result:

\begin{cor} For every $n_1,n_2\ge 1$ the number of cyclic
subgroups of $C_{n_1}\times C_{n_2}$ is
\begin{equation} \label{total_number_cyclic_subgr_2_1}
c(n_1,n_2)=  \sum_{d_1 \mid n_1, d_2\mid n_2} \phi(\gcd(d_1,d_2)).
\end{equation}
\end{cor}

It is the purpose of the present paper to give another direct proof
of Theorem \ref{Th_1}, using simple number-theoretic arguments. In
fact, we prove the following formulae concerning the number
$o_{\delta}(n_1,\ldots,n_r)$ of elements of order $\delta$ in
$C_{n_1}\times \cdots \times C_{n_r}$. Let
$n:=\operatorname{lcm}(n_1,\ldots,n_r)$. Obviously, the order of every element of
the direct product is a divisor of $n$. Let $\mu$ be the M\"obius
function.

\begin{theorem} \label{Th_2} For every $n_1,\ldots, n_r\ge 1$ and every $\delta \mid n$,
\begin{equation} \label{number_elements_order_d_first}
o_{\delta}(n_1,\ldots,n_r)= \sum_{e\mid \delta}  \gcd(e,n_1) \cdots
\gcd(e,n_r) \mu(\delta/e)
\end{equation}
\begin{equation} \label{number_elements_order_d_second}
= \sum_{\substack{d_1\mid n_1,\ldots, d_r\mid n_r
\\ \operatorname{lcm}(d_1,\ldots,d_r)=\delta}} \phi(d_1) \cdots \phi(d_r).
\end{equation}
\end{theorem}

Let $c_{\delta}(n_1,\ldots,n_r)$ denote the number of cyclic
subgroups of order $\delta$ ($\delta \mid n$) of the group
$C_{n_1}\times \cdots \times C_{n_r}$. Since a cyclic subgroup of
order $\delta$ has $\phi(\delta)$ generators,
\begin{equation} \label{number_cyclic_subgroups_order_d}
c_{\delta}(n_1,\ldots,n_r)=\frac{o_{\delta}(n_1,\ldots,n_r)}{\phi(\delta)}.
\end{equation}

Now \eqref{total_number_cyclic_subgroups} follows immediately from
\eqref{number_elements_order_d_second} and
\eqref{number_cyclic_subgroups_order_d} by
$c(n_1,\ldots,n_r)=\sum_{\delta\mid n} c_{\delta}(n_1,\ldots,n_r)$.

\begin{cor} For every prime $p$ and every $a_1,\ldots,a_r\ge 1$, $1\le
\nu \le \max(a_1,\ldots,a_r)$,
\begin{equation} \label{number_p}
c_{p^{\nu}}(p^{a_1},\ldots,p^{a_r})= \frac1{p^{\nu-1}(p-1)} \left(
p^{\min(\nu,a_1)+\ldots+\min(\nu,a_r)}-
p^{\min(\nu-1,a_1)+\ldots+\min(\nu-1,a_r)}\right).
\end{equation}
\end{cor}

Here \eqref{number_p} follows at once by
\eqref{number_elements_order_d_first}. In case of a $p$-group of
type $(\lambda_1,\ldots,\lambda_k)$ and choosing $r=k$,
$a_1=\lambda_1,\ldots, a_k=\lambda_k$, \eqref{number_p} reduces to
\eqref{c_p_group}.

The proof of Theorem \ref{Th_2} is contained in Section
\ref{section_2}. Additional remarks are given in Section
\ref{section_3}. We point out, among others, that the multiplicative
properties \eqref{c_d_multipl} and \eqref{c_multipl} are direct
consequences of the formulae included in Theorems \ref{Th_1} and
\ref{Th_2}.

\section{Proof of Theorem \ref{Th_2}} \label{section_2}

Let $o(x)$ denote the order of a group element $x$. Let
$G=C_{n_1}\times \cdots \times C_{n_r}$ in short. Let $x_i\in
C_{n_i}$ such that $o(x_i)=n_i$ ($1\le i\le r$). Then $G= \{
x=(x_1^{i_1},\ldots,x_r^{i_r}): 1\le i_1\le n_1, ..., 1\le i_r\le
n_r\}$.

Using elementary properties of the order of elements in a group and
of the gcd and lcm of integers, respectively we deduce that for
every $x=(x_1^{i_1},\ldots,x_r^{i_r})\in G$,
\begin{equation*}
o(x)= \operatorname{lcm}(o(x_1^{i_1}),\ldots,o(x_r^{i_r}))= \operatorname{lcm}
\left(\frac{o(x_1)}{\gcd(o(x_1),i_1)},\ldots,
\frac{o(x_r)}{\gcd(o(x_r),i_r)} \right)
\end{equation*}
\begin{equation*}
= \operatorname{lcm} \left(\frac{n_1}{\gcd(n_1,i_1)},\ldots,
\frac{n_r}{\gcd(n_r,i_r)} \right) = \operatorname{lcm} \left(\frac{n}{\gcd(n, i_1
n/n_1)},\ldots, \frac{n}{\gcd(n,i_r n/n_r)} \right)
\end{equation*}
\begin{equation*}
= \frac{n}{\gcd (i_1 n/n_1,\ldots, i_r n/n_r,n)}.
\end{equation*}

Assume that $o(x)=\delta$, where $\delta \mid n$ is fixed. Then
$\gcd (i_1 n/n_1,\ldots, i_r n/n_r,n)=n/\delta$.

Write $i_1 n/n_1=j_1 n/\delta, \ldots, i_r n/n_r=j_r n/\delta$. Then
$\gcd(j_1,\ldots,j_r,\delta)=1$ and $j_1=\delta i_1/n_1$, ...,
$j_r=\delta i_r/n_r$ are integers, that is $\delta i_1\equiv 0$ (mod
$n_1$), ..., $\delta i_r\equiv 0$ (mod $n_r$). We obtain, as solutions
of these linear congruences, that
$i_1=k_1n_1/\gcd(\delta,n_1)$ with $1\le k_1\le \gcd(\delta,n_1)$,
..., $i_r=k_rn_r/\gcd(\delta,n_r)$ with $1\le k_r\le
\gcd(\delta,n_r)$.

Therefore, the number of elements of order $\delta$ in $G$ is the
number of ordered $r$-tuples $(k_1,\ldots,k_r)$ satisfying the
conditions

(i)  $1\le k_1\le \gcd(\delta,n_1),\ldots, 1\le k_r\le
\gcd(\delta,n_r)$ and

(ii) $\gcd \left(k_1\frac{\delta}{\gcd(\delta,n_1)},\ldots,
k_r\frac{\delta}{\gcd(\delta,n_r)}, \delta \right)=1$.

Using the familiar formula $\sum_{d\mid n} \mu(d)=1$ for $n=1$ and
$0$ otherwise, this can be written as
\begin{equation*}
o_{\delta}(n_1,\ldots,n_r) = \sum_{e\mid \delta} \mu(e) \sum_{\substack{k_1\le \gcd(\delta,n_1)\\
e\mid k_1\frac{\delta}{\gcd(\delta,n_1)}}} \cdots
\sum_{\substack{k_r\le \gcd(\delta,n_r)\\ e\mid
k_r\frac{\delta}{\gcd(\delta,n_r)}}} 1.
\end{equation*}

Here the linear congruence $k_1\frac{\delta}{\gcd(\delta,n_1)}
\equiv 0$ (mod $e$) has $\gcd(e,\frac{\delta}{\gcd(\delta,n_1)})$
solutions in $k_1$ (mod $e$), and (mod $\gcd(\delta,n_1)$) it has
exactly
\begin{equation*}
\frac{\gcd(\delta,n_1)}{e}
\gcd\left(e,\frac{\delta}{\gcd(\delta,n_1)}\right)=
\gcd\left(\gcd(\delta,n_1),\frac{\delta}{e}\right) =
\gcd\left(\frac{\delta}{e},n_1\right)
\end{equation*}
solutions (similar for the indexes $2,...,r$). Hence,
\begin{equation*}
o_{\delta}(n_1,\ldots,n_r) = \sum_{e\mid \delta} \mu(e)
\gcd(\delta/e,n_1) \cdots \gcd(\delta/e,n_r)
\end{equation*}
\begin{equation*}
= \sum_{e\mid \delta} \mu(\delta/e) \gcd(e,n_1) \cdots \gcd(e,n_r),
\end{equation*}
which is formula \eqref{number_elements_order_d_first}.

Now using the identity $\sum_{d\mid n} \phi(d)=n$,
\begin{equation*}
o_{\delta}(n_1,\ldots,n_r) = \sum_{e\mid \delta} \mu(\delta/e)
\sum_{d_1\mid \gcd(e,n_1)} \phi(d_1) \cdots \sum_{d_r\mid
\gcd(e,n_r)} \phi(d_r)
\end{equation*}
\begin{equation*}
= \sum_{d_1\mid n_1,\ldots, d_r\mid n_r} \phi(d_1)\cdots \phi(d_r)
\sum_{ab \operatorname{lcm}(d_1,\ldots,d_r)=\delta} \mu(a),
\end{equation*}
where the inner sum is $0$ unless $\operatorname{lcm}(d_1,\ldots,d_r)=\delta$, and
in this case it is $1$. This gives
\eqref{number_elements_order_d_second}. The proof of Theorem
\ref{Th_2} is complete.

\section{Remarks}  \label{section_3}

The function $c(n_1,\ldots,n_r)$ is representing a multiplicative
function of $r$ variables, i.e.,
\begin{equation} \label{c_multiplic}
c(n_1m_1,\ldots,n_rm_r)= c(n_1,\ldots,n_r) c(m_1,\ldots,m_r)
\end{equation}
holds for any $n_1,\ldots,n_r,m_1,\ldots,m_r\ge 1$ such that
$\gcd(n_1\cdots n_r,m_1\cdots m_r)=1$. Therefore,
\begin{equation*}
c(n_1,\ldots,n_r)= \prod_p c(p^{e_p(n_1)}, \ldots,p^{e_p(n_r)}),
\end{equation*}
where $n_i=\prod_p p^{e_p(n_i)}$ is the prime power factorization of
$n_i$, the product being over the primes $p$ and all but a finite
number of the exponents $e_p(n_i)$ are zero ($1\le i\le r$).

The property \eqref{c_multiplic} follows by the next simple
number-theoretic argument: According to
\eqref{total_number_cyclic_subgroups}, $c(n_1,\ldots,n_r)$ is the
$r$ variables convolution of the function $(n_1,\ldots,n_r) \mapsto
\frac{\phi(n_1) \cdots \phi(n_r)}{\phi(\operatorname{lcm}(n_1,\ldots,n_r))}$ with
the constant $1$ function, both multiplicative. Since convolution
preserves the multiplicativity we deduce that $c(n_1,\ldots,n_r)$ is
multiplicative, too. See \cite[Section 2]{Tot2011} for details.

Also, let $n_1,\ldots,n_r,m_1,\ldots,m_r\ge 1$ such that
$\gcd(n_1\cdots n_r,m_1\cdots m_r)=1$. Let
$n:=\operatorname{lcm}(n_1,\ldots,n_r)$, $m:=\operatorname{lcm}(m_1,\ldots,m_r)$. For $\delta
\mid nm$ write $\delta=ab$ with $a\mid n$, $b\mid m$ (which are
unique). Then,
\begin{equation} \label{c_delta_multipl}
c_{\delta}(n_1m_1,\ldots,n_rm_r)= c_a(n_1,\ldots,n_r)
c_b(m_1,\ldots,m_r),
\end{equation}
which follows from \eqref{number_elements_order_d_first} by a short
computation.

Now, \eqref{c_multiplic} and \eqref{c_delta_multipl} show the
validity of \eqref{c_multipl} and \eqref{c_d_multipl} quoted in the
Introduction.

Let $A(n_1,\ldots,n_r)$ denote the arithmetic mean of the orders of
elements in $C_{n_1}\times \cdots \times C_{n_r}$. From
\eqref{number_elements_order_d_second} we deduce

\begin{cor} For every $n_1,\ldots,n_r\ge 1$,
\begin{equation*}
A(n_1,\ldots,n_r)= \frac1{n_1\cdots n_r} \sum_{d_1\mid n_1,\ldots,
d_r\mid n_r} \phi(d_1)\cdots \phi(d_r) \operatorname{lcm}(d_1,\ldots,d_r).
\end{equation*}
\end{cor}

The function $(n_1,\ldots,n_r) \mapsto A(n_1,\ldots,n_r)$ is also
multiplicative. See \cite{zGat2004,LucShp2003} for further
properties of the average of orders of elements in finite Abelian
groups.

Consider the special case $n_1=\ldots =n_r=n$. According to
\eqref{number_elements_order_d_first},
\begin{equation} \label{Jordan}
o_{\delta}(n,\ldots,n) = \sum_{e\mid \delta} e^r \mu(\delta/r) =
\phi_r(\delta)
\end{equation}
is the Jordan function of order $r$. Note that
$\phi_1(\delta)=\phi(\delta)$ is Euler's function.

This means that the number of elements of order $\delta$ ($\delta
\mid n$) in the direct product $C_n\times \cdots \times C_n$, with
$r$ factors, is $\phi_r(\delta)= \delta^r\prod_{p\mid \delta}
(1-1/p^r)$. This is a less known group-theoretic interpretation of
the Jordan function. See G.~A.~Miller \cite{Mil1904,Mil1905},
J.~Schulte \cite[Th.\ 7]{Sch1999}.

Hence the number of cyclic subgroups of order $\delta$ ($\delta
\mid n$) of the group $C_n\times \cdots \times C_n$, with
$r$ factors, is $\phi_r(\delta)/\phi(\delta)$.

Note that the function $k\mapsto \phi_r(k)/(k^{r-2}\phi(k))=
k\prod_{p\mid k} (1+1/p+1/p^2+\ldots+1/p^{r-1})$ ($r\ge 2$) was
considered quite recently in \cite{SolPla2011}, in connection with
Robin's inequality and the Riemann hypothesis.

As a byproduct we obtain from \eqref{number_elements_order_d_second}
and \eqref{Jordan} that for every positive integer $\delta$,
\begin{equation} \label{Sterneck}
\sum_{\operatorname{lcm}(d_1,\ldots,d_r)=\delta} \phi(d_1)\cdots \phi(d_r)
=\phi_r(\delta),
\end{equation}
where the sum is over all ordered $r$-tuples $(d_1,\ldots,d_r)$ such
that $\operatorname{lcm}(d_1,\ldots,d_r)=\delta$. The function in the left hand
side of \eqref{Sterneck} is the von Sterneck function, cf. \cite[p.\
14]{McC1986}.

A final remark: The number $s(n_1,n_2)$ of all subgroups of the
group $C_{n_1}\times C_{n_2}$ is given by
\begin{equation} \label{s_n}
s(n_1,n_2)= \sum_{d_1\mid n_1, d_2\mid n_2} \gcd(d_1,d_2) \quad
(n_1,n_2\ge 1).
\end{equation}

Compare \eqref{s_n} to \eqref{total_number_cyclic_subgr_2_1}. A
proof of \eqref{s_n} will be presented elsewhere.

Group Theory (math.GR); Number Theory (math.NT)