Coordinate Descent Face-Off: Primal or Dual?

Abstract: Randomized coordinate descent (RCD) methods are state-of-the-art algorithms for training linear predictors via minimizing regularized empirical risk. When the number of examples ($n$) is much larger than the number of features ($d$), a common strategy is to apply RCD to the dual problem. On the other hand, when the number of features is much larger than the number of examples, it makes sense to apply RCD directly to the primal problem. In this paper we provide the first joint study of these two approaches when applied to L2-regularized ERM. First, we show through a rigorous analysis that for dense data, the above intuition is precisely correct. However, we find that for sparse and structured data, primal RCD can significantly outperform dual RCD even if $d \ll n$, and vice versa, dual RCD can be much faster than primal RCD even if $n \ll d$. Moreover, we show that, surprisingly, a single sampling strategy minimizes both the (bound on the) number of iterations and the overall expected complexity of RCD. Note that the latter complexity measure also takes into account the average cost of the iterations, which depends on the structure and sparsity of the data, and on the sampling strategy employed. We confirm our theoretical predictions using extensive experiments with both synthetic and real data sets.

\section{Introduction}

In the last 5 years or so, randomized coordinate descent (RCD) methods  \cite{ShalevTewari11, Nesterov:2010RCDM,  UCDC, PCDM} have  become immensely  popular in a variety of machine learning  tasks, with supervised learning being a prime example. The main reasons behind the rise of RCD-type methods is that they can be easily  implemented, have intuitive appeal, and enjoy superior theoretical and practical behaviour when compared to classical methods such as  SGD \cite{robbins1951}, especially in high dimensions, and in situations when solutions of medium to high accuracy are needed. One of the most important success stories of RCD is in the domain of training linear predictors via regularized empirical risk minimization (ERM).

The highly popular SDCA algorithm \cite{SDCA} arises as the application of RCD \cite{UCDC} to the {\em dual problem} associated with the (primal) ERM problem\footnote{Indeed, the analysis of SDCA in \cite{SDCA} proceeds by applying the complexity result from \cite{UCDC} to the {\em dual problem},  and then arguing that the same rate applies to the primal suboptimality as well.}. In practice, SDCA is most effective  in situations where the number of examples ($n$) exceeds the number of features ($d$).  Since the dual of ERM is an $n$ dimensional problem, it makes intuitive sense to apply RCD to the dual. Indeed, RCD can be seen as a randomized decomposition strategy, reducing the $n$ dimensional problem to a sequence of (randomly generated) one-dimensional problems.

However, if the number of features exceeds the number of examples, and especially when the difference is very large, RCD methods \cite{PCDM} have been found very attractive for solving the {\em primal  problem} (i.e., the ERM problem) directly. For instance, distributed variants of RCD, such as Hydra \cite{Hydra} and its accelerated cousin Hydra$^2$ \cite{Hydra2} have been successfully applied to solving problems with billions of features.

Recently, a  variety of novel primal methods for ERM have been designed, including  SAG \cite{SAG}, SVRG \cite{SVRG}, S2GD \cite{S2GD}, proxSVRG \cite{proxSVRG}, mS2GD \cite{mS2GD}, SAGA \cite{SAGA}, MISO \cite{MISO} and S2CD~\cite{S2CD}. As SDCA, all these methods improve dramatically on SGD \cite{robbins1951} as a benchmark,  which they achieve by employing one of a number of variance-reduction strategies. However, these methods have essentially identical identical theoretical behavior to SDCA, including the property that these methods thrive in the data-laden domain (i.e., $n\gg d$). In this sense, in our comparison of primal vs dual RCD, these methods should be viewed as ``dual methods''.

\subsection{Contributions}
 In this paper we provide the first joint study of these two approaches---applying RCD to the primal vs dual problems---and we do so in the context of  L2-regularized ERM. First, we show through a rigorous theoretical analysis that for dense data, the intuition that the primal approach is better than the dual approach when $n\geq d$, and vice versa, is precisely correct. However, we show that for sparse data, this does not need to be the case:  primal RCD can significantly outperform dual RCD even if $d\ll n$, and vice versa, dual RCD can be much faster than primal RCD even if $n\ll d$. In particular, we identify that the face-off between primal and dual RCD boils down to the comparison of as single quantity associated with the data matrix and its transpose.  Moreover, we show that, surprisingly, a single sampling strategy minimizes both the (bound on the) number of iterations and the overall expected complexity of RCD. Note that the latter complexity measure takes into account also the average cost of the iterations, which depends on the structure and sparsity of the data, and on the sampling strategy employed.  We confirm our theoretical findings using extensive experiments with both synthetic and real data sets.

\section{Primal and Dual Formulations of ERM}

Let  $\mathbf{X}\in \mathbb{R}^{d\times n}$ be a data matrix, with $n$ referring to the number of examples and $d$ to the number of features. With each example $\mathbf{X}_{:j}\in \mathbb{R}^d$ we associate a loss function $\phi_j:\mathbb{R}\to \mathbb{R}$, and pick a regularization constant $\lambda>0$. The key problem of this paper is the L2-regularized ERM problem
\begin{equation}
\label{def:Primal}
\min_{w \in \mathbb{R}^d} \left[ P(w) ~{:=}~ \frac{1}{n}\sum_{j=1}^n \phi_j(\langle \mathbf{X}_{:j}, w \rangle) + \frac{\lambda}{2}\|w\|_2^2 \right],
\end{equation}
where $\langle\cdot, \cdot\rangle$ denotes the standard Euclidean inner product and $\|w\|_2~{:=}~\sqrt{\langle w,w\rangle}$. We refer to \eqref{def:Primal} as the  {\em primal problem}. We assume throughout that the functions $\{\phi_j\}$ are convex and $\beta$-smooth: \begin{equation} \label{eq:phi_smooth}
\phi_j(s) + \phi_j'(s)t  \leq \phi_j(s + t) \leq \phi_j(s) + \phi_j'(s)t + \frac{\beta}{2}t^2, \qquad \text{for all} \qquad s,t\in \mathbb{R}.
\end{equation}
The {\em dual problem} of \eqref{def:Primal}  is
\begin{equation}\label{def:Dual}
\max_{\alpha \in \mathbb{R}^n} \left[ D(\alpha) ~{:=}~ - \frac{1}{2\lambda n^2}\left\| \mathbf{X}\alpha\right\|_2^2 - \frac{1}{n}\sum_{j=1}^n \phi_j^*(-\alpha_j)  \right],
\end{equation}
where $\phi_j^*:\mathbb{R}\to \mathbb{R}$ is the convex conjugate of $\phi_j$, defined by $
\phi_j^*(s) ~{:=}~ \sup \{ st - \phi_j(t) \;:\; t \in \mathbb{R}\}.$
It is well known that \cite{SDCA, Quartz} that $P(w) \geq D(\alpha)$ for every pair $(w, \alpha) \in \mathbb{R}^d \times \mathbb{R}^n$ and $P(w^*) = D(\alpha^*)$. Moreover, the primal and dual optimal solutions, $w^*$ and $\alpha^*$, respectively, are unique, and satisfy the relations $w^* = \frac{1}{\lambda n} \mathbf{X} \alpha^*$ and $\alpha_j^* = \phi_j'(\langle \mathbf{X}_{:j}, w^* \rangle)$ for all  $j \in [n] ~{:=}~ \{1, \dots, n \}$, which also uniquely characterize them.

\section{Primal and Dual RCD}

In its general ``arbitrary sampling''  form \cite{NSync}, RCD applied to the primal problem \eqref{def:Primal} has the form \begin{equation}\label{eq:RCD-primal-text}w_i^{k+1} \leftarrow w_i^k -  \frac{1}{u'_i}\nabla_i P(w^k)\quad \text{for} \quad i\in S_k, \qquad \qquad w_i^{k+1} \leftarrow w_k^k \quad \text{for} \quad i\notin S_k,\end{equation}
where $u'_1,\dots,u'_d>0$ are parameters of the method and $\nabla_i P(w) = \frac{1}{n}\sum_{j=1}^n \phi_j'(\langle \mathbf{X}_{:j}, w\rangle ) \mathbf{X}_{ij} + \lambda w_i$ is the $i$th partial derivative of $P$ at $w$. This update is performed for a random subset of the coordinates $i\in S_k\subseteq [d]$ chosen in an i.i.d.\ fashion according to some sampling $\hat{S}_P$. The parameters $u'_i$ are computed ahead of the iterative process and need to be selected carefully in order for the method to work \cite{NSync, ESO}. Specifically, one can set $u'_i ~{:=}~ \frac{\beta}{n}u_i + \lambda$, where $u=(u_1,\dots,u_d)$ is chosen so as to satisfy the ESO (expected separable overapproximation) inequality \begin{equation}\label{eq:ESO_P}\mathbf{P} \circ \mathbf{X} \mathbf{X}^\top \preceq {\rm Diag}(p\circ u),\end{equation} where $\mathbf{P}$ is the $d\times d$ matrix with entries $\mathbf{P}_{ij} = \mathbb{P}(i\in \hat{S}, j\in \hat{S})$, $p  = {\rm Diag}(\mathbf{P})\in \mathbb{R}^d$ and $\circ$ denotes the Hadamard (element-wise) product of matrices. The method is formally described as Algorithm~\ref{alg:nsync}.

\begin{algorithm}[H]
\caption{Primal RCD: NSync \cite{NSync}} \label{alg:nsync}
\begin{algorithmic}
\STATE \textbf{Input:} initial iterate $w^0 \in \mathbb{R}^d$; sampling $\hat{S}_P$; ESO parameters $u_1, \dots, u_d > 0$
\STATE \textbf{Initialize:} $z^0 = \mathbf{X}^\top w^0$
\FOR{$k=0,1,\dots $}
\STATE Sample $S_k \subseteq [d]$ according to $\hat{S}_P$
\FOR{$i \in S_k$}
\STATE Compute $\Delta^{k}_i = -\frac{n}{\beta u_i + \lambda n}\left( \frac{1}{n}\sum_{j=1}^n \phi_j'(z_j^k) \mathbf{X}_{ij} + \lambda w_i^k \right)$
\STATE Update $w_i^{k+1} = w_i^k + \Delta_i^k$
\ENDFOR
\FOR{$i\notin S_k$}
\STATE $w_i^{k+1} = w_i^k$
\ENDFOR
\STATE Update $z^{k+1} = z^k + \sum_{i \in S_k} \Delta_i^k\mathbf{X}_{i:}^\top $
\ENDFOR
\end{algorithmic}
\end{algorithm}

When applying RCD to the dual problem \ref{def:Dual}, we can't proceed as above since the functions $\phi_j^*$ are not necessarily smooth, and hence we can't compute the partial derivatives of the dual objective. The standard approach here is to use a proximal variant of RCD \cite{PCDM}. In particular, Algorithm~\ref{alg:quartz} has been analyzed in \cite{Quartz}. Like Algorithm~\ref{alg:nsync}, Algorithm~\ref{alg:quartz} is also capable to work with an arbitrary sampling, which in this case is a random subset of $[n]$. The ESO parameters $v=(v_1,\dots,v_j)$ must in this case satisfy the ESO inequality \begin{equation}\label{eq:ESO_D}\mathbf{Q}\circ \mathbf{X}^\top \mathbf{X} \preceq {\rm Diag}(q\circ v),\end{equation} where $\mathbf{Q}$ is  an $n\times n$ matrix with entries $\mathbf{Q}_{ij} = \mathbb{P}(i\in \hat{S}_D, j\in \hat{S}_D)$ and $q = {\rm Diag}(\mathbf{Q})\in \mathbb{R}^n$.

\begin{algorithm}[H]
\caption{Dual RCD: Quartz \cite{Quartz}} \label{alg:quartz}
\begin{algorithmic}
\STATE \textbf{Input:} initial dual variables $\alpha^0\in \mathbb{R}^n$, sampling $\hat{S}_D$; ESO parameters $v_1, \dots, v_n > 0$
\STATE \textbf{Initialize:} set $w^0 = \frac{1}{\lambda n} \mathbf{X} \alpha^0 $
\FOR{$k=0, 1, \dots$}
\STATE Sample $S_k \subseteq [n]$ according to $\hat{S}_D$
\FOR{$j \in S_k$}
\STATE Compute $\Delta_j^k = \argmax_{h \in \mathbb{R}} \left\{ -\phi_j^*(-(\alpha_j^k + h)) - h\langle \mathbf{X}_{:j}, w^k \rangle - \frac{v_j h^2}{2 \lambda n} \right\}$
\STATE Update  $\alpha_j^{k+1} = \alpha_j^k + \Delta_j^k $
\ENDFOR
\FOR{$j\notin S_k$}
\STATE $\alpha_j^{k+1} = \alpha_j^k$
\ENDFOR
\STATE Update $w^{k+1} = w^k + \frac{1}{\lambda n} \sum_{j \in S_k} \Delta_j^k \mathbf{X}_{:j}$
\ENDFOR
\end{algorithmic}
\end{algorithm}

 If we assume that $|\hat{S}_P|=1$ (resp.\ $|\hat{S}_D|=1$) with probability 1 (i.e., of the samplings are ``serial''), then it is trivial to observe that \eqref{eq:ESO_P} (resp.\ \eqref{eq:ESO_D}) holds with
 \begin{equation}\label{eq:98s98hsd}u={\rm Diag}(\mathbf{X} \mathbf{X}^\top) \qquad \text{(resp. } v={\rm Diag}(\mathbf{X}^\top \mathbf{X}) \text{)}.\end{equation}
Easily computable expressions for $u$ (resp.\ $v$) for more complicated samplings can be found in \cite{ESO}.

\section{Iteration Complexity and Total Arithmetic Complexity}

In this section we give expressions for the total expected arithmetic complexity of the two algorithms.

\subsection{Number of iterations} Iteration complexity of Algorithms~\ref{alg:nsync} and \ref{alg:quartz} is described in the following theorem.
We include a proof sketch in the appendix.

\begin{theorem} \label{thm:primal_and_dual} \textbf{(Complexity: Primal vs Dual RCD)}
Let $\{\phi_j\}$ be convex and $\beta$-smooth.
\\ (i) If $\hat{S}_P$ is proper (i.e., $p_i>0$ for all $i$), and $u$  satisfies \eqref{eq:ESO_P}, then iterates of primal RCD satisfy
\begin{equation}\label{eq:K_P}k \geq K_P = K_P( \hat{S}_P,\epsilon)~{:=}~ \max_{i \in [d]} \left( \frac{\beta u_i + \lambda n }{p_i \lambda n }\right) \log\left( \frac{C_P}{\epsilon} \right) \quad \Rightarrow \quad \mathbb{E}[P(w^k) - P(w^*)] \leq \epsilon,\end{equation}
where $C_P$ is a constant depending on $w^0$ and $w^*$.\\
(ii) If $\hat{S}_D$ is proper (i.e., $q_i>0$ for all $i$),  and $v$ satisfies \eqref{eq:ESO_D}, then  iterates of dual RCD satisfy
\begin{equation}\label{eq:K_D}k \geq K_D = K_D(\hat{S}_D,\epsilon) ~{:=}~ \max_{j \in [n]} \left( \frac{\beta v_j + \lambda n }{q_j \lambda n }\right) \log\left( \frac{C_D}{\epsilon} \right) \quad \Rightarrow \quad \mathbb{E}[P(w^k) - P(w^*)] \leq \epsilon, \end{equation}
where $C_D$ is a constant depending on $w^0$ and $w^*$.

\end{theorem}

For the dual method a stronger guarantee can be established (see \cite{Quartz}): as soon as $k\geq L_D(\hat{S}_D,\epsilon)$, we have $\mathbb{E}[P(w^k)-D(\alpha^k)]\leq \epsilon$. Clearly, this stronger result implies the claim in part ii) of the above theorem.

\subsection{Average cost of a single iteration}

Let $\|\cdot\|_0$ be the number of nonzeros in a matrix/vector. It is easy to observe that the average  cost of a single iteration of  Algorithm~\ref{alg:nsync} is  \begin{equation}\label{eq:W_P}  W_P(\mathbf{X},\hat{S}_P)~{:=}~ \mathcal{O}\left(\mathbb{E}\left[ \sum_{i \in \hat{S}_P} \|\mathbf{X}_{i:}\|_0\right]\right)= \mathcal{O}\left(\sum_{i=1}^d p_i \|\mathbf{X}_{i:}\|_0\right),\end{equation}
and for Algorithm~\ref{alg:quartz} it is   \begin{equation}\label{eq:W_D}  W_D(\mathbf{X},\hat{S}_D)~{:=}~ \mathcal{O}\left(\mathbb{E}\left[ \sum_{j \in \hat{S}_D} \|\mathbf{X}_{:j}\|_0\right]\right)= \mathcal{O}\left(\sum_{j=1}^n q_j \|\mathbf{X}_{:j}\|_0\right).\end{equation}
We remark that the constant hidden in $\mathcal{O}$ may be larger for Algorithm~\ref{alg:nsync} than for Algorithm~\ref{alg:quartz}. The reason for this is that for Algorithm~\ref{alg:nsync} we compute the one-dimensional derivative $\phi_j'$ for every nonzero term in the sum, while for Algorithm~\ref{alg:quartz} we do this only once. Depending on the loss $\phi_j$, this may lead to slower iterations. For example, if $\phi_j$ isthe  logistic loss, experimentation shows that the constant is around 50. On the other hand, if $\phi_j$ is the squared loss, the constant is 1.
\subsection{Total complexity}

By combining the bounds on the number of iterations provided by Theorem~\ref{thm:primal_and_dual} with the formulas \eqref{eq:W_P} and \eqref{eq:W_D} for the cost of a single iteration we obtain the following expressions for the {\em total complexity} of  the two algorithms, where we ignore the logarithmic terms and drop the $\tilde{O}$ symbol:\begin{equation}\label{eq:T_P}  T_P = T_P(\mathbf{X},\hat{S}_P) ~{:=}~ K_P W_P  \overset{\eqref{eq:K_P} + \eqref{eq:W_P}}{=}  \left( \max_{i \in [d]} \frac{\beta u_i + \lambda n }{p_i \lambda n }\right) \left(\sum_{i=1}^d p_i \|\mathbf{X}_{i:}\|_0\right), \end{equation}  \begin{equation}\label{eq:T_D}  T_D = T_D(\mathbf{X},\hat{S}_D) ~{:=}~  K_D W_D  \overset{\eqref{eq:K_D} + \eqref{eq:W_D}}{=} \left( \max_{j \in [n]} \frac{\beta v_j + \lambda n }{q_j \lambda n }\right) \left(\sum_{j=1}^n q_j \|\mathbf{X}_{:j}\|_0\right).\end{equation}

\section{Choosing a Sampling that Minimizes the Total Complexity}

In this section we  identify the {\em optimal sampling} in terms of the {\em total complexity.} This is different from previous results on {\em importance sampling}, which neglect to take into account the cost of the iterations \cite{NSync,Quartz,IProx-SDCA, ImportanceSrebro}. For simplicity, we shall only consider {\em serial} samplings, i.e., samplings which only pick a single coordinate at a  time. The situation is much more complicated with non-serial samplings where first importance sampling results  have only been derived recently \cite{ISM}.

\subsection{Uniform Sampling}

The simplest serial sampling is the {\em uniform sampling}: it selects every coordinate with the same probability, i.e. $p_i = 1/d, ~\forall i \in [d]$ and $q_j = 1/n, ~\forall j \in [n]$. In view of \eqref{eq:T_P}, \eqref{eq:T_D} and \eqref{eq:98s98hsd}, we get
\[T_P =\|\mathbf{X}\|_0 \left( 1 + \frac{\beta}{\lambda n}\max_{i \in [d]} \|\mathbf{X}_{i:}\|_2^2 \right)\] and \[T_D = \|\mathbf{X}\|_0\left( 1 + \frac{\beta}{\lambda n}\max_{j \in [n]} \|\mathbf{X}_{:j}\|_2^2 \right).\]
We can now clearly see that whether $T_P\leq T_D$ or $T_P\geq T_D$ depends does not simply depend on $d$ vs $n$, but instead depends on the relative value of the quantities $\max_{i \in [d]} \|\mathbf{X}_{i:}\|_2^2$  and $\max_{j \in [n]} \|\mathbf{X}_{:j}\|_2^2$. Having said that, we shall not study these quantities in this paper. The reason for this is that for the cake of brevity, we shall instead focus on comparing the primal and dual RCD methods for optimal sampling which minimizes the total complexity, in which case we will obtain different quantities.

\subsection{Importance Sampling}

By {\em importance sampling} we mean the serial sampling $\hat{S}_P$ (resp.\ $\hat{S}_D$) which minimizes the bounds $K_P$ in \ref{eq:K_P} (resp.\ $K_D$ in \eqref{eq:K_D}). It can easily be seen (see also \cite{NSync}, \cite{Quartz}, \cite{IProx-SDCA}),  that importance sampling probabilities are given by \begin{equation} \label{def:importance_sampling}
p_i^* = \frac{\beta u_i + \lambda n}{\sum_{l} ( \beta u_l + \lambda n)} \qquad \mbox{and} \qquad q_j^* = \frac{\beta v_j + \lambda n}{\sum_{l} (\beta v_l + \lambda n)}.
\end{equation}
On the other hand, one can observe that the average iteration cost of importance sampling may be larger than the average iteration cost of uniform serial sampling. Therefore, it is a natural question to ask, whether it is necessarily better. In view of \eqref{eq:T_P}, \eqref{eq:T_D} and \eqref{def:importance_sampling}, the total complexities for importance sampling are
\begin{equation} \label{eq:MMM}  T_P =\|\mathbf{X}\|_0 + \frac{\beta}{\lambda n}\sum_{i=1}^d \|\mathbf{X}_{i:}\|_0 \|\mathbf{X}_{i:}\|_2^2 , \qquad  T_D = \|\mathbf{X}\|_0 + \frac{\beta}{\lambda n}\sum_{j=1}^n \|\mathbf{X}_{:j}\|_0\|\mathbf{X}_{:j}\|_2^2. \end{equation}
Since a weighted average is smaller  than the maximum, the total complexity of both methods with importance sampling is always better than with uniform sampling. However, this does not mean that importance sampling is the sampling that minimizes total complexity.

\subsection{Optimal Sampling}

The next theorem states that, in fact, importance sampling {\em does} minimize the total complexity.

\begin{theorem}\label{thm:runtime} The optimal  serial sampling (i.e., the serial sampling minimizing the total expected complexity $T_P$ (resp, $T_D$))  is the importance sampling \eqref{def:importance_sampling}.
\end{theorem}

\section{The Face-Off}

In this section we investigate the two quantities in \eqref{eq:MMM}, $T_P$  and $T_D$, measuring the total complexity of the two methods   as functions of the data $\mathbf{X}$. Clearly, it is enough to focus on the quantities
\begin{equation} \label{eq:two_quantities}   C_P(\mathbf{X}) ~{:=}~ \sum_{i=1}^d \|\mathbf{X}_{i:}\|_0 \| \mathbf{X}_{i:}\|^2\qquad \text{and} \qquad C_D(\mathbf{X}) ~{:=}~ \sum_{j=1}^n \|\mathbf{X}_{:j}\|_0 \| \mathbf{X}_{:j}\|^2.\end{equation}
We shall ask questions such as: when is $C_P(\mathbf{X})$ larger/smaller than $C_D(\mathbf{X})$, and by how much. In this regard, it is useful to note that  $C_P(\mathbf{X}) = C_D(\mathbf{X}^\top)$.  Our first result gives tight lower and upper bounds on their ratio.

\begin{theorem}\label{thm:Bound_Any_Data} For any $\mathbf{X}\in \mathbb{R}^{d \times n}$ with no zero rows or columns,  we have the bounds $\|\mathbf{X}\|_F^2 \leq C_P(\mathbf{X}) \leq n\|\mathbf{X}\|_F^2$ and $\|\mathbf{X}\|_F^2\leq C_D(\mathbf{X}) \leq d\|\mathbf{X}\|_F^2$. It follows that $ 1/d\leq C_P(\mathbf{X}) / C_D(\mathbf{X}) \leq n.$ Moreover, all these bounds are tight.
\end{theorem}

Since $C_P(\mathbf{X})$ (resp. $C_D(\mathbf{X})$) can dominate the expression \eqref{eq:T_P} (resp.\ \eqref{eq:T_D}) for total complexity, it follows that, depending on the data matrix $\mathbf{X}$, {\em the primal method can be up to $d$ times faster than the dual method, and up to $n$ times slower than the dual method.}

\subsection{Random  Data and Dense Data}

Assume now that the entries of $\mathbf{X}$ are chosen in an i.i.d.\ manner from some distribution with mean $\mu$ and variance $\sigma^2$. While this is not a realistic scenario, it will help us build intuition about what we can expect the quantities $C_P(\mathbf{X})$ and $C_D(\mathbf{X})$ to look like. A simple calculation reveals that $\mathbb{E}[C_P(\mathbf{X})] = dn\sigma^2 + dn^2\mu^2$, and $\mathbb{E}[C_D(\mathbf{X})] = dn\sigma^2 + nd^2\mu^2$. Hence, \[\mathbb{E}[C_P(\mathbf{X})]\leq \mathbb{E}[C_D(\mathbf{X})]\] precisely when $n\leq d$, which means that  the primal method is better when $n < d$ and the dual method is better when $n>d$.

If $\mathbf{X}$ is a dense deterministic matrix ($\mathbf{X}_{ij}\neq 0$ for all $i,j$), then $C_P(\mathbf{X}) = n \|\mathbf{X}\|_F^2$ and $C_D(\mathbf{X})=d \|\mathbf{X}\|_F^2$, and we reach the same conclusion as for random data: everything boils down to $d$ vs $n$.

\subsection{Binary Data} \label{sec:binary}

In this part we identify a class of data matrices for which one can have $C_P\leq C_D$ even if $d\ll n$. This class is by no means exhaustive, and serves as an example which we use to illustrate the phenomenon.

Let $\mathbb{B}^{d\times n}$ denote the set of $d\times n$ matrices $\mathbf{X}$ with (signed) binary elements, i.e., with $\mathbf{X}_{ij} \in \{-1,0,1\}$ for all $i,j$. For $\mathbf{X} \in \mathbb{B}^{d\times n}$, the expressions in \eqref{eq:two_quantities} can be also written in the form $C_P(\mathbf{X}) = \sum_{i=1}^d \|\mathbf{X}_{i:}\|_0^2$ and $C_D(\mathbf{X}) = \sum_{j=1}^n \|\mathbf{X}_{:j}\|_0^2$.
By $\mathbb{B}_{\neq 0}^{d\times n}$ we denote the set of all matrices in $\mathbb{B}^{d\times n}$ with  nonzero columns and rows.

For positive integers $a,b$ we write $\bar{a}_b ~{:=}~ b\left\lfloor\frac{a}{b}\right\rfloor$ (i.e., $a$ rounded down to the closest multiple of $b$). Further, we write \[R(\alpha,d,n) ~{:=}~ U(\alpha,d,n)/ L(\alpha,n),\]  where
\[L(\alpha,n) ~{:=}~ \frac{1}{n} (\bar{\alpha}_n^2 + (\alpha - \bar{\alpha}_n)(2\bar{\alpha}_n + n))\]
 and  \[U(\alpha,d,n) ~{:=}~ (d + 1)\overline{(\alpha - n)}_{d-1} + n - 1 + (\alpha - n + 1 - \overline{(\alpha - n)}_{d-1})^2.\]

The following is a refinement  of Theorem~\ref{thm:Bound_Any_Data} for binary matrices of fixed cardinality $\alpha$.

\begin{theorem} \label{lem:bound_ratio}
	For all $\mathbf{X} \in \mathbb{B}^{d\times n}_{\neq 0}$ with $\alpha = \|\mathbf{X}\|_0$ we have the bounds  $1/R(\alpha,n,d) \leq C_P(\mathbf{X}) / C_D(\mathbf{X}) \leq R(\alpha, d, n)$. Moreover, these bounds are tight.
\end{theorem}

The above theorem follows from Lemma~\ref{lem:general_binary_maxmin}, which we formulate and prove in the Appendix. This lemma establishes formulas for the minimum and maximum of $C_D$ and $C_P$, subject to the constraint $\|\mathbf{X}\|_0=\alpha$, in terms of the functions $L$ and $U$. Further, as we show in Lemma~\ref{lem:bound_Rabc} in the Appendix, if $d \geq n$ and $\alpha \geq n^2 + 3n$, then $R(\alpha,d,n) \leq 1$. Likewise, if $n \geq d$ and $\alpha \geq d^2 + 3d$, then $R(\alpha,n,d) \leq 1$. Combined with Theorem~\ref{lem:bound_ratio}, this has an interesting consequence, spelled out in the next theorem and its corollary.

\begin{theorem} \label{thm:98y9s8s} Let $\mathbf{X} \in \mathbb{B}^{d\times n}_{\neq 0}$.
If $d \geq n$ and   $\| \mathbf{X} \|_0 \geq n^2 + 3n$, then $C_P(\mathbf{X}) \leq C_D(\mathbf{X})$. By symmetry, if $n \geq d$ and $\| \mathbf{X}\|_0 \geq d^2 + 3d$, then $C_D(\mathbf{X}) \leq C_P(\mathbf{X})$.
\end{theorem}

This result says that for binary data, and $d\geq n$, the primal method is better than the dual method even for non-dense data, as long as the  the data is ``dense enough''.	Observe that as long as $d \geq n^2 + 3n$, 	all matrices $\mathbf{X} \in \mathbb{B}^{d\times n}_{\neq 0}$ satisfy $\|\mathbf{X}\|_0 \geq d \geq n^2 + 3n \geq n$. This leads to the following corollary.

\begin{corollary}
If $d \geq n^2 + 3n$, then for all $\mathbf{X} \in \mathbb{B}^{d\times n}_{\neq 0}$ we have $C_P(\mathbf{X}) \leq C_D(\mathbf{X})$. By symmetry, if $n \geq d^2 + 3d$, then for all  $\mathbf{X} \in \mathbb{B}^{d\times n}_{\neq 0}$ we have $C_D(\mathbf{X}) \leq C_P(\mathbf{X})$.\end{corollary}

In  words, the corollary states that for binary data where the number of features ($d$) is large enough in comparison with the number of examples ($n$), the primal method will be always better. On the other hand, if $n$ is large enough, the dual method will be always better. This behavior can be observed in Figure~\ref{fig:alpha}. For large enough $d$, all the values $R(\alpha,d,n)$ are below 1.

\begin{figure}[H]
	\centering

	\begin{subfigure}{.30\linewidth}
		\label{fig:plotR1000}
		\centering
		\includegraphics[width = \linewidth]{code/plots/plotR1000.pdf}
	\end{subfigure}
	\caption{The value $R(\alpha,d,n)$ plotted for $n=10^3$, $n\leq d\leq n^2$ and $\max\{d,n\} \leq \alpha \leq nd$.}
	\label{fig:alpha}
\end{figure}

\section{Experiments}

We conducted experiments on both real and synthetic data. The problem we were interested in is a standard logistic regression with an L2-regularizer, i.e., \[P(w) = \frac{1}{n}\sum_{j=1}^n \log(1 + \exp(-y_j \langle \mathbf{X}_{:j}, w \rangle)) + \frac{\lambda}{2}\|w\|_2^2.\] In all our experiments we used $\lambda = 1/n$ and we normalized all the entries of $\mathbf{X}$ by the average column norm.
 Note that for logistic loss there is no closed form solution for $\Delta_j^k$ in Algorithm~\ref{alg:quartz}. Therefore we use a variant of Algorithm~\ref{alg:quartz} where $\Delta_j^k = \eta (\phi_j'(\langle \mathbf{X}_{:j}, w \rangle) + \alpha_j^k)$ with the step size $\eta$ defined as $\eta = \min_{j \in [n]}(q_j \lambda n)/(\beta v_j + \lambda n)$. This variant has the same convergence rate guarantees as Algorithm~\ref{alg:quartz} and does not require exact minimization. Details can be found in \cite{Quartz}.

We plot the training error against the number of passes through the data. The number of passes is calculated according to the number of visited nonzero entries in the matrix $\mathbf{X}$. One pass means that we look at $\|\mathbf{X}\|_0$ nonzero entries of $\mathbf{X}$, but not necessarily all of them. We look at the problems from the perspective of the primal approach. The same could be done symmetrically for the dual approach.

\subsection{General Data}

We look at the matrices which give worst-case bounds for general matrices (Theorem~\ref{thm:Bound_Any_Data}) and their empirical properties for different choices of $d$ and $n$. The corresponding figures are Figure~\ref{subfig:general1} and \ref{subfig:general2}. For a square dataset, we can observe a large speed-up. For large $n$ we can observe, that the theory holds and the primal method is still faster, but because of numerical issues (we need very small and very large numbers in matrix) and the fact that the optimal value is very close to an "initial guess" of the algorithm, the difference in speed is more difficult to observe.

\begin{figure}
\centering
	\begin{subfigure}{0.25 \textwidth}
		\includegraphics[width = \textwidth]{code/plots/general_logistic_mean_n__1000_0,1000_0,2_0e9_.pdf}
		\caption{$1,000 \times 1,000$}
		\label{subfig:general1}
	\end{subfigure}
	\begin{subfigure}{0.25 \textwidth}
		\includegraphics[width = \textwidth]{code/plots/general_logistic_mean_n__100_0,100000_0,2_0e9_.pdf}
		\caption{$100 \times 100,000$}
		\label{subfig:general2}
	\end{subfigure}
	\begin{subfigure}{0.25 \textwidth}
		\includegraphics[width = \textwidth]{code/plots/news_logistic_mean_n.pdf}
		\caption{news dataset}
		\label{subfig:news}
	\end{subfigure}
	\begin{subfigure}{0.25 \textwidth}
		\includegraphics[width = \textwidth]{code/plots/leu_logistic_mean_n.pdf}
		\caption{leukemia dataset}
		\label{subfig:leukemia}
	\end{subfigure}
	\caption{Testing the worst case for general matrices and real datasets}
	\label{fig:general_real}
\end{figure}

\subsection{Synthetic Binary Data}

 We looked at matrices with all entries in  $\{a, -a, 0\}$ for some $a \neq 0$. We fixed the number of features to be $d = 100$ and we varied the number of examples $n$ and the sparsity level $\alpha=\|\mathbf{X}\|_0$. For each triplet $[d,n,\alpha]$ we produced the worst-case matrix for dual RCD according to the developed theory. The results are in Figure~\ref*{tab:table_experiments}.

\begin{figure}[]
	\begin{tabular}{m{0.02\textwidth}   >{\centering\arraybackslash} m{0.28\textwidth}
			>{\centering\arraybackslash} m{0.28\textwidth}  >{\centering\arraybackslash} m{0.28\textwidth} }
		& nnz $ \sim 1\% $ &   nnz $ \sim 10\% $ & nnz $ 100\% $ \\
		\rotatebox[origin=c]{90}{$ n = 100 $} & \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,100_0,199_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,100_0,1179_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,100_0,10000_0_.pdf}\\
		\rotatebox[origin=c]{90}{$ n = 1,000 $} & \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,1000_0,1099_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,1000_0,10989_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,1000_0,100000_0_.pdf}\\
		\rotatebox[origin=c]{90}{$ n = 10,000 $} & \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,10000_0,10099_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,10000_0,109089_0_.pdf}& \includegraphics[width=0.25\textwidth]{code/plots/alpha_logistic_mean_n__100_0,10000_0,1_0e6_.pdf}
	\end{tabular}
		\caption{Worst-case experiments with various dimensions and sparsity levels for $d = 100$}
	\label{tab:table_experiments}
\end{figure}

\subsection{Real Data}

We used two real datasets to showcase our theory: news and leukemia\footnote{both datasets are available from https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/}. The news dataset in Figure~\ref{subfig:news} is a nice example of our theory in practice. As shown in Table~\ref{tab:alltable} we have $d \gg n$, but the dual method is empirically faster than the primal one. The reason is simple: the news dataset uses a bag of words representation of news articles. If we look at the distribution of features (words), there are many words which appear just very rarely and there are words commonly used in many articles. The features have therefore a very skewed distribution of their nonzero entries. On the other hand, the examples have quite uniform distribution, as the number of distinct words in an article acts nicely. This distribution of nonzero entries highly favors the dual approach, as shown in the theory. The leukemia dataset in Figure~\ref{subfig:leukemia} is a fully dense dataset and  $d \gg n$. Therefore, as our theoretical analysis shows, the primal approach should be better. The ratio between the runtimes is not very large, as the constant $\|\mathbf{X}\|_0$ is of similar order as the additional term in the computation of the true runtime. The empirical speedup in Figures~\ref{subfig:news} and Figures~\ref{subfig:leukemia}  matches the theoretical predictions from Table~\ref{tab:alltable}.

\begin{table}[]
	{

	\begin{center}
	\begin{tabular}{l r r r r l l cc}
		dataset & $d$ & $n$ & density & $\|\mathbf{X}\|_0$ & $C_P$ & $C_D$ &  $T_P/T_D$ \\
		\hline news & 1,355,191 & 19,996 & 0.03\% & 9,097,916 & $3 \times 10^7$ & $9 \times 10^6$ & 2.0  \\
		leukemia & 7,129 & 38 & 100.00\% & 270,902 & $1\times 10^7$ & $2 \times 10^9$ & 0.5
		\\\end{tabular}
		\end{center}
		}
	\caption{Details on the datasets used in the experiments}
	\label{tab:alltable}
\end{table}

\section{Conclusions and Extensions} We have shown that the question whether RCD should be applied to the primal or the dual problem depends on the structure of the training dataset. For dense data, this simply boils down to whether we have more data or parameters, which is intuitively appealing. We have shown, both theoretically, and through experiments with synthetic and real datasets, that contrary to what seems to be a popular belief, primal RCD can outperform dual RCD even if $n\gg d$.

 In order to focus on the main message, we have chosen to present our results for simple (as opposed to ``accelerated'') variants of RCD. However, our results can be naturally extended to accelerated  variants of RCD, such as APPROX \cite{APPROX}, ASDCA~\cite{ASDCA}, APCG~\cite{APCG}, ALPHA~\cite{ALPHA} and SPDC~\cite{SPDC}.

 Likewise, for simplicity, we focused on serial sampling (i.e., sampling a single coordinate). However, it is possible to use our approach to gain insights into the performance of primal vs dual RCD for arbitrary sampling strategies \cite{NSync, Quartz, ALPHA, ESO}.

{