cludegraphics[trim={0.93cm 0.5cm 0.93cm 0.5cm},clip,width=0.4\linewidth]{resnet50-eps-converted-to.pdf}&   
     \includegraphics[trim={0.93cm 0.5cm 0.93cm.5cm},clip,width=0.4\linewidth]{wrn-eps-converted-to.pdf}  \\
     $m$ & $m$
\end{tabular}
        \caption{The effect of varying $m$ on accuracy. We see that increasing $m$ up to 32 results in better accuracy. However, increasing $m$ further leads to worse results/marginal improvements.}
        \label{fig:mb}
\end{figure}


\paragraph{Effect of varying $m$:}
In our experiments, we have observed that a larger value of $m$ often leads to better test accuracy. We recover SAM by setting $m=1$, which produces inferior results. To test this hypothesis, we set up the experiments with the CIFAR100 dataset on two CNNs, ResNet50 and WRN-28-10, in the same setup as in Section~\ref{imagesec}. We run mSAM for different values of $m\in\{4,8,16,32,64\}$. The accuracy results for these experiments are shown in Figure~\ref{fig:mb}.

Increasing $m$ improves the performance up to $m\approx 32$. However, a value of $m$ larger than this threshold either leads to worse performance or marginal improvements, so increasing $m$ does not necessarily result in better generalization. Intuitively, when the micro-batch is too small, the perturbation derived according to the micro-batch might not be a good estimate of the actual SAM perturbation, leading to worse performance. We leave the theoretical analysis of such a phenomenon an interesting direction for future research. We also note that understanding how the optimal value of $m$ and batch size interact is an open question for future work.

\begin{table}[htbp]
\small
\centering
    \caption{$\lambda_{\max}$ for CNNs}
\label{eigen-table}
\begin{footnotesize}
\begin{sc}
\begin{center}
\begin{tabular}{cccc}
\toprule
 Model & Vanilla & SAM &  mSAM
\\  
\midrule
ResNet50     &  $26\pm2$  & $21\pm3$  & $18\pm1$ \\
WRN-28-10     &   $92\pm4$ & $30\pm2$  & $17\pm1$ \\
\bottomrule
\end{tabular}
\end{center}
\end{sc}
\end{footnotesize}
\end{table}


\paragraph{Are mSAM solutions flat?}
The SAM algorithm hypothesizes that flat solutions generalize better. Since mSAM consistently outperforms SAM, it is worth investigating if mSAM settles for even flatter solutions. To that end and to quantify sharpness, we calculate the largest eigenvalue of the Hessian of the loss function $\mathcal{L}_{\mathcal{S}}(\Vec{w})$ at the final solution, denoted as $\lambda_{\max}$. This metric is widely accepted to be a good indicator of the sharpness/flatness of a solution~\cite{wucw18}. We calculate $\lambda_{\max}$ over the full train data using power iteration, as implemented by~\cite{hessian-eigenthings}. We use the ResNet50 and WRN-28-10 models trained on CIFAR100 (see Section~\ref{imagesec}) to calculate $\lambda_{\max}$. The average results for these experiments are reported in Table~\ref{eigen-table}. We see that mSAM leads to solutions with smaller $\lambda_{\max}$ than SAM and vanilla SGD, which agrees with our theoretical results from Section~\ref{sec:theory}.


\paragraph{mSAM runtime:}\label{app:runtime}
A general misconception about mSAM is that it is computationally inefficient, as the total number of forward-backwards passes in the network is multiplied by $m$ \cite{nonsensemsam}. However, note that these passes are performed on micro-batches, which are $m$ times smaller than the actual minibatch. Hence, the overall computational cost gets amortized and is never as high as $m$ times the cost of SAM. In practice, on large networks, the runtime of mSAM is \emph{only} $1.2$-$1.3$ times more compared to SAM. In Appendix~\ref{app:switch}, we present numerical evidence for mSAM efficiency and discuss a few hybrid algorithms to reduce the computational cost of mSAM even further.





\section{mSAM and Computational Efficiency}\label{app:switch}
Since SAM requires two forward-backwards passes for each batch of data, SAM is almost twice as slow as vanilla training. In our experiments, mSAM appears to be slower than SAM, although not $m$ times slower, as suggested by~\citet{nonsensemsam}. To demonstrate that, we report the average runtime of our experiments from Section~\ref{imagesec} on CIFAR100 data in Table~\ref{runtime-table} and ImageNet data in Table~\ref{runtime-table-imagenet}. Expectedly, SAM is almost twice as slow as the vanilla method in most cases (despite ViT-S/32 experiment where the common data pre-processing stage requires more time for preparing the images into a sequence of 32 patches). Interestingly, in the worst case, mSAM is \emph{only} twice as slow as SAM, and in the best case, the computational penalty is only within $10\%$ increase compared to SAM. The primary reason is that the overall computation complexity of forward-backwards passes for mSAM and SAM is not too different. In each epoch, mSAM performs $m$ times more passes on micro-batches that are $m$ times smaller.

\begin{table}[ht!]
\caption{Runtime of different methods and architectures on CIFAR100 data (in seconds)}
\label{runtime-table}
\begin{footnotesize}
\begin{sc}
\begin{center}
\begin{tabular}{ccccc}
\toprule
 Model & Vanilla & SAM &  mSAM 
\\  
\midrule
ResNet50      & $4497\pm11$ & $7440\pm9$  &  $16196\pm77$ \\
 WRN     & $10675\pm18$  & $17483\pm40$  &   $22261\pm24$\\
 ViT-B/16      & $4349\pm21$ & $7007\pm43$ &   $8163\pm14$ \\
 \bottomrule
\end{tabular}
\end{center}
\end{sc}
\end{footnotesize}
\end{table}

\begin{table}[ht!]
\caption{Runtime of different methods and architectures on ImageNet data (in hours)}
\label{runtime-table-imagenet}
\begin{footnotesize}
\begin{sc}
\begin{center}
\begin{tabular}{ccccc}
\toprule
 Model & Vanilla & SAM &  mSAM 
\\  
\midrule
ResNet50      & $14.74\pm0.68$ & $25.54\pm0.25$  & $25.87\pm0.08$ \\
 WRN-50-2-bottleneck     & $27.68\pm0.18$ & $52.02\pm0.36$  & $58.93\pm0.26$ \\
 ViT-S/32       & $19.91\pm0.13$ & $24.50\pm0.26$  & $30.66\pm0.15$ \\

 \bottomrule
\end{tabular}
\end{center}
\end{sc}
\end{footnotesize}
\end{table}



\begin{figure}[htbp]
\centering


\begin{tabular}{cc}
 ResNet50 & WRN-28-10 \\
     \includegraphics[scale=0.40]{switch_res-eps-converted-to.pdf}&   
     \includegraphics[scale=0.40]{switch_wrn-eps-converted-to.pdf}  
\end{tabular}
\caption{Effect of switching training algorithm}
\label{fig:switch}
\end{figure}

Although mSAM does not appear to be computationally prohibitive in our experiments, it is still not as efficient as vanilla training, leaving room for further improving its efficiency. To that end, we conduct the following set of experiments. Building on our CIFAR100 experiments from Section~\ref{imagesec}, we start the training either with mSAM or vanilla training and then switch to the other training algorithm at some point. We keep all other training parameters fixed. The accuracy results for this setup for ResNet50 and WRN-28-10 are