abstract: Estimating the remaining surgery duration (RSD) during surgical procedures can be useful for OR planning and anesthesia dose estimation. With the recent success of deep learning-based methods in computer vision, several neural network approaches have been proposed for fully automatic RSD prediction based solely on visual data from the endoscopic camera. We investigate whether RSD prediction can be improved using unsupervised temporal video segmentation as an auxiliary learning task. As opposed to previous work, which presented supervised surgical phase recognition as auxiliary task, we avoid the need for manual annotations by proposing a similar but unsupervised learning objective which clusters video sequences into temporally coherent segments. In multiple experimental setups, results obtained by learning the auxiliary task are incorporated into a deep RSD model through feature extraction, pretraining or regularization. Further, we propose a novel loss function for RSD training which attempts to counteract unfavorable characteristics of the RSD ground truth. Using our unsupervised method as an auxiliary task for RSD training, we outperform other self-supervised methods and are comparable to the supervised state-of-the-art. Combined with the novel RSD loss, we slightly outperform the supervised approach.
author: Dominik Rivoir; Sebastian Bodenstedt; Felix von Bechtolsheim; Marius Distler; Jürgen Weitz; Stefanie Speidel
title: Unsupervised Temporal Video Segmentation as an Auxiliary Task for Predicting the Remaining Surgery Duration[^1]

# Introduction

Resources in the operating room (OR) are among the most expensive in a hospital and careful OR planning is crucial in order to minimize waiting times and idle phases. Estimating the remaining surgery duration (RSD) at specified points during an intervention can facilitate more efficient utilization of OR resources.
This work builds on deep learning-based methods for fully automated RSD prediction based solely on endoscopic video data . Since the remaining time for each frame can be inferred automatically from a given videos, RSD prediction is a self-supervised task. This property is especially useful in medical applications, where manually annotating data is expensive.
However, RSD prediction is an extremely challenging task due to the complexity and uniqueness of a surgical procedure. It appears to require a high-level understanding of the workflow and progress of the surgery. These factors probably contribute to RSD models tending to overfit without proper regularization or pretraining . To alleviate this problem, Twinanda et. al. propose an RSD prediction network which is encouraged to learn progress-related features and utilizes the elapsed time in addition to visual features . Bodenstedt et. al. use multimodal sensor data from the OR including visual data and tool signals for their prediction . State-of-the-art results are obtained by Aksamentov et. al. who suggest to pretrain the RSD model on surgical phase recognition as an auxiliary task . However, surgical phase recognition is a supervised task and therefore reduces the advantages of self-supervised RSD training.
Our contributions consist of proposing an unsupervised auxiliary task to improve RSD prediction, namely unsupervised temporal video segmentation. To solve the auxiliary task, we present a method for finding segmentations that capture the progress of a surgery similar to surgical phases but without the need for manual annotations. As indicated in , progress-related features can be beneficial for RSD prediction. Using an unsupervised auxiliary task makes this approach widely applicable to different datasets. Several image-based unsupervised temporal video segmentation methods have been proposed . We adopt the method from since its iterative procedure allows us to learn task-related image features. The other approaches extract or learn features prior to segmentation, making them unsuitable as an auxiliary task. Finally, we propose a novel loss function that targets undesirable characteristics of the RSD ground truth.

# Methods

Our approach combines models for RSD prediction and unsupervised temporal video segmentation. A model consisting of a Convolutional Neural Network (CNN) for visual feature extraction and a Long Short-Term Memory network (LSTM) for propagating information through time is trained to perform our main task, RSD prediction, similar to . For the temporal segmentation task, we use an unsupervised approach to train a discriminative-generative model alternating between learning segmentation labels through a generative model and learning visual features in a discriminative CNN-LSTM network. The results obtained by solving the temporal segmentation task can be leveraged for RSD prediction in several ways. First, we assume that the temporal segmentation training encourages the CNN-LSTM model to learn features relevant for RSD prediction. Thus, we investigate reusing the learned feature representations by initializing the CNN-LSTM model for RSD prediction with the learned network weights. We then pursue two different strategies for further training the RSD model: we either finetune only the upper layers or none of the layers in the CNN. In a complementary approach, we use the obtained segment labels to formulate an additional objective to regularize the RSD model during training.

## RSD Model

For our RSD model (Fig. , right), we use an AlexNet-style CNN  to extract visual features from the video frames of a recorded surgical procedure. The feature representations are concatenated with the elapsed time $t_{el}$ of the procedure and fed into an LSTM, similar to . The LSTM can consider features from the current and previous frames and produces an RSD estimate for each frame of the video. The network predicts the remaining duration in minutes, scaled by a factor of $0.05$ due to high values of up to 100 minutes. RSD prediction is formulated as a regression task and optimized according to the SmoothL1 loss . We use a simpler model instead of *RSDNet* , since the latter showed no empirical improvement in combination with our auxiliary task.

## Unsupervised Temporal Video Segmentation Model

We extend a method from for recognition and segmentation of complex activities in videos, i.e. activities consisting of several subactivities. The author's definiton of a complex activity can be applied to surgeries, where subactivities could represent surgical phases or similar steps.
The unsupervised learning algorithm alternates between learning frame features and subactivity labels (Fig. ). Given the current subactivity labels, a discriminative appearance model learns frame features in a supervised manner. A generative temporal model is then estimated, which models the distribution of subactivity lengths and subactivity orders, given the distribution of frames in the learned appearance space. The subactivity lengths and order determine the segmentation of a video. After sampling new lengths and orders and subsequently updating subactivity labels, the algorithm continues to learn new frame features.
The discriminative appearance model is a CNN-LSTM model (Fig. , left) optimized via the cross-entropy loss. An extensive hyperparameter search suggested the use of ten subactivities. Opposed to our deep learning approach, the appearance model in the original paper learns a simple linear embedding of image features. When replacing this simple model by a complex CNN-LSTM model, care must be taken to avoid overfitting on unrefined segmentations from early iterations. To this end, only the top two layers of the network are optimized in the first iteration and layers are added incrementally after each iteration (Fig. , left). In turn, the incremental depth increase requires an initialization of the fixed layers. We pretrain the CNN using the 2nd-order temporal coherence objective , which has shown promising results on a similar task .
The generative temporal model estimates the joint distribution of frame features and subactivity segmentations. The distribution over segmentations is modeled by distributions over the length of each subactivity *(Multinomial)* and over the order of subactivities *(Generalized Mallows Model)*. Sampling-based approximations are used to infer segmentations. The generative temporal model is almost identical to the one proposed in . We only drop background model.
The method produces new models after each iteration. Hence, we need to evaluate and select a model to use as a support for the RSD model. Since the ground truth segmentation labels are unknown, we require a surrogate quality measure. We define a measure $TC$ which quantifies the temporal coherence of subacitivty predictions by the appearance model. More precisely, we measure the prediction's accuracy with respect to the best match of coherent segmentations with the same subacitivity lengths. This measure intends to capture how well a model has learned progress-related features. Preliminary experiments showed that the measure selects models which are beneficial for RSD prediction.

## Combined Learning Pipelines

Fig. shows three strategies for combining models.
**Feature extraction:** The unsupervised temporal segmentation method is used to train the CNN-LSTM network of the discriminative appearance model. The weights learned from layers *Conv1* to *FC1* are then re-used for the RSD model. While training the RSD model, the initialized layers are fixed. Only layers *FC2*, *LSTM* and *FC3* are optimized. This method is equivalent to feature extraction, where layers *Conv1* to *FC1* serve as a feature extractor for a shallow RSD model.
**Pretraining:** Pretraining is almost identical to feature extraction, except that the layers *Conv5* and *FC1* are optimized during RSD training after being initialized by the temporal segmentation method. In order to prevent the previously learned information from being overwritten too quickly, a lower learning rate is applied to pretrained layers. To summarize, layers *Conv1* to *Conv4* are fixed, *Conv5* and *FC1* are optimized with a low learning rate, and *FC2*, *LSTM* and *FC3* are optimized using the regular learning rate.
**Regularization:** The resulting subactivity labels of a learned temporal segmentation model are re-used for supervision during RSD training. First, segmentations are learned for each video by the unsupervised temporal segmentation model. Then, the RSD model is jointly trained on RSD prediction and predicting the current subactivity according to the previously found segmentations.

## Corridor-based RSD Loss Function

In the early stages of a procedure it is extremely challenging to correctly predict the remaining duration since later occurring events are not yet known. To account for this, we propose an alternative RSD loss function which reduces the influence of early errors. Intuitively, we do not want to penalize the best guess at the beginning of a procedure, which is the average length of the given procedure type. For each video, we therefore define an area between the ground truth $gt(t)$ over time and a naïve median-based prediction $n(t) = max(t_{median} - t, 0)$, where $t_{median}$ is the median duration of all procedures in the training set. Errors within this corridor are decreased by a weighting function $\pi$ (Fig. ). The corridor border $c(t)=\alpha_t gt(t) + (1-\alpha_t) n(t)$ is a linear combination of the ground truth $gt(t)$ and the median-based prediction $n(t)$.
Here $\alpha_t= 1 - \frac{2}{1+e^{5\cdot prog(t)}}$ is a time-dependent linear factor similar to the tanh function, where $prog(t) = \frac{t}{gt(t)+t}$ is the progress of the surgery in percent. Intuitively, $c(t)$ is closer to the median-based prediction $n(t)$ at early time points, when little information is available, and approaches the ground truth $gt(t)$ as the procedure progresses. The weight $\pi(y,t)$ for a prediction $y$ at time $t$ is given by $$\pi(y,t)=
\begin{cases}
\left(\frac{|y-gt(t)|}{|c(t)-gt(t)|}\right)^2, & \text{if } c(t) \leq y \leq gt(t) \text{ or } gt(t) \leq y \leq c(t)\\
1, & \text{otherwise}
\end{cases}$$ $\pi$ realizes a smooth weighting distribution along the $y$-axis inside the corridor from $y=gt(t)$ to $y=c(t)$ (with $\pi(gt(t),t)=0$ to $\pi(c(t),t)=1$). For predictions $y$ outside the corridor, $\pi(y,t)=1$. The corridor-weighted loss is finally given by $$CorrSmoothL1(y,t) = \pi(y,t) \cdot SmoothL1(y,gt(t))$$

# Evaluation

We evaluate our proposed models on the publicly available Cholec80 dataset . We use 50 videos for training, 10 for validation and 20 for testing. Video frames are extracted at 1fps. We train the RSD models using the Adam optimizer (200 epochs, learning rate $10^{-5}$, batch size 384, $\mathcal{L}_2$-weight $10^{-5}$). For the pretraining pipelines, we use SGD, run 250 epochs and update pretrained layers with a learning rate of $10^{-6}$ since these settings empirically perform better. The other settings are kept. For the segmentation model, Adam, learning rate $10^{-5}$, batch size 384, 5 epochs per iteration, 8 iterations, $\mathcal{L}_2$-weight $10^{-4}$ are used. We select the best model from iterations 6 to 8 according to our TC measure (Sec. ).

## Baselines

We consider four baselines for RSD prediction: The simplest baseline is the RSD model from Section trained only on single-task RSD prediction with no auxiliary task *(None)*. The other baselines are supported by auxiliary tasks each using all three proposed pipelines from Sec. . The first auxiliary task is temporal segmentation into 10 uniform segments *(Uniform)*. This is an interesting baseline that can provide insight into how much RSD-relevant information is gained by learning more refined segmentations. The other two auxiliary tasks are state-of-the-art approaches, namely supervised surgical phase recognition *(Phase)*  and self-supervised prediction of progress $prog(t)$ *(Progress)* reimplemented from , which is an updated version of the RSDNet from . For the phase approach, we use the regularized RSD model from Fig. in order preserve compariability to our methods. The main differences to the architecture from are that we use an AlexNet-style CNN like in and that we incorporate the elapsed time into the prediction like in . Hyperparameters of the optimization are identical to the proposed methods.

## Results

```latex
\resizebox{.85\linewidth}{!}{
\begin{tabular}{|l|c|c|c|}
\hline
\textbf{Auxiliary Task} & \textbf{Feature Extraction} & \textbf{Pretraining} & \textbf{Regularization}\\
\hline
Unsup. temp. seg. \textit{(ours)} & 9.0 $(\pm 0.1)$ & 9.3 $(\pm 0.2)$ & 9.2 $(\pm 0.5)$ \\
\hline
None & \multicolumn{3}{|c|}{9.7 $(\pm 0.1)$}\\
Uniform & 9.4 & 9.4 & 9.4\\
Progress & 9.0 $(\pm 0.1)$ & 9.5 & 9.6\\
Phase \textit{(supervised)} & \textbf{8.9} $(\pm 0.1)$ & \textbf{8.9} & 9.1\\
\hline
\end{tabular}}
```
```latex
\resizebox{.9\linewidth}{!}{
\begin{tabular}{|l|c|c|c|c|}
\hline
& \multicolumn{2}{|c|}{\textbf{Feature extraction}} & \multicolumn{2}{|c|}{\textbf{Regularization}}\\
\hline
\textbf{Auxiliary Task} & \textbf{SmoothL1} & \textbf{CorrSmoothL1} & \textbf{SmoothL1} & \textbf{CorrSmoothL1}\\
\hline
Unsup. temp. seg. \textit{(ours)} & 9.0 $(\pm 0.1)$ & 9.1 $(\pm 0.2)$ & 9.2 $(\pm 0.5)$ & \textbf{8.7} $(\pm 0.2)$\\
\hline
None* & 9.7 $(\pm 0.1)$ & 9.1 $(\pm 0.5)$ & 9.7 $(\pm 0.1)$ & 9.1 $(\pm 0.5)$\\
Uniform & 9.4 & 9.3 & 9.4 & 9.4\\
Progress & 9.0 $(\pm 0.1)$ & 9.1 & 9.6 & 9.2 $(\pm 0.4)$\\
Phase \textit{(supervised)} & 8.9  $(\pm 0.1)$ & 9.0 & 9.1 & 8.9 $(\pm 0.1)$\\
\hline
\end{tabular}}
```

Table shows the mean average error (MAE) in minutes for each of our proposed models as well as all baselines using the SmoothL1 loss. All experiments involving our proposed method are performed four times, averaged and indicated by a standard deviation. Baseline experiments for settings which were effective for our method are repeated four times, in order to obtain more significant results.
Comparing our proposed methods, feature extraction achieves the best results ($9.0\pm0.1$ min. MAE), while pretraining performs worst ($9.3\pm0.2$) and high variances were observed during regularization ($9.2\pm0.5$). The high expressivity of RSD models likely causes overfitting in the two latter setups. In the pretraining setting, the RSD model is the least expressive, as only the top layers are optimized after initialization by the segmentation method. Hence, it is supposedly less prone to overfitting. We also observe that our approach outperforms or matches the self-supervised approaches (single-task RSD, uniform segmentation and progress) for all learning pipelines. Using feature extraction, we even achieve results comparable to the supervised phase-based approach (9.0 vs. 8.9).
Next, we compare the CorrSmoothL1 loss to SmoothL1 on the previously most successful feature extraction and the regularization pipeline (Table ), since the high variance in regularization experiments indicates potential for improvement. The first two result columns show RSD errors for both loss functions on the feature extraction pipeline. No clear difference can be observed. The single-task RSD model as well as most regularized models, however, improve drastically. Since CorrSmoothL1 aims to reduce overfitting, it is more effective on very expressive deep models such as the regularization models or the single-task model. In the feature extraction setup, which has significantly fewer trained parameters during RSD training, the model's low expressivity probably prevents further improvement. Using regularization, our approach improves from 9.2 to 8.7 minutes MAE and therefore exceeds our previously best result as well as all baselines. We even outperform all supervised phase-based setups. It is not clear how significant this difference is, since the supervised approach performed slightly better than ours in the SmoothL1 setup. However, even comparable results are very promising and our approach performs at least on a similar level as supervised methods. Fig. shows that subactivity labels corresponds fairly well to surgical phases but are more fine grained due to the higher number of segments. Using a hand-picked mapping from subactivities to phases, we achieve an accuracy of 71% and 72% on the training and test set for surgical phase recognition. A limitation of our proposed method remains the complexity of the whole model and achieving stable results poses a challenge.

# Conclusion

We present unsupervised temporal video segmentation as a novel auxiliary task for video-based RSD prediction and propose three different learning pipelines to utilize unsupervised temporal segmentation learning for RSD modeling. In our experiments on the Cholec80 dataset, our approach compares favorably with self-supervised auxiliary tasks and performs comparably to the state of the art, which utilizes supervised surgical phase recognition as auxiliary task. This is very promising since the method does not require any manual annotations and therefore has potential for improvement by utilizing larger, unlabeled datasets. Further, we specifically target the problem that RSD ground truth labels can be misleading in early stages of a procedure. Our novel corridor-based loss shows clear improvements on deep RSD models. Using the corridor-based loss, we even outperform the state of the art when we regularize the RSD model with the unsupervised temporal segmentation task. Future work could evaluate our method on procedure types with higher variance in duration and therefore lower correlation between RSD and progress. Analyzing how our method transfers to these procedures is interesting since temporal segmentations can potenially model more complex temporal structures than progress. Also, the similarity of unsupervised segmentations and surgical phases induces interesting new research directions.

[^1]: Funded by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part of Germany's Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of Excellence "Centre for Tactile Internet with Human-in-the-Loop" (CeTI) of Technische Universität Dresden.


abstract: Deep neural networks are widely used in various fields because of their powerful performance. However, recent studies have shown that deep learning models are vulnerable to adversarial attacks, i.e., adding a slight perturbation to the input will make the model obtain wrong results. This is especially dangerous for some systems with high-security requirements, so this paper proposes a new defense method by using the model super-fitting state to improve the model's adversarial robustness (i.e., the accuracy under adversarial attacks). This paper mathematically proves the effectiveness of super-fitting and enables the model to reach this state quickly by minimizing unrelated category scores (MUCS). Theoretically, super-fitting can resist any existing (even future) CE-based white-box adversarial attacks. In addition, this paper uses a variety of powerful attack algorithms to evaluate the adversarial robustness of super-fitting, and the proposed method is compared with nearly 50 defense models from recent conferences. The experimental results show that the super-fitting method in this paper can make the trained model obtain the highest adversarial robustness.
address: 1School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450000, China
  2Information Engineering University, Zhengzhou 450001, China
author: Youhuan Yang $^{1,2}$, Lei Sun $^{2}$, Leyu Dai $^{2}$, Song Guo $^{2}$, Xiuqing Mao $^{2}$, Xiaoqin Wang $^{2}$, Bayi Xu $^{1,2}$
title: CE-based white-box adversarial attacks will not work using super-fitting

# Introduction

Many studies have shown that the deep learning models widely used in various fields are very fragile. Attackers could fool the classifier to achieve evasion attacks by adding slight perturbations to the original examples (adversarial examples) that are difficult to distinguish by human eyes. Meanwhile, various types of defense algorithms with different capabilities (adversarial robustness) have been proposed. Though these algorithms can well handle existing attack methods, they cannot achieve the same effect on future attack methods. Image preprocessing and defensive distillation do not depend on adversarial examples. These algorithms are effective for various attack methods, but their performance is suboptimal. Adversarial training is considered the most effective defense method at present (many methods are based on adversarial training), and these algorithms rely on adversarial examples (or specific types of adversarial attack algorithms). The existing adversarial defense methods share some shortcomings: 1) the adversarial robustness (accuracy) is too low, i.e., the performance of the defense model on adversarial examples is much lower than that on the original clean examples; 2) the generalization against unknown attacks is poor, and these algorithms often cannot defense effectively when a more advanced (or future) attack algorithm is proposed; 3) the implementation is complicated, and most of these defense methods have too many hyperparameters. Also, the hyperparameters of the training process requires careful tuning. Therefore, it is acknowledged that adversarial defense is a very difficult task.

Since Szegedy et al. first discovered the existence of adversarial examples, various studies on adversarial attack algorithms have developed rapidly. However, most of these attack algorithms (including white- and black-box attacks) are based on Cross-Entropy (CE) loss, and only a few attack algorithms use other (or self-define) loss functions (such as AutoAttack , C&W , etc.). This paper proposes a new defense method based on super-fitting for these widely CE-based white-box adversarial attacks. Super-fitting is a state of model training. In this state, the gradient information calculated by the CE loss will die out (i.e., Gradient Vanishing). In the experiment, it is found that the model in this state defense against almost any CE-based white-box adversarial attacks in existence. In the subsequent introductions, this paper explains the reason for this phenomenon and the mathematical principle of defense and proposes a method called MUCS to enable the defense model to quickly reach the super-fitting state. The contributions of this paper are as follows:

-   This paper proposes the concept of super-fitting in the model training phase. In this state, the model can defend almost all existing CE-based white-box adversarial attacks, and the training phase is not related to the adversarial attack algorithm. Through analysis, it can be seen that the model trained using the super-fitting method can defend against future CE-based white-box adversarial attacks.

-   This paper analyzes the reason for the super-fitting phenomenon and proposes the MUCS method to reduce the number of model training iterations to reach this state, which makes it easy to train a super-fitting model without high-power graphics.

-   Finally, this paper experimentally proves the effectiveness of the proposed method. Compared with nearly 50 defense models from top conferences (e.g., ICML, NeurIPS, ICLR, ICCV, CVPR). The experimental results indicate that super-fitting obtains the highest adversarial robustness against CE-based white-box adversarial attacks.

# Preliminaries

## Adversarial attack

Although deep neural networks perform well in many fields (e.g., face recognition, object detection, medical image process, malicious traffic detection, etc.), many studies show that deep neural networks are very fragile and vulnerable to adversarial attacks, i.e., adding slight perturbations to the input that are hard to distinguish with the human eyes (adversarial perturbations) will make the classifier obtain unexpected results. The constrained optimization problem is defined as:

$$y\neq\arg\max f(x+\delta)$$ $$s.t.\ {||\delta||}_p\le\epsilon$$

In Eq. (1), ${||\bullet||}_p$ denotes $L-p$ norm. $p\in\{1,2,\infty\}$, and this paper sets $p=\infty$, i.e., just considers $L-\infty$ attack; $\epsilon$ is a hyperparameter that limits the size of adversarial perturbations; $x$ and $x+\delta\in[0,1]^{D}$ denote origin images and adversarial images, respectively; $\delta\in[-\epsilon,+\epsilon]^{D}$ denote perturbations; $x$ is the input of classifier $f$, and $y$ is the true label. Goodfellow et al. proposed the Fast Gradient Sign Method (FGSM) based on the definition of adversarial attacks in Eq. (1). An adversarial attack could be formulated as follows: $$x_{adv}=x+\epsilon sign(\mathrm{\nabla}_x\mathcal{L}(f(x),y))$$

In Eq. (2), $sign$ represents the sign function, $\mathcal{L}$ is the loss function (i.e., CE loss in this paper), and $x_{adv}$ is the adversarial example generated by the adversarial attack algorithm. Based on FGSM, Kurakin et al. proposed a stronger multi-step iterative attack algorithm called Basic Iterative Method (BIM), which uses a smaller step size in each gradient ascent iteration process (attack process). Similar to the BIM algorithm, Madry et al. found that adding random noise to the input during the restart phase of the attack algorithm can make it stronger (i.e., approaching the lower bound of robustness). Projected Gradient Descent (PGD) generates an adversarial example by performing the iterative update: $$x_{adv}^{t+1}={{Proj}_{(x,\epsilon)}(x}_{adv}^t+\alpha{sign(\mathrm{\nabla}}_{x_{adv}^t}\mathcal{L}(f(x_{adv}^t),y)))$$ where, $$x_{adv}^0=x_{orig}+\delta_0$$ $$s.t.\ {||\delta_0||}_\infty\le\epsilon$$

In Eq. (3), $t$ is the number of iterations, $x_{orig}$ is the original image, $\delta_0$ is the randomly initialized noise, ${Proj}_{(x,\epsilon)}$ is the operation to clip perturbations in $\epsilon$-ball constrain, and $\alpha$ is the step size of every single attack. Many researchers have made improvements to PGD. For example, Croce et al. proposed Adaptive Projected Gradient Descent (APGD). Unlike the original PGD, APGD does not have a fixed step size, i.e., $\alpha$ will dynamically change in each iteration according to the value of the loss function.

Obtaining an attack starting point $x_{adv}^0$ by random initialization is always suboptimal (e.g., ODI and AutoAttack ). So, Liu et al. proposed Adaptive Auto Attack ($A^3$), which is quite different from PGD in starting point initialization. To be specific, given a random direction of diversification $w_d$ sampled from uniform distributions $U{(-1,1)}^K$ ($K$ is the total number of categories for the classification task), $A^3$ firstly calculates a normalized perturbation vector as follows: $$v(x,f,w_d)=\frac{\mathrm{\nabla}_xw_d^Tf(x)}{||\mathrm{\nabla}_xw_d^Tf(x)||}$$

Then, the starting point is calculated through the following iterative process: $$x_{init}^{t+1}={P_{\left(x,\epsilon\right)}(x}_{init}^t{+\alpha}_{init}sign(v(x_{init}^t,f,w_d)))$$

When the $T$-th initialization iteration ends, $x_{adv}^0=x_{init}^T$ is selected as the attack starting point, and then $A^3$ chooses an attack process similar to PGD. Differently, its step size is not fixed compared to PGD. The step size of the $n$-th round of attack (assuming a total of $N$ rounds of iterative attacks) is calculated as follows:

$$\alpha^n=\frac{1}{2}\epsilon\bullet(1+cos(\frac{n\ mod\ N}{N}\ \pi))$$

It is worth noting that the original $A^3$ uses the statistical results to modify the distribution information of the $w_d$ part of the dimension after randomly obtaining $w_d$. The subsequent experimental part of this paper does not take this approach. This is because the increase in revenue is small. Also, the original $A^3$ uses Margin loss for attacks, but this paper uses the CE loss.

It can be seen that in general, the white-box attack is based on the gradient, and most attack methods use the CE loss when calculating the gradient information to attack the classifier. The main reason is that the current classification tasks generally use the CE loss for model training. Its calculation formula is as follows: $$\mathcal{L}(z,y)=-z_y+ln{(\sum_{i=0}^{K-1}e^{z_i})}$$

In Eq. (7), $z$ is the output of the classifier (logits vector, $z\in\mathbb{R}^K$), and $y$ is the true label of the example ($y\in[0,K)$). Although CE loss is the most popular loss function to obtain gradients for adversarial attacks, its shortcomings are also obvious. When $z_k\rightarrow-\infty$ ($k\in[0,K)$, $k\neq y$),there will be a *Gradient Vanishing* situation. The specific explanation and formulation will be presented in Section 3.

## Adversarial defense

To address the adversarial attack threat, different types of defense algorithms have been proposed, e.g., image pre-processing , adversarial training , gradient obfuscation/hiding , regularization , etc. Among them, adversarial training is currently considered to be the most effective defense method, and its optimization process is defined as follows: $$\min_{\theta}E_{(x,y)^{D}}\max_{{||\delta||}_p\le\epsilon}\mathcal{L}(\theta,f(x+\delta),y)$$

Eq. (8) contains two optimization problems, namely the inner maximization and outer minimization problems. The inner maximization problem always tries to find the worst-case examples (adversarial examples), and the outer minimization problem is to train a model robust to adversarial examples. Many studies are based on the above-mentioned standard adversarial training. For example, CAT uses a small number of iterations in the initial stage of model training, and it increases the number of iterations of the attack when the model achieves a high accuracy against the current attack. FAT adopts early stopping in the inner maximization problem to approach the lower robustness boundary. In addition to this, there is a part of work dedicated to reducing the time of adversarial training, e.g., YOPO , FAST-AT, , etc. For example, TRADES attempts to balance accuracy between clean examples and adversarial examples. There are many methods based on adversarial training, and the performance gap between these methods is not large. A common point of view is the improvement of robustness largely depends on the internal maximization problem. A stronger attack algorithm will make the defense model more robust, but the cost will also be greater.

Different from adversarial training, defensive distillation is a gradient hiding method, which uses real labels (hard labels) to train the teacher network and uses the output of the teacher network as the soft label of the student network. The calculation of the softmax function with temperature (used to calculate the loss value) is defined as: $$F(z,y,T)=\frac{e^{z/T}}{\sum_{i=0}^{K-1}e^{z_i/T}}$$

Particularly, Eq. (9) is a normal softmax function when $T=1$. It can be seen that although defensive distillation can hide the gradient information of the teacher network to a certain extent, gradient information will still be leaked through the student network.

Although various defense methods have been proposed to resist adversarial attacks, the results are unsatisfactory, and there is no effective method that can make the defense model achieve a similar accuracy on adversarial examples to that on clean examples. Most importantly, current defense methods cannot defend well both mathematically and experimentally, even in the face of CE-based adversarial attacks that have existed for a long time.

# Super-fitting

In this section, the super-fitting phenomenon in the model training phase and its explanation are first introduced. Then, the method MUCS is proposed to accelerate the model to reach the super-fitting state.

## Phenomenon

The general model training state changes from under-fitting (at the beginning of the training phase) to fitting (the training is completed) and finally reaches the state of over-fitting. Since over-fitting will reduce the generalization of the trained model, researchers always choose to stop training when the model reaches the fitting state during the training process. However, through an experiment, this paper found that the concepts of different fitting states are relative, and over-fitting on the original/clean examples is under-fitting on the adversarial examples. By increasing the number of iterations of the training phase, the model can also reach the state of fitting on adversarial examples. This state is far behind over-fitting for clean examples, which is called super-fitting in this paper. This paper uses MiddleCNN (the network consists of three convolutional layers with 64, 128 and 256 output channels, respectively, and two fully connected layers with 1024 and 10 units, respectively. Each convolutional layer is followed by BatchNormalization , LeakReLu and Maxpooling layer.) and ResNet-18 to train defense models on the CIFAR-10 dataset to make them super-fitting. Fig. 1 shows the partial information of the two defense models in the training phase (top MiddleCNN, bottom ResNet-18), including the accuracy of clean examples for the training set and the accuracy of adversarial examples for the test set (left) and the value of the loss function (right) calculated on adversarial examples using the CE loss.

In the experiment, both models are trained using clean examples, the CE loss, and the Adam optimizer (the learning rate is set to 1e-3, and other parameters used in the training process are set to the default). No defensive measures are used, and just the number of training iterations is increased (max epoch is 30000). Afterward, PGD-20 ($\epsilon=8.0/255$, step size is $\epsilon/10$) is used on the CIFAR-10 test set to evaluate the accuracy. The accuracy of MiddleCNN on the adversarial examples generated by PGD-20 is 83.72% (the accuracy on clean examples is 88.97%), and the adversarial robustness of ResNet-18 is 68.75% (the accuracy on clean examples is 83.85%).

Obviously, super-fitting only sacrifices a part of generalization (the accuracy on the clean examples is lower than that in the fitting state), but it greatly improves the adversarial robustness of the model. Surprisingly, the loss function value of the adversarial examples calculated with the CE loss in Fig. 1 is increasing (theoretically it should be a very small value). To find the reason for this phenomenon, this paper shows the statistics of the prediction results (logits) of each category of CIFAR-10 by MiddleCNN in the super-fitting state in Fig. 2.

In Fig. 2, the output information of the same categories of examples is in the same subfigure. There are 10 bars in each subplot, and the values are the mean of model output in each dimension on the category of examples, i.e., the prediction scores of each category given by the model. It can be seen from the figure that the prediction scores of the examples in the super-fitting state of the MiddleCNN are polarized. The predicted values of examples in the dimension of the real label are much larger than others, and the values in the dimensions that do not belong to the real label are generally much smaller than 0.

Inspired by the distribution of the example output results in Fig. 2, this paper analyzes the impact of logits changes on the CE loss. It is found that when the model outputs meet $z_k\rightarrow-\infty$( $k\in[0,K)\ and\ k\neq y$), the CE-based white-box adversarial attack algorithm will not work.

## Analysis

Since $\ln{x}$ cannot be differentiated when $x\le0$, the CE loss shown in Eq. (7) often adopts an equivalent calculation: $$\mathcal{L}(b,t)=-\sum_{i=0}^{K-1}{t_ib}_i$$ where, $$t_i=\left\{\begin{matrix}
                1&i=y\\
                0&others
            \end{matrix}\right.$$ $$b_i=ln{\frac{e^{z_i}}{\sum_{j=0}^{K-1}e^{z_j}}=z_i-ln{\sum_{j=0}^{K-1}e^{z_j}}}$$ $b$ is the result of the model output after passing through the softmax and ln functions; $t$ is the one-hot encoding format of the true label of the example. From Eq. (10) and Eq. (11), it can be deduced: $$\frac{\partial\mathcal{L}}{\partial b_i}=-t_i=\left\{\begin{matrix}
                -1&i=y\\
                0&others
            \end{matrix}\right.$$ $$\frac{\partial\mathcal{L}}{\partial z_i}=\sum_{j=0}^{K-1}{\frac{\partial\mathcal{L}}{\partial b_j}\frac{\partial b_j}{\partial z_i}=\frac{\partial\mathcal{L}}{\partial b_y}\frac{\partial b_y}{\partial z_i}=-\frac{\partial b_y}{\partial z_i}}$$ $$b_y=z_y-ln{\sum_{j=0}^{K-1}e^{z_j}}$$

Then, $$\frac{\partial\mathcal{L}}{\partial z_i}=\left\{\begin{matrix}
                \frac{e^{z_y}}{\sum_{j=0}^{K-1}e^{z_j}}-1&i=y\\
                \frac{e^{z_i}}{\sum_{j=0}^{K-1}e^{z_j}}&others
            \end{matrix}\right.$$

It is easy to find $e^{z_k}\rightarrow0$ when $z_k\rightarrow-\infty(k\in[0,K)\ and\ k\neq y)$. Then, $$\frac{\partial\mathcal{L}}{\partial z_i}=0$$

There will be a gradient vanishing at this time, i.e., the CE-based white-box adversarial attack will not work because the gradient information cannot be propagated. Therefore, this paper mathematically explains how super-fitting resists adversarial attacks. Based on this, the method proposed in this paper can defend against future CE-based white-box adversarial attacks.

However, super-fitting will face two problems: 1) $z_k$ cannot reach the mathematical $-\infty$; 2) cost too much time (ResNet-18 needs about 30,000 iterations or a few days to reach the super-fitting state even with an RTX3090 graphics card). Under the floating-point operation of modern computers, the first problem can be solved easily by optimizing $z_k$ to a very small value ($z_k<0$, it does not need to reach the mathematical $-\infty$). In this case, $e^{z_k}$ will be zero, and Eq. (16) holds. To solve the second problem, a new optimization process is proposed in this paper to minimize unrelated category scores, and the specific implementation is introduced in Section 3.3.

## MUCS: Minimize unrelated category scores

Although training the model with the CE loss can also make it super-fitting, the process consumes a lot of time. Also, it is difficult to achieve a perfect super-fitting state (e.g., experiment in Fig. 1) when the model parameters are large (e.g., ResNet-18). To address these issues, this paper proposes a new optimization method called MUCS, and the new optimization problem is defined as: $$minimize\ {(z}_s-z_y)$$ where, $$s=argmax\ z_i \ and \ s\neq y$$

In Eq. (17), $z_s$ is the maximum value of prediction scores other than the true label (i.e., class scores that are not related to the true label). In fact, the goal of minimizing the scores of all irrelevant classes can be achieved just by minimizing $z_s$.

Table 1 shows the effect of MUCS and the CE loss on the super-fitting state of CIFAR-10, including the convergence speed and the final robustness accuracy (against PGD-20).

```latex
\begin{table*}
\centering
\begin{footnotesize}
\caption{Performance of MUCS compare to the CE loss}
\begin{tabular}{ccccc}
\hline\hline
\multicolumn{2}{c}{Model}&Iterations&Clean Accuracy&Robustness Accuracy\\
\hline
\multirow{2}{*}{MiddleCNN}&CE-Loss&30,000&88.97\%&83.45\%\\
&MUCS&500&86.86\%&85.83\%\\
\multirow{2}{*}{ResNet-18}&CE-Loss&30,000&84.10\%&69.72\%\\
&MUCS&3,000&82.65\%&84.21\%\\
\hline\hline
\end{tabular}
\end{footnotesize}
\end{table*}
```
```latex
\begin{table*}
\centering
\begin{footnotesize}
\caption{Performance comparison of super-fitting and defensive distillation (DD)}
\begin{tabular}{p{0.15\textwidth}<{\centering} p{0.15\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering}}
\hline\hline
\multicolumn{2}{c}{Model}&Clean&FGSM&BIM-20\\
\hline
\multirow{2}{*}{MiddleCNN}&DD&86.79\%&82.31\%&82.21\%\\
&Ours&89.05\%&89.05\%&89.05\%\\
\multirow{2}{*}{ResNet-18}&DD&80.98\%&78.40\%&78.35\%\\
&Ours&83.87\%&83.87\%&83.87\%\\
\hline\hline
\end{tabular}
\end{footnotesize}
\end{table*}
```
```latex
\begin{table*}
\centering
\begin{footnotesize}
\caption{Performance of super-fitting against powerful attack}

\begin{tabular}{p{0.2\textwidth}<{\centering} p{0.2\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering}}
\hline\hline

\multicolumn{2}{c}{CIFAR-10 Defense Model}&Clean&PGD&APGD&$A^3$\\
\hline

MiddleCNN\textbf{(20.33MB)}&super-fitting&87.32\%&\textbf{86.56\%}&\textbf{84.33\%}&\textbf{78.59\%}\\
\hline

\multirow{7}{*}{ResNet-18(43.96MB)}&Adv-regular\cite{19}&\textbf{88.99\%}&80.76\%&68.03\%&69.29\%\\
&FBTF\cite{23}&86.74\%&69.41\%&60.67\%&47.26\%\\
&CNL\cite{26}\ddag&84.60\%&71.25\%&64.62\%&76.71\%\\
&Understanding FAST\cite{27}&83.97\%&66.19\%&60.77\%&47.34\%\\
&Proxy\cite{28}&84.01\%&65.09\%&60.49\%&59.68\%\\
&DNR\cite{29}&84.35\%&62.12\%&58.03\%&45.62\%\\
&OAAT\cite{30}&83.95\%&61.44\%&57.78\%&57.34\%\\
\hline

WRN-28-4(48.95MB)&MMA\cite{31}&87.08\%&61.89\%&60.71\%&52.03\%\\
\hline

ResNet-50(95.66MB)&Robustness\cite{32}&87.06\%&61.67\%&60.50\%&52.57\%\\
\hline

\multirow{11}{*}{WRN-28-10(205.17MB)}&MART\cite{33}\dag&84.35\%&61.32\%&57.99\%&61.01\%\\
&Feature-scatter\cite{34}&84.92\%&62.29\%&58.69\%&72.34\%\\
&Adv-inter\cite{35}&85.51\%&63.51\%&60.06\%&75.23\%\\
&AWP\cite{36}\dag&85.86\%&63.69\%&60.51\%&65.39\%\\
&Geometry\cite{37}\dag\ddag&86.15\%&63.96\%&60.89\%&67.81\%\\
&Hydra\cite{38}\dag&86.28\%&63.50\%&60.61\%&58.98\%\\
&Pre-train\cite{39}&86.32\%&63.12\%&60.39\%&57.65\%\\
&Rst\cite{40}\dag&86.54\%&63.13\%&60.53\%&63.51\%\\
&ULAT\cite{41}\dag&86.71\%&63.32\%&60.86\%&67.42\%\\
&Fix-data\cite{42}&86.80\%&63.53\%&61.21\%&66.48\%\\
&RLPE\cite{43}\dag&87.03\%&63.60\%&61.38\%&65.23\%\\
\hline

\multirow{10}{*}{WRN-34-10(257.42MB)}&TRADES\cite{18}\ddag&86.95\%&63.25\%&61.13\%&57.57\%\\
&AWP\cite{36}&86.95\%&63.17\%&61.15\%&60.85\%\\
&Proxy\cite{28}\dag&86.92\%&63.09\%&61.15\%&61.48\%\\
&Self-adaptive\cite{44}\ddag&86.74\%&62.80\%&60.93\%&56.71\%\\
&Sensible\cite{45}&86.96\%&62.67\%&60.74\%&62.26\%\\
&YOPO\cite{22}&86.94\%&62.03\%&60.17\%&46.09\%\\
&IAR/SAT\cite{46}&86.92\%&61.72\%&59.90\%&53.82\%\\
&LBGAT\cite{47}\ddag&86.94\%&61.37\%&59.60\%&54.84\%\\
&FAT\cite{21}&86.90\%&61.25\%&59.54\%&57.42\%\\
&OAAT\cite{30}&86.85\%&61.41\%&59.74\%&65.07\%\\
\hline

WRN-34-15(517.84MB)&RLPE\cite{43}\dag&87.02\%&61.67\%&60.52\%&61.25\%\\
\hline

\multirow{4}{*}{WRN-34-20(866.20MB)}&Hyper-embe\cite{48}&86.81\%&61.44\%&59.79\%&63.12\%\\
&Overfit\cite{49}&86.75\%&61.34\%&59.74\%&58.67\%\\
&ULAT\cite{41}&86.71\%&61.25\%&59.68\%&58.75\%\\
&LBGAT\cite{47}\ddag&86.80\%&61.09\%&59.56\%&58.12\%\\
\hline

\multirow{3}{*}{WRN-70-16(1294.64MB)}&ULAT\cite{41}&87.15\%&61.84\%&60.56\%&60.46\%\\
&ULAT\cite{41}\dag&87.30\%&61.84\%&60.42\%&67.96\%\\
&Fix-data\cite{42}&87.16\%&62.17\%&60.98\%&67.03\%\\

\hline\hline

% \end{tabular}
% \end{footnotesize}
% \end{table*}

% \doublerulesep 0.1pt
% \begin{table*}[ht]
% \centering
% \begin{footnotesize}
% \begin{tabular}{p{0.2\textwidth}<{\centering} p{0.2\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering} p{0.1\textwidth}<{\centering}}
% \hline\hline

\multicolumn{2}{c}{CIFAR-100 Defense Model}&Clean&PGD&APGD&$A^3$\\
\hline

MiddleCNN\textbf{(20.68MB)}&super-fitting&59.21\%&\textbf{55.19\%}&\textbf{52.32\%}&\textbf{41.71\%}\\
\hline

\multirow{2}{*}{ResNet-18(44.14MB)}&Overfit\cite{49}&55.16\%&37.95\%&36.30\%&20.46\%\\
&OAAT\cite{30}&57.06\%&36.04\%&34.78\%&32.96\%\\
\hline

\multirow{2}{*}{WRN-28-10(205.39MB)}&Pre-train\cite{39}&58.32\%&36.07\%&35.03\%&35.85\%\\
&Fix-data\cite{42}&59.13\%&36.27\%&35.37\%&37.73\%\\
\hline

\multirow{4}{*}{WRN-34-10(257.64MB)}&AWP\cite{36}&59.23\%&36.21\%&35.42\%&35.54\%\\
&IAR/SAT\cite{46}&59.65\%&34.69\%&33.86\%&26.40\%\\
&LBGAT\cite{47}\ddag&59.86\%&34.76\%&34.01\%&36.56\%\\
&OAAT\cite{30}&60.27\%&34.78\%&34.05\%&34.68\%\\
\hline

WRN-34-20(866.64MB)&LBGAT\cite{47}\ddag&60.47\%&34.81\%&34.07\%&35.23\%\\
\hline

\multirow{3}{*}{WRN-70-16(1295.00MB)}&ULAT\cite{41}&60.44\%&34.12\%&33.54\%&32.50\%\\
&ULAT\cite{41}\dag&62.61\%&35.63\%&35.16\%&40.00\%\\
&Fix-data\cite{42}&\textbf{62.93\%}&36.27\%&35.81\%&39.53\%\\

\hline\hline
\end{tabular}
\end{footnotesize}
\end{table*}
```

The experimental parameter settings in Table 1 are consistent with those in Fig. 1. The experimental results show that compared with the CE loss, although the accuracy of MUCS on the original examples (Clean Accuracy) is reduced, its adversarial robustness (accuracy against adversarial attacks) is improved (Robustness Accuracy). Also, the number of iterations required to reach the super-fitting state is greatly reduced (Iterations). Therefore, this paper combines the CE loss and MUCS in the subsequent experiments and uses Eq. (18) to optimize the model: $$minimize\ CE(z,y)\ +\ {(z}_s-z_y)$$

Eq. (18) can ensure that the number of iterations of model training is greatly reduced, and the accuracy of the original examples and the adversarial examples can reach a high level (i.e., the trade-off will be better).

# Experiment

In this section, to demonstrate the effectiveness of the method proposed in this paper, super-fitting is compared with defensive distillation, which also uses gradient-based defense. This paper conducts extensive experiments to compare the adversarial robustness of 50 defense models on CIFAR-10 and the more difficult task CIFAR-100. There are 38 CIFAR-10 defense models and 12 CIFAR-100 defense models, and most of these are from recent top conferences (e.g., ICML, NeurIPS, ICLR, ICCV, CVPR).

**Setup.** 1) In the comparison experiment of super-fitting and defensive distillation, this paper adopts the weak attack algorithms FGSM and BIM-20 for adversarial robustness evaluation; 2) To compare with state-of-the-art defense models, this paper adopts the more powerful PGD-100, APGD-100, and standard $A^3$; 3) The Adam optimizer is used for super-fitting training in all experiments, and the learning rate is 1e-3. The hyperparameters of the attack algorithm are set as $\epsilon=8.0/255$ and step size = $\epsilon/10$. The CE loss is used to calculate the gradient during the attack, and other parameters are kept at default.

**Results.** This paper shows the final experimental comparison results in Table 2 and Table 3. Table 2 shows that super-fitting performs better for the gradient-based defense methods. It is theoretically possible to make the gradient vanish completely. Table 3 shows that the model is robust against CE-based white-box adversarial attacks on the CIFAR-10 and CIFAR-100 datasets, and the super-fitting method performs better than other methods.

The experiments in Table 2 are performed on CIFAR-10, and two network architectures are taken for comparison (MiddleCNN and ResNet-18). The teacher network and student network model architectures used in defense distillation are consistent, and the distillation temperature is set to 100. The experimental results show that our proposed method works better than defensive distillation. Also, super-fitting can completely defend against weak attack algorithms like FGSM and BIM-20 (adversarial accuracy equal to the clean one).

This paper compares the performance of super-fitting with other recent excellent defense methods on the test sets of CIFAR-10 and CIFAR-100 datasets (1280 images are randomly selected, and the batch size is set to 128). Meanwhile, this paper uses the original examples, the adversarial examples generated by PGD-100, APGD-100, and the default version of $A^3$ in nine network model architectures (three basic architectures of MiddleCNN, ResNet, Wide ResNet/WRN ) to evaluate the adversarial robustness of the models. The results are shown in Table 3, where the first column presents the defense methods, network architectures, and model parameters (the one marked with  denotes that $\epsilon=0.031$, and the models marked with  denote that they use an additional unlabeled dataset in the training phase). "Clean" presents the accuracy of the model under the original examples; PGD, APGD, and $A^3$ respectively present the accuracy of the models under adversarial examples generated by PGD-100, APGD-100, and the default version of $A^3$ algorithms. Obviously, the super-fitting method proposed in this paper can obtain higher adversarial robustness with smaller models (i.e., fewer parameters).

# Conclusion

This paper first introduces the super-fitting state in the model training process. In this state, the accuracy of a trained model under clean examples will decrease slightly, but its adversarial robustness will be greatly improved. Then, this paper analyzes the mathematical reasons that lead to the super-fitting phenomenon and theoretically proves that under the floating-point operation of the modern computer, super-fitting can perfectly handle CE-based white-box adversarial attacks (even future ones). Finally, for the problem that super-fitting requires a lot of iterative training, this paper proposes an optimization strategy called MUCS, which greatly reduces the number of iterations required for the model to reach the super-fitting state. Compared with about 50 recent defense models, the proposed defense method performs the best against CE-based white-box adversarial attacks.


author: S. S. Huh$^{1,2}$, J. J. Seo$^{3}$, B. S. Kim$^{1,2}$, S. H. Cho$^{1,2}$, J. K. Jung$^{1,2}$, S. Kim$^{4}$, Y. Y. Koh$^{5}$, C. I. Kwon$^{5,6}$, Jun Sung Kim$^{5,6}$, W. S. Kyung$^{7}$, J. D. Denlinger$^7$, Y. H. Kim$^8$, B. N. Chae$^8$, N. D. Kim$^8$, Y. K. Kim$^{4,*}$, C. Kim$^{1,2,*}$
title: Lifted electron pocket and reversed orbital occupancy imbalance in FeSe

**The FeSe nematic phase has been the focus of recent research on iron based superconductors (IBSs) due to its unique properties. A number of electronic structure studies were performed to find the origin of the phase. However, such attempts came out with conflicting results and caused additional controversies. Here, we report results from angle resolved photoemission and X-ray absorption spectroscopy studies on FeSe with detwinning by a piezo stack. We have fully resolved band dispersions with orbital characters near the Brillouin zone corner which reveals absence of a Fermi pocket at the Y point in the 1Fe Brillouin zone. In addition, the occupation imbalance between $d_{xz}$ and $d_{yz}$ orbitals is found to be opposite to that of iron pnictides, which is consistent with the identified band characters. These results settle down controversial issues in the FeSe nematic phase and shed light on the origin of nematic phases in IBSs.**

Nematic phase is a state with a broken rotational symmetry but with an intact translational symmetry. It has attracted renewed attention with a notion that it may provide an important clue to the mystery of unconventional superconductivity. Its region almost coincides with that of the superconducting region in the phase diagram for both cuprates and iron based superconductors (IBS) , implying its possible connection to the superconductivity. Finding the origin of nematic phases thus has been considered to be one of the most important goals in the research on unconventional superconductivity, especially for IBSs where studies on nematic phases were first initiated.

Recently, the nematic phase in FeSe has attracted attention due to its distinct properties from those of pnictide nematic phases. The most peculiar aspect is the absence of long range magnetic order which always coexists with orbital order in pnictide nematic phases . Furthermore, it was revealed that resistivity anisotropy for FeSe has the opposite sign compared to that of iron pnictides; resistivity along the longer $a$-axis is smaller than that along the shorter $b$-axis for iron pnictides while it is the other way around for FeSe. These two observations prompted the conjecture that the nematic phase in FeSe may be different from that of iron pnictides. Therefore, understanding the FeSe nematic phase may provide insight on universal understanding of the nematic phase in IBS. More importantly, considering the recent discovery of orbital selective correlation and superconductivity in FeSe in STM studies, superconductivity mechanism could be addressed by understanding the origin of the nematic phase because the nematic phase is widely believed to induce orbital selectivity.

In this regard, a number of angle resolved photoemission spectroscopy (ARPES) experiments have been performed to investigate the electronic structure of FeSe. However, interpretations from different experiments vary and caused more controversies on the origin of the nematic phase in FeSe. Various conflicting scenarios were proposed as the origin of the nematic phase based on ARPES results such as simple ferro-orbital ordering, d-wave orbital order, unidirectional nematic bond order and reversal sign ordering. Probable cause of such controversies is the lack of full and accurate electronic structure information which may be obtained only from fully detwinned single crystals. Therefore, the issue may be resolved only with full electronic structure information. Only then, the origin of the nematic phase in FeSe may be addressed.

Here, we present results of electronic structure studies by AREPS and X-ray linear dichroism (XLD) on fully detwinned FeSe in its nematic phase. Full detwinning was obtained by using a piezo stack based strain device, the first attempt to use such device in ARPES and XLD experiments (See Figs. 1A and 1B). Difficulty in using mechanical strain methods such as uncertainty of strain direction from 'accident detwin' by anisotropic shrink of epoxy has been overcome by using piezo bias on/off which provides strain on/off. As a result, we could investigate detailed band dispersions with full orbital characters (with ARPES) as well as the orbital occupancy imbalance between $d_{xz}$ and $d_{yz}$ orbitals (by XLD). Our ARPES results show that there are orbital dependent band shift and hybridization that lead to only one Fermi surface pocket in 1Fe BZ scheme, which may explain why there is no long-range magnetism. Furthermore, XLD reveals an unexpected reversed occupation imbalance between $d_{xz}$ and $d_{yz}$ orbital ($n_{xz}$ $<$ $n_{yz}$) which naturally explains the opposite resistivity anisotropy. These results resolve controversial issues on the nematic phase of FeSe and may shed light on the driving instability of nematic phases in IBS.

**Results** **Electronic structures of twinned and detwinned FeSe.** The key issue in the study of the FeSe electronic structure is if, in addition to an elliptical pocket, there is another large pocket which mainly consists of the $d_{xy}$ orbital around the Brillouin zone (BZ) corner. Therefore, we first focus on the overall Fermi surface topology. Fermi surface maps from twin and single domains (Figure 1c) indeed show clear differences, and may provide answers to the issue. For the twin domain case (left), two perpendicularly crossing elliptical pockets are observed at zone corner X/Y points. However, Fermi surfaces from a single domain sample (right) consists only of a single elliptical pocket at each zone corner. Note that, if there is a large pocket of $d_{xy}$ orbital character, it should be visible in our data as the experimental geometry allows the transition from the $d_{xy}$ initial state. Based on this experimental observation as well as the full band dispersion characterization along with its temperature dependence (discussed below), we conclude that there is only one elliptical pocket at zone corners as illustrated in Figure 1d. It immediately implies that, one of two pockets in the normal state should disappear across the nematic phase transition. That is, the pocket at the Y point in 1Fe BZ scheme shown in Figure 1e should disappear while the pocket at the X point remains.

**Dispersions and orbital characters of bands.** To proceed with the discussion on how and why the pocket at the Y point in 1Fe BZ disappears, band dispersions as well as orbital characters should be fully identified first. Figures 2a and 2b show overall band dispersions along the $\Gamma$-X and $\Gamma$-Y directions from a single domain sample in the normal and nematic states. Starting from isotropic normal state band dispersions, highly anisotropic band dispersions develop in the nematic phase. Especially, the band dispersions around the zone corner are dramatically renormalized and become those of two merged Dirac cones (possibly with small gaps at band crossing points) (see Figure 2b). The $\Gamma$-X ($k_{x}$-direction) data shows a sizable electron band and two split hole bands, while a tiny electron band near $E_{F}$ and two closely located hole bands at a higher binding energy are seen in the cut along the $\Gamma$-Y ($k_{y}$-direction). These observed dispersions are consistent with previous results . In order to understand the band structure more concretely, polarization dependent experiment is required to provide exact orbital characterization. The results of orbital characterization of bands are shown in Figures. 2c-2j. In particular, orbital characters of bands near the zone corner are newly interpreted. Along the $k_{x}$-direction, the large electron band consists mainly of $d_{xy}$ orbital while the two upper and lower hole bands have $d_{yz}$ and $d_{xy}$ characters. Along the ($k_{y}$-direction, the tiny electron band consists $d_{yz}$ orbital and hole bands have $d_{xz}$ and $d_{xy}$ characters (see Figures 2e and 2i). As for the zone center, the polarization dependence yields orbital characters of the hole bands that are consistent with previous results (See Figures 2f and 2j) .

The resulting full orbital characters of the bands are schematically summarized in Figures 2k and 2l. The result reveals two unique features of the FeSe electronic structure in the nematic phase. First of all, there is a reversal behavior in the relative energy positions of $d_{xz}$ and $d_{yz}$ hole bands. The $d_{xz}$ hole band at Y point is located below the $d_{yz}$ hole band at the X point (zone corners) while the $d_{xz}$ hole band is placed above the $d_{yz}$ band at the zone center. The other and more important feature is the reduced number of electron bands at X and Y points. In the normal state, there are two electron bands at both X and Y points - the common $d_{xy}$ electron band and $d_{xz}$ ($d_{yz}$) electron band for the X (Y) point. On the other hand, only one electron band exists at each point in the nematic phase: $d_{xy}$ ($d_{yz}$) at the X (Y) point. Other two electron bands - $d_{xz}$ and $d_{xy}$ at X and Y points, respectively - are not observed in any experimental geometry, i.e. polarization. Therefore, it is reasonable to speculate that those two bands do not exist below $E_{F}$, and that they should be pushed above $E_{F}$ across the nematic phase transition.

**Temperature evolution of electronic structure.** The temperature evolution of electronic structure across the nematic phase transition clearly shows that what we speculated is indeed the case. Figure 3a shows temperature dependence of band dispersion along $k_{x}$-direction at the Y point, with overlaid momentum distribution curves taken at 5meV above $E_{F}$. In the normal state, $d_{xz}$ electron band crosses $E_{F}$ and peak positions in the curve have finite momentum values (indicated with red arrows). As the temperature decreases, peak positions for the $d_{xz}$ electron band shift towards the Y point. This indicates that the $d_{xz}$ electron band shifts upward and is finally pushed above $E_{F}$. Meanwhile, the $d_{yz}$ ($d_{xz}$) hole band around the X (Y) point shifts upward (downward) as the temperature decreases, as seen in temperature dependent spectra at X and Y points (Figures. 3b and 3c). The splitting at a low temperature is about 60 meV and remains finite above the nematic phase transition temperature, possibly due to the applied strain (Figure 3c). It is noteworthy that the two hole bands do not cross $E_{F}$ and remain under $E_{F}$ after the shift.

Now we are ready to discuss how the electron pocket at Y disappears in 1Fe BZ, that is, how the number of electron bands is reduced across the nematic phase transition. We argue that the observed electronic structure evolution can be explained within the picture of orbital dependent band shift (or splitting) and hybridization. Let us look at the case for the zone corner in 1Fe BZ scheme. Starting from symmetric bands in the normal state (Figure 3d left), the $d_{yz}$ ($d_{xz}$) band shifts upward (downward) as the temperature goes down below $T_{S}$ (Figure 3d middle), which we call 'nematic band shift'. Then, the orbital dependent hybridization comes in. For the $d_{yz}$ band case, there is only weak mixing with $d_{xy}$ and thus the overall dispersion remains intact. The $d_{xz}$ band, on the other hand, strongly hybridizes with $d_{xy}$ band, pushing the hole band down below $E_{F}$ while lifting both $d_{xz}$ and $d_{xy}$ electron bands above $E_{F}$ (Figure 3d right). This, results in vanishing electron pocket at Y in the 1Fe BZ scheme.

**Reversed orbital occupancy imbalance.** A surprising implication of the above interpretation is that $d_{xz}$ orbital should be less occupied than $d_{yz}$, opposite to the iron pnictide case and also to the prediction of ferro-orbital order scenario . At the zone center, the energy position of the $d_{xz}$ band is higher than that of $d_{yz}$, and thus $d_{xz}$ state is less occupied. Disappearance of the $d_{xz}$ electron band at the zone corner also leads to less occupied $d_{xz}$ orbital ($d_{xz}$ and $d_{yz}$ hole bands at the zone corner are irrelevant as both of them are fully occupied). In order to obtain direct information on such anomalous orbital occupancy, we performed XLD measurements on detwinned samples. XLD measurement, a local probe for orbital selective density of states, can provide direct proof of the imbalance in the orbital occupancy .

Figure 4a shows the experimental geometry with two light polarizations, parallel and perpendicular to the strain direction. Fe $L$-edge absorption spectra from detwinned FeSe taken with the two light polarizations at 10 K are plotted in Figure 4b. With the given experimental geometry, the linear dichroism or XLD shown with the black solid line in Figure 4b should reflect the imbalance in $d_{xz}$ and $d_{yz}$ orbital occupancy. A complication is that not only the orbital occupation imbalance but also the orthorhombic structural distortion is known to contribute to the XLD signal . It was previously shown that the two contributions can be separated by considering their distinct behaviors in the temperature dependence . A close inspection of the temperature dependent XLD data in Figure 4c reveals that XLD signal starts to appear below $T_{S}$ and monotonically increases as the temperature decreases down to 10 K. Such monotonic increase of XLD can be more clearly visualized by plotting the integrated XLD (see the figure caption for the definition) as a function of temperature (Figure 4d). If the XLD contains only the structure contribution, it should immediately saturate below $T_{S}$ as the structure contribution should follow the orthorhombicity of the crystal (see the overlaid diffraction data) . Therefore, the non saturating increase far below $T_{S}$ indicates that XLD does contain contribution from orbital imbalance. An important point to note is that the orbital contribution is positive. Positive XLD from orbital means that $d_{xz}$ orbital is less occupied ($n_{xz}$ $<$ $n_{yz}$), contrary to the case of simple ferro-orbital order scenario and to the iron pnictide case ($n_{xz}$ $>$ $n_{yz}$) (Figure 4e).

**Discussion** As our interpretation of the electronic structure evolution across the nematic phase transition in FeSe is confirmed by observation of the reversed orbital occupancy imbalance, its implication may be discussed. First of all, the sign reversal in the hole band splitting and the reduced number of electron bands in our interpretation do not support the unidirectional nematic bond order scenario which requires absence of hole band splitting. On the other hand, d-wave form splitting and sign reversal order scenario are partially consistent with our result when the hole band splitting is considered. From these results, we learn that the evolution of electron band dispersions as well as the role of the $d_{xy}$ band are taken into account to obtain a fully consistent picture, that is to say, full understanding of the nematic phase.

With the concrete understanding of the electronic structure, the unique properties of FeSe nematic phase - absence of magnetism and opposite resistivity anisotropy - can also be understood. The absence of the magnetism can be explained for both weak and strong coupling pictures. In the weak coupling picture, a weak Fermi surface nesting condition stemming from opposite orbital characters of Fermi surface pockets could explain the absence of magnetism; the inter-orbital nesting between Fermi surfaces at the zone center (mostly $d_{xz}$ with small contribution from $d_{yz}$) and corner (mostly $d_{yz}$), which is considered to be the source of the magnetism in the weak coupling picture, is mostly suppressed due to the opposite orbital characters. In the strong coupling picture, the absence of the magnetism could be explained within an orbital weight redistribution scheme. A strong hybridization between $d_{xy}$ and $d_{xz}$ tends to open a gap near the $E_{F}$, resulting in suppression of the $d_{xy}$ and $d_{xz}$ orbital weight. Since the magnetic moment dominantly comes from the $d_{xy}$ orbital , the reduced $d_{xy}$ orbital weight should weaken the magnetic instability. Meanwhile, the opposite resistivity anisotropy can be easily explained within the observed reversed orbital occupancy imbalance if we simply follow the argument that orbital occupation imbalance characterizes the resistivity anisotropy .

Another important implication of our work is that it provides a new perspective on the origin of the nematic phase. It has been believed that the nematic band shift and the occupancy imbalance between $d_{xz}$ and $d_{yz}$ orbitals are equivalent. Therefore, they represent a single phenomenon of the ferro-orbital order stemming from symmetry breaking at the atomic level. Our observation of the reversed orbital occupancy imbalance, an occupancy imbalance opposite to the nematic band shift, clearly shows that they are not equivalent. Furthermore, their opposite behaviors strongly suggest that the nematic band shift is neither a manifestation of nor generated by the development of occupancy imbalance. The occupancy imbalance should rather be a by-product of the nematic band shift as well as the role of $d_{xy}$ orbital discussed above. In short, our results indicate that the ferro-orbital order is unlikely the driving instability of the nematic phase in IBSs. Instead, the instability responsible for the system independent nematic band shift should be the true driver of the nematic phase. It could be an instability with a spin origin or still an orbital origin but in a different form. In any case, our findings provide crucial information on the nematic phase origin issue by eliminating the ferro-orbital order from the candidate list. It should further shed light on the origin of the nematic phase in IBSs.

**Methods** **Experiments** XAS experiments were performed at the beam line 2A of the Pohang Light Source and spectra were recorded in the TEY mode. All spectra were normalized by the incident photon flux intensity measured from a gold mesh and calibrated with respect to $L_{3}$ absorption peak of Fe$_2$O$_3$ alloy located in front of the analysis chamber. ARPES measurements were carried out at the Beamline 4.0.3 of the Advanced Light Source. Linearly polarized light with the photon energy of 56 eV was used. All crystals were cleaved in situ in a pressure better than 9$\times$`<!-- -->`{=html}10$^{-10}$ Torr for XAS and 4$\times$`<!-- -->`{=html}10$^{-11}$ Torr for ARPES experiments. Samples were detwinned using uniaxial strain which is applied along the tetragonal \[110\] direction. The best quality data used in the figures were obtained from accidentally detwinned domain.

<embed src="Fig1.pdf" />

<embed src="Fig3.pdf" />

<embed src="Fig4.pdf" />