abstract: Sleep scoring is a necessary and time-consuming task in sleep studies. In animal models (such as mice) or in humans, automating this tedious process promises to facilitate long-term studies and to promote sleep biology as a data-driven field. We introduce a deep neural network model that is able to predict different states of consciousness (Wake, Non-REM, REM) in mice from EEG and EMG recordings with excellent scoring results for out-of-sample data. Predictions are made on epochs of 4 seconds length, and epochs are classified as artifact-free or not. The model architecture draws on recent advances in deep learning and in convolutional neural networks research. In contrast to previous approaches towards automated sleep scoring, our model does not rely on manually defined features of the data but learns predictive features automatically. We expect deep learning models like ours to become widely applied in different fields, automating many repetitive cognitive tasks that were previously difficult to tackle.
author: Justus T. C. Schwabedal, Daniel Sippel, Moritz D. Brandt, Stephan Bialonski [^1] [^2]
title: Automated Classification of Sleep Stages and EEG Artifacts in Mice with Deep Learning

# Introduction

Humans and other animals spend much of their time in sleep. Despite its importance, many aspects of sleep are not yet fully understood. In order to uncover mechanisms and functions of sleep, scientists study humans and animal models. Rodents (mice and rats) are often studied as these are well established mammal models which show sleep characteristics that are comparable to human sleep (i.e., NREM and REM sleep). An important step in many studies is the identification of different states of consciousness from different recording modalities such as electroencephalography (EEG) and electromyography (EMG). For many years, sleep scoring has been a manual and time consuming process. The time required for manual sleep scoring depends on various factors (i.e., data quality, artifacts, sleep fragmentation). While experienced personnel needs up to 3 hours to thoroughly score 24 hours of EEG/EMG recordings, untrained staff may spend up to 6 hours to score the same amount of data. This tedious process poses not only a challenge to human scorers who often face the problem of decreasing scoring accurracies as concentration fades during such monotonous tasks. It also poses a challenge for studies to achieve a suitable statistics (with a reasonable number of mice scored) and to establish long-term studies in which potentially several days and weeks of sleep recordings need to be scored.

Early approaches towards automating the sleep scoring process date back to the late 1960s . Since then, the field has made rapid progress over the years (see  for a historic overview), yielding methods that aim at interactively supporting human scorers or even fully automating the scoring process. Most methods rely on hand-crafted features (feature engineering), many of which are derived from power spectra of the EEG (see, e.g., Refs. ) where researchers focused on classical frequency bands in rodents (delta, theta, and sigma bands). Other approaches include features derived from bispectra , or wavelet coefficients (see, e.g., Ref. ). While feature engineering is often the only feasible way to create predictive models in cases in which data is not abundant, this process relies on expert knowledge and may not yield features that are optimal for the sleep scoring task at hand. However, if data is abundant, feature engineering can in many cases be replaced by automated feature learning, which can lead to better prediction results as has been shown in the context of deep learning in various disciplines in recent years .

We introduce a deep neural network model that is trained end-to-end to predict different states of consciousness in mice from EEG and EMG recordings (see Fig. ). Instead of engineering features, features are learnt automatically by the model. The model architecture (see Fig. ) and the training of the model draws on recent advances in convolutional neural network research which has demonstrated the usefulness of learnt features for time sequenced data in a number of applications (e.g., generative modeling of audio data , machine translation ). Our model achieves excellent scoring results for out-of-sample data and in artifact-free conditions. Furthermore, we demonstrate that increasing the number of recording channels can partially counteract the decrease of scoring performance for artifact-contaminated data.

# Methods

## Data

The data was recorded from 22 mice (age 10-11 weeks, male, C57BL/6 strain). All mice were kept at the animal facility of the Center for Regenerative Therapies of the Technische Universität Dresden, Germany. All applicable local and federal regulations of animal welfare in research were followed. The experiments were approved by the responsible authority, Landesdirektion Sachsen, Germany. The mice were chronically implanted with a synchronous recording system of 1 EMG and 2 EEG electrodes (8201, Pinnacle Technology Inc., Lawrence, KS) 10-14 days before the first recording. Four stainless steel screws sitting on top of the cortex at the following coordinates (relative to bregma) were used as EEG electrodes: anterior-posterior (AP): +2 mm, medial-lateral (ML): +1.5 mm (EEG2) / ML: -1.5 mm (ground) and AP: -4 mm, ML: +1.5 mm (EEG1) / ML: -1.5 mm (reference). With this setup, we sampled the electrical activity of the frontal (EEG2) and the parieto-occipital lobes (EEG1), respectively. Two flexible stainless steel wires were inserted into the neck muscles to measure the EMG signal. All electroencephalographic (EEG) and electromyographic (EMG) time series were sampled at 400 Hz. 82 recordings, each lasting 24 hours and containing all three channels, were available for the creation of training, validation and test sets.

All recordings were divided into non-overlapping consecutive *epochs* of 4 seconds. One of the authors (DS) scored each epoch manually as sleep stage "Non REM", "REM" or "Wake". In addition, each epoch was scored according to whether recording artifacts were present (artifact-contaminated epoch) or not (artifact-free epoch). Manual scoring was performed using our own customized software *edfView* that was configured to show $5$ consecutive epochs of all three channels, and to allow our scoring specialist to assign one of the labels to the middle epoch: The Wake state is characterized by high EMG activity (active locomotion, higher muscle tone even when resting) and a mixed frequency EEG signal (without a clear peak at a certain frequency in the power spectrum). Non REM sleep shows lower EMG activity compared to the Wake state as well as a power spectrum peak in the delta band (0.5–4 Hz). REM sleep is usually characterized by very low EMG activity (with only seldom muscle twiches and a clear power spectrum peak in the theta band (at 7–8 Hz)).

Time series were low-pass filtered with a 4th-order Butterworth filter (cutoff frequency: 25.6 Hz) using a forward-backward scheme to ensure zero-phase filtering . The filtered time series were downsampled from 400 Hz to 64 Hz by linearly interpolating between neighboring sampling points. Each of the 82 recordings was divided into different parts, and segments (epochs) of these parts were assigned to the training, test, and validation set, respectively (cf. Tab. ). We made sure that data of the test set came from different mice.

Table  summarizes the number of labeled epochs for the whole dataset. The most frequent class was "Wake" ($57\%$), followed by "Non REM" ($33\%$) and "REM" ($5.9\%$) sleep. Artifacts were most pronounced during Wake ($3.6\%$) while they were barely present during REM sleep ($0.04\%$). Such large class imbalances are known to negatively affect training results. The large variability in the frequencies of class members can adversely affect the model fitting process. We accounted for the class imbalance by resampling with replacement and used the resulting *rebalanced training set* for training our models. Indeed, we observed better fitting results with this rebalanced training set, and the sampling probabilities (class percentages) used to produce the rebalanced training set are reported in table .

|                  | training set | test set | validation set |  all data |     |
|-----------------:|-------------:|---------:|---------------:|----------:|----:|
|   recording days |           65 |        4 |             13 |        82 |     |
| number of epochs |    1,407,706 |   84,694 |        275,599 | 1,767,999 |     |
|       percentage |       79.6 % |    4.8 % |         15.6 % |     100 % |     |

**Characteristics of the data set.**

Statistics of labeled epochs of the whole dataset ("all data"), of the training set and the rebalanced training set (see text). NR: "Non REM sleep", R: "REM sleep", W: "Wakefulness". (A): artifact-contaminated. perc: percentage, (reb.) train. set: (rebalanced) training set, # epochs: number of epochs.

label
NR
NR (A)
R
R (A)
W
W (A)

all data

# epochs
586,832
1,973
103,546
672
1,010,412
64,564

perc.
33.19 %
0.11 %
5.86 %
0.04 %
57.15 %
3.65 %

train. set

# epochs
465,134
1,290
82,646
429
807,419
50,788

perc.
33.04 %
0.09 %
5.87 %
0.03 %
57.36 %
3.61 %

reb. train. set

# epochs
305,061
165,145
188,307
164,293
406,125
178,775

perc.
21.67 %
11.73 %
13.38 %
11.67 %
28.85 %
12.70 %

## Machine-Learning Model architecture

To closely imitate the scoring procedure (cf. section ), our learning model uses information of $5$ consecutive time series epochs in order to classify the middle epoch to belong to one of three distinct sleep state classes and to decide whether the middle epoch contains artifacts or not. More formally, our learning model is a function $F$ that maps $5$ consecutive epochs of multivariate time series $\vec{x}_i$, $i\in\{1,\ldots,5\}$, to the vector $\vec{p}_\text{state}$ and to a scalar $p_\text{artifact}$. Values of $p_\text{artifact}$ larger than a threshold value of $0.5$ indicate the middle epoch $\vec{x}_{3}$ to contain artifacts. The entries of $\vec{p}_\text{state}$ are interpreted as probabilities of the middle epoch $\vec{x}_{3}$ to reflect wakefulness (Wake), REM, or non-REM sleep. The model identifies the maximum component of $\vec{p}_\text{state}$ and outputs its associated class, thereby producing a label of the most probable sleep state.

The model architecture (cf. fig. ) is inspired by recent advances in deep learning and consists of two parts: (1) A feature extractor successively downsamples and nonlinearly transforms multivariate time series to create a set of features that are used by (2) the classifier to classify the time series. The feature extractor consists of 8 convolutional layers without padding where each layer has 64 kernels of size $5$. Kernels (also called filters) are shifted with a stride of 1 in layers 1, 3, 5, and 7. The other convolutional layers use a stride of 2 and thus successively downsample the nonlinearly transformed data. We used Rectified Linear Units (ReLUs) as nonlinearities after each convolutional layer since ReLUs were observed to show superior training behavior in previous studies . To stabilize training, we reduced internal covariate shift by applying Batch Normalization (BN) after the nonlinearities of each convolutional layer[^3]. In addition, we also applied Batch Normalization to the input data (i.e., before the first convolutional layer). The batch normalized 64 feature maps after the last convolutional layer are flattened and concatenated to form a single feature vector which is the output of the feature extractor.

The classifier consists of two fully connected (fc) layers, fc1 and fc2. fc1 is composed of 80 neurons with ReLU nonlinearities and processes the feature vector produced by the feature extractor. fc2 consists of 4 neurons which process the output activities of fc1. Instead of ReLU nonlinearities, we apply a softmax on the output of the first three neurons, yielding the probabilities $p_j$, $j\in\{1,\ldots,3\}$ of the input epoch to belong to the three classes. The output of the fourth neuron is transformed into a value $p_a$ by a sigmoid function, which maps values into the interval $[0,1]$. We interpret the first three components of the resulting output vector as class probabilities while the last component shall indicate whether the input epoch contains artifacts (1) or not (0). Since fully connected layers contain many free parameters that make overfitting more likely, we employed dropout regularization (dropout probability: $0.2$) on the weights connecting the feature vector to the neurons of fc1 and on the weights connecting the output of fc1 to the neurons of fc2.

We implement the training objective by a loss function $\mathscr{L}=L_1+L_2$ that becomes small ($L_1$) if the classifier assigns a large probability to the correct class and becomes small ($L_2$) if the classifier correctly indicates whether the input contains artifacts or not. The loss is computed over a minibatch of the input. Let $c_k\in\{1, 2, 3\}$ denote the correct class index for the middle input epoch $\vec{x}_k$ and let $p_{k,c_k}$ be the predicted class probability of $\vec{x}_k$ for the correct class $c_k$. We define $L_1$ as a negative log-likelihood, $$L_1 = - \frac{1}{N_b} \sum_{k=1}^{N_b} \log(p_{k,c_k})$$ where $N_b$ is the minibatch size. The second term $L_2$ is defined as the binary cross entropy between the predicted value $p_{k,a}$ for an input $\vec{x}_k$ and the true label $t_{k,a}\in\{0,1\}$ indicating whether an artifact is present (1) or not (0), $$L_2 = \frac{1}{N_b} \sum_{k=1}^{N_b} \left(-t_{k,a}\log{p_{k,a}} + (1-t_{k,a})\log{(1-p_{k,a})}\right).$$ The model is differentiable and can be trained via gradient descent.

## Training

The model was trained by RMSprop (Root Mean Square Propagation), a variant of stochastic gradient descent where the learning rate is adapted separately for each free parameter (weight) of the model . Let $w(t)$ denote a weight and let $\Delta(t) = \frac{\partial \mathscr{L}}{\partial w}$ denote the gradient of the loss function with respect to the weight obtained for the minibatch at step $t$ of the training. We update the weight by $$w(t+1) = w(t) - \frac{\eta}{\sqrt{R(t)}} \Delta(t)$$ where $\eta$ is the learning rate. $R$ is a running mean of the squared magnitudes of recent gradients for that weight, $$R(t+1) = \alpha R(t) + (1-\alpha) \Delta(t)^2,$$ where we set the smoothing factor $\alpha$ to $\alpha=0.99$. After the gradients for all weights are determined, we checked whether a gradient exceeded the value $0.1$. In such a case, we rescaled all gradients such that the largest value was $0.1$ (maximum-norm normalization of gradients).

We adjusted the learning rate according to the following learning protocol: during model exploration (section ), the learning rate $\eta$ was linearly increased from $0$ to $\eta^*=0.00128 N_b$ (warm-up period) in the first $5$ training cycles, where $N_b$ denotes the size of the minibatches. This linear scaling of the learning rate with minibatch size combined with a warm-up period was observed to lead to better generalization properties of models when trained with large minibatch sizes . A single training cycle was completed when all minibatches of the training set were used once for a stochastic gradient descent step. During the following $5$ training cycles, the learning rate was linearly decreased again to $0$ (cool-down period). For the training of the final models (section ), we linearly increased the learning rate from $0$ to $\eta^*$ in the first 5 training cycles. During the subsequent 10 training cycles, the learning rate was $\eta^*$, while during the next 5 training cycles we linearly decreased $\eta$ again from $\eta^*$ to $0$. All trainings were carried out with a minibatch size of $N_b=256$, and we did not observe moderate changes in minibatch sizes to affect training outcomes.

## Evaluation.

To evaluate the prediction performance of our model, we employed classical techniques to characterize the quality of classifications in binary as well as in multiclass classification settings.

#### Artifact classification

We used the *$F_1$ score* (also known as *$F$ measure*, ) to quantify the prediction performance of our model to distinguish between artifact-free and artifact-contaminated epochs (binary classification) in a given sleep stage. The $F_1$ score is a summary statistics which takes on values between $0$ and $1$, where larger values indicate better predictive performance. It is defined as the harmonic mean of *precision* ($p_\text{precision}$) and *recall* ($p_\text{recall}$), $$F_1 = \left( \frac{1}{\ensuremath{p_\text{precision}}} + \frac{1}{\ensuremath{p_\text{recall}}}\right)^{-1},$$ where precision is also known as *positive predictive value*. In a binary classification setting, the precision is the number of epochs for which a class was correctly predicted by the model (number of true positives) divided by the total number of epochs that were predicted by the model to belong to that class, $$\ensuremath{p_\text{precision}}= \frac{\text{number of true positives}}{\text{number of true and false positives }}.$$ The recall, also known as *sensitivity*, is the number of epochs for which a class was correctly predicted by the model divided by the total number of epochs of that class, $$\ensuremath{p_\text{recall}}= \frac{\text{number of true positives}}{\text{number of true positives and false negatives}}.$$ Both, precision and recall, take on values between $0$ and $1$.

```latex
\begin{tabular}{c|r|c|r|r|r|r|r|r|}
 \multirow{2}{*}{\makecell{prediction \\ target}} & & & \multicolumn{3}{c|}{96 k} & \multicolumn{3}{c|}{$4$ l} \\
 & condition & set & 4 l & 8 l & 10 l & 32 k & 64 k & 96 k\\\hhline{=========}
 \multirow{4}{*}{\makecell{sleep \\ stages}}
 &no artifact  & training   & 0.95 & 0.98 & 0.98 & 0.92 & 0.94 & 0.95 \\
 &             & validation & 0.93 & \cellcolor[gray]{0.90} 0.95 & \cellcolor[gray]{0.90} 0.95 & 0.91 & 0.93 & 0.93 \\\hhline{~--------}
 &artifact     & training   & 0.98 & 1.00 & 1.00 & 0.93 & 0.97 & 0.98 \\
 &         & validation & \cellcolor[gray]{0.8} 0.88 & \cellcolor[gray]{0.8} 0.85 & \cellcolor[gray]{0.8} 0.82 & 0.85 & 0.88 & 0.88 \\\hhline{=========}
 \multirow{6}{*}{\makecell{artifact \\ yes/no}}
 & Non-REM     & training & 0.94 & 0.94 & 0.96 & 0.70 & 0.88 & 0.94\\
       &       & validation & 0.60 & 0.80 & 0.81 & 0.48 & 0.56 & 0.81\\\hhline{~--------}
    & REM      & training   & 1.00 & 1.00 & 1.00 & 0.84 & 0.97 & 0.99\\
        &      & validation & 0.60 & 0.76 & 0.76 & \cellcolor[gray]{0.8} 0.53 & \cellcolor[gray]{0.8} 0.63 & \cellcolor[gray]{0.8} 0.58\\\hhline{~--------}
 & Wake & training   & 0.89 & 0.95 & 0.96 & 0.82 & 0.87 & 0.89\\
&          & validation & 0.85 & 0.86 & 0.87 & 0.81 & 0.83 & 0.84\\\hhline{---------}
\end{tabular}
```

#### Sleep-stage classification

We quantified the prediction performance of our model to distinguish between different sleep stages (multiclass classification) by calculating averages of $F_1$ scores. More precisely, we converted the multiclass classification problem into multiple binary classifications: We determined the $F_1$ score for each class $c$ (sleep stage) separately ($F_1^c$) where precision and recall were calculated with respect to the distinction between class $c$ and everything else ($\neg c$). The resulting $F_1$ scores were then averaged to arrive at a final score, $$\bar{F}_1 = \frac{1}{N_c} \sum_{c=1}^{N_c} F_1^c,$$ where $N_c$ denotes the number of classes ($N_c = 3$ sleep stages). The $\bar{F}_1$ score varies between $0$ and $1$ where large values indicate better predictive performance. Next to the $\bar{F}_1$ score, we also determined confusion matrices in order to investigate whether our models systematically confused one class with another.

# Results

We investigated our learning model with respect to its ability (i) to accurately predict sleep stages and (ii) to detect artifacts. In a first series of experiments, we investigated to what extent changes of our basic model architecture (as shown in figure ) affected prediction performance. In this *model exploration* step, we evaluated the performance of different model architectures on the validation set (see section ). Based on these results, we chose a final model architecture. In a second series of experiments (see section ), we addressed the question whether the predictive performance changes when varying the number of input channels presented to the model. Here we evaluated the performance of our model on the test set.

## Model exploration

We varied our model architecture (cf. figure ) and investigated which of the resulting architectures yielded the best prediction performance. We varied (i) the number of convolutional layers and (ii) the number of kernels per convolutional layer. The input for all models consisted of three input time series (EEG1, EEG2, EMG) that were simultaneously recorded, and each model was trained as described in section .

When increasing the number of layers or the number of kernels, we increase the complexity of the model and thus its ability to fit the training data. Indeed, we observed $F_1$ scores obtained on the training set (see table ) to increase with an increasing number of layers and number of kernels per layer. Best $\bar{F}_1$ scores were obtained for the architecture with largest model complexity (10 convolutional layers with 96 kernels each) which reach $0.98$ and $1.00$ for sleep stage classification from artifact-free and artifact-contaminated epochs, respectively. For the same model architecture in the artifact-free condition, our model generalized well to unseen epochs as indicated by an $\bar{F}_1$ score of $0.95$ for the validation set. Due to this high value, we expect our model to very reliably predict sleep stages on unseen and artifact-free epochs.

Our models were able to fit artifact-contaminated time series in the training set, reaching $\bar{F}_1$ scores larger than $0.92$ for all model architectures (cf. table ). However, as compared to the artifact-free case, our models were not able to generalize as well, which was indicated by lower $\bar{F}_1$ scores between $0.82$ and $0.88$ on the validation set. This phenomenon may be related to class imbalances of the training set, where artifact-free epochs were much more frequent than artifact-contaminated epochs (cf. table ). We observed the same phenomenon, namely less generalization performance, for the prediction of artifacts under the condition of non-REM, REM, and Wake stages (cf. table , last three rows).

We noticed tendencies of slight overfitting (decreasing $\bar{F}_1$ scores for increasing model complexity for the validation set) when predicting sleep stages from artifact-contaminated epochs and increasing the number of layers (see dark gray cells in table ). This tendency was also observed for the prediction of artifacts from epochs recorded during REM sleep and increasing the number of kernels. While we do not consider these cases of overfitting to be severe, we chose our final model architecture, $8$ layers and $96$ kernels, as a compromise between high $\bar{F}_1$ scores for sleep stage prediction (see light gray cells in table ) and reduced overfitting.

```latex
\begin{threeparttable}
\caption{\textbf{Model performance.} $\bar{F}_1$ (top two rows) and $F_1$ (last three rows); ic: input channels; grey cells: largest $\bar{F}_1$ scores for predicting sleep stages in artifact-free and artifact-contaminated epochs in the test set.\tnote{1}}
\tabcolsep=8.0pt
\begin{tabular}{c|r|c|r|r|r|}
 \multirow{2}{*}{\makecell{prediction \\ target}} & & & \multicolumn{3}{c|}{96 kernels, 8 layers}\\
 & condition & set & 1 ic & 2 ic & 3 ic\\\hhline{======}
 \multirow{6}{*}{\makecell{sleep \\ stages}}
 &no artifact  & training   & 0.95 & 0.97 & 0.98 \\
 &             & testing    & 0.93 & 0.95 & \cellcolor[gray]{0.8} 0.96 \\
 &             & validation & 0.92 & 0.94 & 0.95 \\\hhline{~-----}
 &artifact     & training   & 1.00 & 1.00 & 1.00 \\
 &             & testing    & 0.82 & 0.92 & \cellcolor[gray]{0.8} 0.93 \\
  &            & validation & 0.77 & 0.85 & 0.84  \\\hhline{======}
 \multirow{9}{*}{\makecell{artifact \\ yes/no}}
 & Non-REM     & training   & 0.87 & 0.91 & 0.95 \\
&              & testing    & 0.12 & 0.15 & 0.17 \\
 &             & validation & 0.68 & 0.68 & 0.81\\\hhline{~-----}
    & REM      & training   & 0.99 & 1.00 & 1.00\\
 &             & testing    & 0.40 & 0.36 & 0.56 \\
        &      & validation & 0.68 & 0.65 & 0.73 \\\hhline{~-----}
 & Wake & training   & 0.92 & 0.94 & 0.96 \\
 &             & testing    & 0.74 & 0.75 & 0.83 \\
 &             & validation & 0.79 & 0.80 & 0.87 \\\hhline{------}
\end{tabular}
 \label{tab:model_performance}
 \begin{tablenotes}
  \item[1]  Small differences between values reported in table~\ref{tab:model_exploration} and in table~\ref{tab:model_performance} are due to a retraining of the models and the stochastic nature of the optimization algorithm (stochastic gradient descent).
 \end{tablenotes}
 \end{threeparttable}
```

## Model performance

We studied the predictive performance of the final model and whether and how it changed when predictions were based on a different number of simultaneously recorded time series (input channels). This question becomes important when experimental restrictions and setups do not allow for all three channels to be recorded. A reliable prediction of sleep stages and artifacts based on few channels is highly desirable in such cases.

We varied the number of input channels from one channel ("EEG2"), to two ("EEG2, EMG"), and three ("EEG1, EEG2, EMG") and trained one final model for each of the three cases using the final architecture (i.e., 8 convolutional layers with 96 kernels each) as determined in the previous section. $\bar{F}_1$ scores for sleep stage prediction and $F_1$ scores for artifact prediction obtained for all datasets are shown in table . We observed $\bar{F}_1$ scores and $F_1$ scores obtained on the training set to increase with increasing the number of input channels, indicating our models to better predict sleep stages or artifacts the more input channels were available. This tendency was also observed for the test set and, as such, for epochs not used for training. We obtained best $\bar{F}_1$ scores for sleep stage prediction of $0.96$ and $0.93$ (grey cells in table ) for the three channels condition and for artifact-free and artifact-contaminated epochs, respectively. More importantly, $\bar{F}_1$ scores obtained on the test set only slightly decreased when decreasing the number of input channels. This indicates that our model is able to reliably predict sleep stages even in experimental setups in which only one channel is recorded.

$F_1$ scores for predicting artifacts obtained low values on the test set, e.g. $0.12$ for one input channel and artifact prediction in Non-REM epochs. We hypothesized that these low values are related to the small size of the test set that only encompasses $4.8\,\%$ of the labeled total data (cf. table ). Since the test set is small, it has only few artifact-contaminated epochs that are, moreover, assigned to different sleep stages. Due to this difficult statistical situation, we also reported $F_1$ scores(cf. table ) obtained on the validation set (which was larger than the test). Indeed, for the validation set and the prediction of artifacts, we observed $F_1$ scores which are close to those reported in the previous section.

To offer insight into the prediction of individual labels, we provide confusion matrices (figure ) grouped for artifact-contaminated and artifact-free epochs. For three input channels and for artifact-free epochs, our final model correctly classified 95.4 % of all "REM" epochs, 96.5 % of all "Non REM" epochs, and $97.1\,\%$ of all "Wake" epochs. These numbers only slightly decrease when reducing the number of input channels, where we obtained $91.6\,\%$ as the lowest fraction of correct predictions for artifact-free REM epochs (cf. figure ). As expected, prediction performance worsened for artifact-contaminated epochs, where our model showed tendencies to confuse "Non REM" or "REM" sleep stages with "Wake". This confusion was most pronounced when using a single input channel ($27.5\,\%$ of "REM" epochs were misclassified as "Wake"). Increasing the number of input channels, however, led to more robust predictions in artifact-contaminated epochs, i.e. to a larger fraction of correctly predicted sleep stages.

# Discussion

We introduced a deep neural network model that predicts different states of consciousness in mice from EEG and EMG recordings. Unlike many previous approaches towards automated sleep scoring, our model does not rely on manually defined features (such as power in frequency bands). Instead, our model was trained end-to-end, thereby automatically learning features of the data that were successfully used by the classifier of the model to distinguish between different sleep stages. While many previous approaches predicted sleep stages on epochs of 10 seconds length, our model allows for higher time resolution and was able to make reliable predictions for epochs of 4 seconds length. Such time resolutions are important to capture short arousals and frequent changes in sleep states that are typical in mice. High time resolutions particularly call for approaches towards automatizing potentially time-consuming manual scoring procedures.

We observed our model to achieve high specificity and sensitivity (as indicated by large $F_1$ scores) on artifact-free out-of-sample (test) data. Decreasing the number of input channels that were available for our model to infer sleep stages led only to a slight decrease of prediction performance for artifact-free data. This indicates that our model will yield good prediction performance even for experiments in which constraints prohibited to simultaneously capture multiple recording modalities. For artifact-contaminated data, $F_1$ scores became better when the model based its predictions on more input channels. This observation agreed with our expectation that recordings will likely contain partially redundant information which renders inferring sleep classes from artifact-contaminating epochs easier if other recording channels are available. Nevertheless, prediction performance of our model was decreased for artifact-contaminated data when compared to artifact-free data. We speculate that the deteriorating effect of artifacts may also be related to our training set that contained much more artifact-free than artifact-contaminated epochs. We addressed these class imbalances by sampling with replacement but expect larger training sets to allow future models to even further increase their prediction performance on artifact-contaminated data.

In our study, we assessed prediction performance on test data which was created from 22, male, 10-11 week old, genetically identical C57BL/6 mice with a fixed electrode placement scheme. This restriction was due to experimental constraints when creating the data set and limits our ability to draw conclusions on prediction performance for other mice strains and electrode placements. However, we are confident that our deep learning model can be successfully employed to predict sleep stages in other mice strains, models and/or other implantation schemes after retraining. Such a retraining could substantially profit from transfer learning techniques  which have the potential to significally reduce the amount of training data required. Next to the aforementioned restriction, we also did not assess inter-rater and intrarater variability, i.e. the scoring variability between different persons scoring our data and the same person scoring the same data multiple times, respectively. Such assessments become increasingly challenging as the amount of data to score increases. We speculate, however, that intrarater variability sets an upper limit to the maximum achievable sensitivity and specificity of our model. Also, previous studies observed inter-rater reliability to be never 100 % but rather in the range of 83–96 % . Future studies on model architectures like the one introduced here that assess rater variability will be able to relate scoring accuracies achieved by models to those achieved by human scorers.

We expect models like ours (a TensorFlow.js implementation is available online ) to facilitate or even enable long-term studies in sleep research. Automating the time-consuming process of sleep scoring will contribute to support sleep research as a data-driven field and will help researchers and lab personnel to become more productive. This is in line with what we expect to happen in the early 21st century: The automation of cognitive repetitive tasks. In this context, deep learning models and automated feature learning will play an important role.

[^1]: Justus T. C. Schwabedal, Department of Biomedical Informatics, Emory University, GA, USA — Daniel Sippel, Department of Psychiatry and Psychotherapy, University Hospital Tübingen, Tübingen, Germany; Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany — Moritz D. Brandt, Department of Neurology, Technische Universität Dresden, Dresden, Germany; German Center for Neurodegenerative Diseases (DZNE) Dresden, Dresden, Germany — Stephan Bialonski, FH Aachen University of Applied Sciences, Aachen, Germany.

[^2]: Manuscript received XXX, revised XXX

[^3]: We note that in Ref.  Batch Normalization (BN) was carried out before the nonlinearities of a layer. However, we observed our model to fit our data well when applying BN after the nonlinearities.


abstract: Entanglement is a holistic property of multipartite quantum systems, which is accompanied by the establishment of nonclassical correlations between subsystems. Most entanglement mechanisms can be described by a coherent interaction Hamiltonian, and entanglement develops over time. In other words, the generation of entanglement has a time arrow. Dissipative structure theory directs the evolving time arrow of a non-equilibrium system. By dissipating energy to the environment, the system establishes order out of randomness. This is also referred to as self-organization. Here, we explore a new mechanism to create entanglement, utilizing the wisdom of dissipative structure theory in quantum systems. The entanglement between subsystems can emerge via the dissipation-structured correlation. This method requires a non-equilibrium initial state and cooperative dissipation, which can be implemented in a variety of waveguide-coupled quantum systems.
author: Zhi-Bo Yang; Yi-Pu Wang; Jie Li; C.-M. Hu; J. Q. You
date: 2024-09-30
title: Entanglement Emerges from Dissipation-Structured Quantum Self-Organization

*Introduction.—*The time arrow and irreversible processes in physics are closely related to the formulation of the second law of thermodynamics, which asserts that the entropy of a closed system does not decrease. The system's entropy will reach its maximum at the stable state of thermodynamic equilibrium, which is delimited by a loose boundary. The dissipative structure theory  extends the second law of thermodynamics to non-equilibrium systems, settling the boundary to the state of "least dissipation" . The steady states correspond to a minimum of entropy production in linear non-equilibrium thermodynamics. The dissipative structures describe a novel type of dynamic states of matter, where *dissipative* processes create and maintain the self-organized non-equilibrium structures.

Under the *constructive* role of irreversible processes, non-equilibrium dynamics become a source of order. However, non-equilibrium is required but not sufficient for the formation of dissipative structure and, ultimately, self-organization . The critical condition is that the dissipations of the subsystems are synergic rather than disordered. This mechanism can be found in the classical world, such as Huygens pendulum synchronization, wherein two pendulums dissipate energy to the same house beam . The fireflies synchronize by copying other fireflies around them, as the result of all the fireflies glowing to the same environment. Synchronizations in this context are unambiguous self-organization behaviors. Such mechanisms can also be found in more complex systems, e.g., organism dynamics  and social economic activities .

Time arrow also exists in the quantum world. A key question is how dissipative structure theory's wisdom will behave in a quantum system. It will be revealed that non-equilibrium dynamics and dissipation can also create a new order in the quantum system. *Quantum* self-organization leads to *entanglement*. Two factors are required: the system should be perturbed to become non-equilibrium and the dissipations of the subsystems should be synergic. Cooperative dissipation has been studied in a variety of systems and in many aspects, including Friedrich-Wintgen bound state in the continuum , subradiance , decoherence-free subspaces , vacuum-induced coherence , and cancellation of spontaneous emission . They are all related to the destructive interference in radiative decay channels and a "least dissipation" state can be observed. This cooperative dissipation effect has recently been termed as dissipative couplings . Previous studies have primarily been focused on equilibrium issues and steady states of the dissipatively coupled systems. The dynamic property of quantum state evolution and self-organization behavior remain underexplored.

In this Letter, we investigate the non-equilibrium state (NES) evolution of qubits dissipating cooperatively to a common reservoir. In the *ideal* case of two resonant qubits, we first perturb the system to a NES, where the total decoherence processes of the two individual qubits are synergic, and the system finally evolves to an entangled mixed state. The two-qubit system can no longer radiate, and cooperative dissipation drives the system to enter the "least dissipation" state. The entanglement is the result of dissipation-structured self-organization. Then we consider the situation where the dissipations are non-synergic and the two qubits have some detuning. We see that the system tends to establish entanglement, but the damping factors eventually destroy the quantum correlation. Correspondingly, we solve this problem by introducing a *quantum clockwork* (uninterrupted pump) into the system. The entanglement can be sustained. Further, we extend the mechanism to a three-qubit system. As expected, self-organized entanglement still emerges, because the systems described by dissipative structure theory are generally large ensembles. The quantum dissipative structure presents a novel method to entangle a large number of qubits. Extending dissipation-based self-organization to the quantum region opens up new research directions.

*Model.—*The essential system is represented schematically in Fig. (a) as waveguide-coupled quantum emitters, where two qubits dissipate to a zero-temperature common reservoir (e.g., a waveguide). Simultaneously, the two qubits can also dissipate to their own reservoirs, which is attributed to the non-cooperative dissipations. When only considering the cooperative dissipation, the dissipative system can be described by the following Lindblad master equation $$\begin{aligned}
\label{e001}
    \frac{d}{dt}\rho&=&-\frac{i}{\hbar}[H,\rho]+\kappa\mathcal{L}[o^{-}]\rho,
\end{aligned}$$ where $H/\hbar=\omega\sigma_1^+\sigma_1^-+(\omega+\delta)\sigma_2^+\sigma_2^-$ denotes the free Hamiltonian of the two qubits with excitation frequency $\omega$ and detuned frequency $\delta$. The two qubits have *no* direct interaction, and they have the same decay rate $\kappa$. Here $\sigma_{1,2}^+$ ($\sigma_{1,2}^-$) is the Pauli raising (lowering) operator. $\mathcal{L}[o^{-}]$ is the standard dissipative superoperator, and it can be expanded as follows: $$\begin{aligned}
\label{e002}
    \mathcal{L}[o^{-}]\rho=2o^{-}\rho o^{+}-o^{+}o^{-}\rho-\rho o^{+}o^{-},
\end{aligned}$$ where the jump operator needs to take the form $o^{-}=\sigma_1^-+\sigma_2^-$ for depicting the cooperative dissipation .

*Entanglement emerges from cooperative dissipations.—*Our approach starts with the *ideal* case of two resonant qubits, we first explore the eigenstates and eigenvalues, and then analyse the state and energy evolution from a non-equilibrium initial state. In the four-dimensional Hilbert space, and under the resonance condition ($\delta=0$), the effective Hamiltonian is written as $$\begin{aligned}
\label{e003}
H/\hbar&=&(\omega-2i\kappa)\vert\Psi_{+}\rangle\langle\Psi_{+}\vert+\omega\vert\Psi_{-}\rangle\langle\Psi_{-}\vert\nonumber\\
&&+2(\omega-i\kappa)\vert11\rangle\langle11\vert
\end{aligned}$$ with $\vert\Psi_{\pm}\rangle=(\vert10 \rangle\pm\vert01 \rangle)/\sqrt{2}$ . We can see that: (i) $\vert\Psi_{+}\rangle$, the symmetric Bell state, is a bright state that will decay at twice the decay rate due to the superradiance effect; and (ii) $\vert\Psi_{-}\rangle$, the anti-symmetric Bell state, is a dark state, which will not dissipate due to the subradiance effect. There are entangled states in the eigenstates of the dissipatively coupled system and one of them is the "least dissipation" state, which essentially lays out the main point of this Letter. Next, we need to investigate whether the system can evolve towards the corresponding "least dissipation" state from different initial states, and then finally establish entanglement.

The evolution of energy (in units of $\hbar\omega$) from four different initial states ($\vert11 \rangle$, $\vert10 \rangle$, $\vert\Psi_{-}\rangle$, $\vert\Psi_{+}\rangle$) are calculated by using Eq. ()  and plotted in Fig. (a). Looking at the eigenvalues, it is clear that the $\vert11 \rangle$ and $\vert\Psi_{+}\rangle$ will eventually dissipate all of the energy, whereas the dark state $\vert\Psi_{-}\rangle$ will never dissipate. The most fascinating thing happens when the initial state is a NES, which is $\vert10\rangle$. After dissipating half of the energy, the two-qubit system will become stable. This indicates that the system finally evolves to the "least dissipation" state. To see the detailed state evolution, we monitor the state fidelity with respect to the dark state. The fidelity is defined as *F* $=(\vert\Psi_{-}\rangle$, $\rho)=tr\sqrt{\langle\Psi_{-}\vert\rho\vert\Psi_{-}\rangle}$. *F* as a function of evolution time $\omega t$ is shown in Fig. (b). We can find that the fidelity of evolution states from initial states $\vert11 \rangle$ and $\vert\Psi_{+}\rangle$ relative to state $\vert\Psi_{-}\rangle$ is always zero. This implies that states $\vert11 \rangle$ and $\vert\Psi_{+}\rangle$ will never evolve to the dark state. While the states revolting from $\vert10 \rangle$ and $\vert\Psi_{-}\rangle$, the fidelities relative to state $\vert\Psi_{-}\rangle$ are always 0.5 and 1, respectively. It is worth noting that state $\vert10 \rangle$ can be written as a superposition state $(\vert\Psi_{+}\rangle+\vert\Psi_{-}\rangle)/\sqrt{2}$. During the evolution process, $\vert\Psi_{+}\rangle/\sqrt{2}$ will evolve to the ground state, so the fidelity of state $\vert10 \rangle$ is always $0.5$. That is, the final state is $\rho_{2}(\infty)=(\vert\Psi_{-}\rangle\langle\Psi_{-}\vert+\vert00\rangle\langle00\vert)/2$, which is an entangled mixed state .

The main distinction between the $\vert01\rangle$ state and the others is that it is a NES with unequal excitation probabilities for the two qubits. The physics behind this is that any NES contains a dark state component. For two qubits $A$ and $B$, we can have $\rho_{A}=a\vert0\rangle\langle0\vert+b\vert1\rangle\langle1\vert$, $\rho_{B}=c\vert0\rangle\langle0\vert+d\vert1\rangle\langle1\vert$, $a+b=c+d=1$, and $\rho_{AB}=ac\vert00\rangle\langle00\vert+bd\vert11\rangle\langle11\vert+2(bc-ad)\vert\Psi_{-}\rangle\langle\Psi_{-}\vert+2(ad+bc)\vert\Psi_{+}\rangle\langle\Psi_{+}\vert$. If $b\neq d$, then $2(bc-ad)\neq 0$. As a result, we have a two-qubit NES, and it clearly indicates the presence of the dark-state component. All non-dark states will be dissipated as the system evolves, resulting in the final entangled mixed state. This finding enriches the classical dissipative structure theory. The presence of non-equilibrium indicates the existence of the dark-state component, which is why the non-equilibrium is the source of order. We can now say that non-equilibrium *contains* order. The most significant advantage of the cooperative dissipation system is that once the entangled state is established, it will not be decohered, which will greatly increase the coherence time, especially when continuous pumping is lacking.

*Non-ideal cases and quantum clockwork*.—This section focuses on the entanglement decay caused by non-cooperative dissipations and detuning between qubits. As depicted in Fig. (b), the individual dissipations of the qubits to their respective reservoirs 1 and 2 are attributed to the non-cooperative dissipations $\alpha$, which can be introduced into the system by inserting the standard Lindblad operator $\sum_{j=1}^{2}\alpha\mathcal{L}[\sigma^{-}_j]\rho$ into Eq. (). Then we introduce a four-level qudit to provide auxiliary energy levels for loading the pump, which does not disturb the cooperative dissipation evolved in the entanglement generation. For the cooperative dissipations $\kappa$, they only play a role in the decay of non-dark states as shown in Fig. (a). However, non-cooperative dissipations $\alpha$ have an effect not only on the decay of the non-dark state, but also on the decay of the dark state, shifting the "least dissipation" state from zero dissipation one to finite dissipation one. Furthermore, the frequency detuning $\delta$ between the qubits will break the dark state, resulting in persistent state leaking.

In order to show the effects of non-cooperative dissipations and detuning on the entangled state, we solve the dynamic equation and track the time evolution of both energy and fidelity. The effects of non-cooperative dissipation $\alpha$ on the energy and state evolutions are depicted in Figs. (a) and (b), respectively. We can see that if the initial state is a dark state, the energy and fidelity will only be affected by non-cooperative dissipation, leading to a smooth exponential decay (green dotted curve). When the initial state is a NES $\vert10\rangle$, the reduction of fidelity is only attributed to the dark state decay. However, due to the synthetic cooperative and non-cooperative dissipations, the energy will quickly decay to $0.5~\hbar\omega$ (red dot-dashed curve). At this point, most of the non-dark state component is exhausted, and the residual part is mainly the dark state. An entangled mixed state is formed. Then the non-cooperative dissipation further decays the state to the ground state slowly. In this case, the system tends to reach a metastable state (i.e., an entangled mixed state) before it reaches the ground state. If the initial state is an equilibrium excited state $\vert11\rangle$ or bright state, the energy will rapidly decay to zero, as shown by the orange solid and blue dashed curves. Since NES is the necessity of a dissipative structure, we can conclude that the energy of a quantum system decays at a faster rate than would occur if the dissipative structure did not exist.

Setting $\alpha=0$, we investigate the entanglement and energy decay caused by frequency detuning $\delta$, as depicted in Figs. (c) and (d). We find that even without non-cooperative dissipation, the energy will eventually be exhausted regardless of the system's initial state. Indeed, the presence of detuning breaks the dark state, resulting in both a decay factor and a phase factor of the "least dissipation" eigenstate . This causes the fidelity and energy to rapidly decay to zero with periodic oscillations.

In the pendulum synchronization experiment, due to the intrinsic friction loss, the synchronized pendulums will eventually stop moving, corresponding to the ground state of the quantum system. The similarity between the classical and quantum systems is that before the energy is completely dissipated, self-organization driven by non-equilibrium will cause the classical and quantum systems to enter the "least dissipation" state. It is the synchronized state in the classical case, and the entangled state in the quantum case. Interestingly, a clockwork can be added to provide continuous power to the classical pendulum without affecting its normal swing period. Drawing on this wisdom, we replace one qubit with a four-level qudit and pump the corresponding transition, so as to provide a continuous working energy source for the evolution of quantum self-organization, i.e., introducing the idea of *quantum clockwork*, as schematically shown in Fig. (c). See supplementary material for more details . While pumping the system, we can monitor the entanglement evolution of the qubit-qudit system. The entanglement can be characterized by the logarithmic negativity $E_N=\log_{2}\Vert\rho^{T}\Vert_1$, where the symbol $\Vert$ $\Vert_1$ denotes the trace norm, and $\rho^{T}$ is the partial transpose of the reduced density matrix $\rho$ of the two sub-systems . The logarithmic negativity $E_N$ is plotted as a function of time $\kappa t$ in Fig. (e), demonstrating that the steady-state entanglement can be generated with the help of quantum clockwork, even when the non-cooperative dissipation and detuning exist.

*Three-qubit W-like state generation.—*Dissipative structures in nature typically contain many subsystems. After a long evolution time, self-organization is finally established. Quantum system has no exception. Starting from NES, we can establish multi-qubit quantum entanglement in principle through cooperative dissipation. Next, we will take a three-qubit system as an example to discuss the process of building the *W*-like entangled state in detail, and discuss the possibility of extending it to *N*-qubit systems. In this section, we consider the ideal case where the properties of the three qubits are exactly the same and the non-cooperative dissipations are zero.

We can find that any single-excitation NES can be represented by basis vectors $\vert\Psi_{1}\rangle=(\vert 100\rangle+\vert 010\rangle+\vert 001\rangle)/\sqrt{3}$, $\vert\Psi_{2}\rangle=(2\vert100\rangle-\vert010\rangle-\vert001\rangle)/\sqrt{6}$, and $\vert\Psi_{3}\rangle=(\vert010\rangle-\vert 001\rangle)/\sqrt{2}$, where $\vert\Psi_{2}\rangle$ and $\vert\Psi_{3}\rangle$ are dark states of the system. For example, the non-equilibrium initial state $\vert100\rangle$ can be written as $\vert100\rangle=\vert\Psi_{1}\rangle/\sqrt{3}+2\vert\Psi_{2}\rangle/\sqrt{6}$. However, self-organizing behavior will dissipate all non-dark states. Therefore, the final state of the system is $\rho_3(\infty)=(2\vert\Psi_{2}\rangle\langle\Psi_{2}\vert+\vert000\rangle\langle000\vert)/3$, which is a *W*-like entangled mixed state . In the dynamical illustration, the photon dissipated by qubit 1 enters the common reservoir, and this photon may be absorbed by either qubit 2 or qubit 3. This process realizes the entanglement among the three qubits. It should be noted that when the non-equilibrium initial state is $\vert 011 \rangle$, since all the double-excitation basis vectors are non-dark states , the system will further decay and keep evolving to the "least dissipation" state $\rho_3(\infty)$ .

From different initial states ($\vert 100 \rangle$ and $\vert 011 \rangle$), the energies *E* as a function of time are plotted in Fig. (a). When the system establishes three-qubit entanglement through cooperative dissipation, the system will definitely evolve to the state $\rho_3(\infty)$. In this process, all non-dark states will decay to the ground state $\vert 000 \rangle$, and approximately $0.67~\hbar\omega$ energy will remain, which is the intrinsic energy of the dark state part $2\vert\Psi_{2}\rangle/\sqrt{6}$ of the initial state. The fidelities versus the dark state $\vert \Psi_2 \rangle$ as a function of time are plotted in Fig. (b). The $\vert 100 \rangle$ state contains a certainly proportion of the dark state $\vert \Psi_2 \rangle$, and $\vert \Psi_2 \rangle$ will not decay. Therefore, the fidelity is invariable. For the initial state $\vert 011 \rangle$, the excitation number is 2. At $\omega t=0$, the fidelity versus the single-excitation state $\vert \Psi_2 \rangle$ is zero. As the system dissipates energy, it finally evolves to $\rho_3(\infty)$, and the fidelity reaches the same value as that in the $\vert 100 \rangle$ case.

Finally, the non-equilibrium *N*-qubit cooperative dissipation system can also generate the *W*-like state through the quantum self-organization, which is schematically shown in Fig. (d). The cooperative dissipation system composed of *N* identical qubits with non-equilibrium initial state can evolve to the *W*-like entangled mixed state $\rho_{N}(\infty)$ (least dissipation state), which only contains the dark-state components and the *N*-qubit ground state. Also, the relative phases between the dissipations of the qubits should be considered, which is closely related to the relative position between the qubits in the waveguide setup.

*Conclusion and perspectives.—*Dissipation is not necessarily a detrimental occurrence. In this work, we study the framework of dissipative structures in the quantum world. The key finding is that non-equilibrium and cooperative dissipation can drive the quantum self-organization. The second law of thermodynamics specifies which processes are allowed to occur in nature, such as the requirement of nondecreasing entropy in an isolated system. It predicts that everything will eventually approach the maximum entropy state. The dissipative structure theory, on the other hand, tells which processes tend to occur. An example is that if the system is perturbed, the entropy production will increase, but the system reacts by returning to the minimum value of the entropy production state. This state is also the "least dissipation" state. By introducing this wisdom to the quantum world, we find that a quantum system also tends to "drop" into the quantum "least dissipation" state. Profoundly, this state could be an entangled mixed state, which is unique to quantum systems when compared with classical systems. Our work reveals the tip of the iceberg of quantum self-organization behavior, and will open up a new direction for studying dissipative structure theory in quantum systems.

We also discuss the new entanglement generation mechanism in detail, and study the *ideal* and *non-ideal* cases, two-qubit and three-qubit models, and the possibility of extending it to *N*-qubit systems. The scheme could be implemented in waveguide-coupled ultracold atoms , superconducting qubits , or hybrid quantum emitters and circuits .


abstract: Heavy fermion (HF) materials exhibit a rich array of phenomena due to the strong Kondo coupling between their localized moments and itinerant electrons. A central question in their study is to understand the interplay between magnetic order and charge transport, and its role in stabilizing new quantum phases of matter. Particularly promising in this regard is a family of tetragonal intermetallic compounds Ce$TX$$_2$ ($T=$ transition metal, $X=$ pnictogen), that includes a variety of HF compounds showing $T$-linear electronic specific heat $\bf{C_e \sim \gamma T}$, with $\gamma\sim$ 20-500 mJ$\cdot$mol$^{-1}$ K$^{-2}$, reflecting an effective mass enhancement ranging from small to modest. Here, we study the low-temperature field-tuned phase diagram of high-quality CeAgBi$_2$ using magnetometry and transport measurements. We find an antiferromagnetic transition at ${T_{N} = 6.4}$ K with weak magnetic anisotropy and the easy axis along the $c$-axis, similar to previous reports (${T_{N} = 6.1}$ K). This scenario, along with the presence of two anisotropic Ruderman-Kittel-Kasuya-Yosida (RKKY) interactions, leads to a rich field-tuned magnetic phase diagram, consisting of five metamagnetic transitions of both first and second order. In addition, we unveil an anomalous Hall contribution for fields $H<54$ kOe which is drastically altered when $H$ is tuned through a trio of transitions at 57, 78, and 84 kOe, suggesting that the Fermi surface is reconstructed in a subset of the metamagnetic transitions.
author: S. M. Thomas; P. F. S. Rosa; S. B. Lee; S.A. Parameswaran; Z. Fisk; J. Xia
date: 2024-09-29
title: Hall effect anomaly and low-temperature metamagnetism in Kondo compound CeAgBi$_{2}$

[^1]

[^2]

In heavy fermion (HF) materials, the Kondo coupling between local moments and itinerant electrons plays a central role in determining magnetic and transport properties, particularly at low temperatures. Classic examples of the unusual behavior include quantum criticality in YbRh$_{2}$Si$_{2}$ , unconventional superconductivity in CeCoIn$_{5}$ , and metamagnetism in CeRu$_{2}$Si$_{2}$ . Ce-based HF materials which host such exciting properties often crystallize in tetragonal structures and their ground state is on the border of antiferromagnetism (AFM).

Our focus in this paper is the HF family Ce*TX*$_2$ ($T =$ transition metal, $X =$ pnictogen), a class of intermetallic compounds that crystallize in the tetragonal ZrCuSi$_2$-type structure (space group $P4/nmm$) with a stacking arrangement of Ce$X$-$T$-Ce$X$-$X$ layers. Great attention has been given to the antimonide ($X =$ Sb) members due to the presence of anomalous ferromagnetism in CeAgSb$_{2}$ and, more recently, due to the report of field-induced quantum criticality in CeAuSb$_{2}$ . Although the investigation of the antimonides is abundant, fewer reports can be found on the bismuthide ($X =$ Bi) members. Recent studies of high-quality CeCuBi$_{2}$ and CeAuBi$_2$ revealed AFM ordering temperatures of $T_N=$`<!-- -->`{=html}16 K and 12 K, respectively. Both materials show a moderate mass enhancement from $T$-linear electronic specific heat ($\gamma$ on the order of 100 mJ$\cdot$mol$^{-1}$ K$^{-2}$) and exhibit a single spin-flop transition with field applied along the $c$-axis .

CeAgBi$_2$, a third member of the isovalent series, has been reported previously to order antiferromagnetically ($T_N = 6.1$ K) at zero magnetic field and to undergo three field-induced magnetic transitions at 2 K ($H_{c} = 35, 50, 83$ kOe) . However, a detailed analysis of the underlying interactions and a deep understanding of the phase diagram at low temperatures is still missing.

Single crystals of CeAgBi$_2$ were grown from Bi-flux with starting composition Ce:Ag:Bi=1:$x_{nominal}$:8 ($1 \le x_{nominal} \le 3$). The mixture was placed in an alumina crucible and sealed in a quartz tube under vacuum. The sealed tube was heated up to $1050 ^{\circ}\mathrm{C}$ for 8 h and then cooled down at $10 ^{\circ}\mathrm{C/h}$. The excess of Bi flux was removed by centrifugation after 24 h of annealing at $500 ^{\circ}\mathrm{C}$. Single crystals with dimensions $\sim 3 \times 3 \times 0.5$ mm$^{3}$ were ground and their crystal structure was checked by X-ray powder diffraction experiments using Cu $K\alpha$ radiation at room temperature. Several single crystals from different batches were also submitted to elemental analysis using a commercial Energy Dispersive Spectroscopy (EDS) microprobe coupled to a FEG SEM microscope. From the EDS analysis, we have extracted the actual $x_{Ag}$ concentration. The precision of the analysis was calculated by $\sigma$/$\sqrt{N}$, where $\sigma$ is the standard deviation of the measurements, and $N$ is the number of points analyzed.

Magnetization measurements were performed using a commercial superconducting quantum interference device (SQUID) down to 1.8 K. Below 1.8 K, cantilever-based torque magnetometry was used to measure the magnetization in a dilution refrigerator. Electrical resistivity measurements were made using a low-frequency ac resistance bridge and a four-point configuration. Hall data was obtained by measuring in both positive and negative applied fields and taking the difference ($\left[R(H+) - R(H-)\right]/2$). Due to hysteresis in the transitions data was paired for subtraction based on whether the magnitude of the field was increasing or decreasing. All measurements reported here, except for some of the magnetic susceptibility measurements, were made with the applied field parallel to the c-axis. For resistivity measurements, the current was applied in the ab-plane.

# Experimental Results and CEF fits

Fig. a shows the temperature dependence of in-plane resistivity, $\rho(T)$, down to 0.5 K at zero magnetic field. At high temperatures ($T \gtrsim 200$ K), $\rho(T)$ shows metallic behavior, decreasing linearly with decreasing temperature. However, further decrease in temperature reveals a resistivity minimum, followed by a logarithmic increase due to incoherent Kondo scattering. Below $\sim 25$ K, $\rho(T)$ drops abruptly, suggesting that this is the energy scale of either CEF depopulation or Kondo coherence. We will discuss these possibilities below. The kink in resistivity observed at $T_{N} = 6.4$ K, shown in the top inset of Fig. 1a, indicates the transition to the AFM phase. This represents a 0.3 K higher ordering temperature compared to previous studies of CeAgBi$_2$ . Although this increase in $T_{N}$ is somewhat small, EDS measurements reveal that it is caused by a substantial decrease both in the number of vacancies and in the inhomogeneity at the Ag site. The bottom inset of Fig. 1a shows the linear dependence of $T_{N}$ on the occupation at the Ag site, $x_{\rm EDS}$. For the most deficient samples, with Ag occupation of $82(4)$%, the transition temperature matches previous studies, $T^{\text{def}}_{N} = 6.1$ K. For the best samples obtained to date, the Ag occupancy reaches $87(2)$%, confirming the trend that transition-metal deficiency is an intrinsic feature of this family of compounds. We note that less deficient samples are also accompanied by higher resistance ratios and lower residual resistivity.

Like other Ce-based bismuthides, CeAgBi$_2$ also exhibits magnetic anisotropy. Fig. b shows the temperature dependence of the inverse magnetic susceptibility, $1/\chi(T)$, when a magnetic field of $H$ = 1 kOe is applied parallel ($\chi_{\parallel}$) and perpendicular ($\chi_{\perp}$) to the crystallographic $c$-axis. The inset of Fig. b presents the low-temperature $\chi(T)$ data in which a sharp peak is observed at $T_{N}$. The ratio $\chi_{||}/\chi_{\perp} \approx 3.5$ at $T_{N}$ is mainly determined by the tetragonal CEF splitting and reflects the low-$T$ Ce$^{3+}$ single ion anisotropy. This ratio is smaller than what is found in other bismuthides, suggesting a smaller CEF splitting between the ground state and the first excited state as well as a less anisotropic ground state. This scenario will be confirmed below.

At high temperatures ($T > 150$ K), $\chi(T)$ is well-described by a Curie-Weiss (CW) law plus a $T$-independent Pauli term, $\chi(T) = \chi_{0} + C/(T-\theta_{CW})$. We obtain an effective moment of $\mu_{eff}\approx 2.5(1)\mu_{B}$ for both directions and also for the polycrystalline average, in agreement with the theoretical value of $\mu_{eff}\approx 2.54 \mu_{B}$ for Ce$^{3+}$ free ions. On the other hand, the $\theta_{CW}$ values are anisotropic, with $\theta_{||}=5.7$ K and $\theta_{\perp}=-8.5$ K. For the polycrystalline averaged data (not shown), we obtain $\theta_{p} = -4$ K, consistent with AFM order at $\sim$ 6 K.

```latex
\begin{table*}[t]\begin{tabular*}{0.85\textwidth}{@{\extracolsep{\fill}} cccccccc}
\hline
\multicolumn{8}{c}{CEF parameters} \\
\cline{1-8}\\
& $B_{2}^{0}$ &$B_{4}^{0}$ & $B_{4}^{4}$ & & $z_{FM}J_{FM}$ & & $z_{AFM}J_{AFM}$\\

& -1.78~K& 0.168~K &0.71e-3~K& &-0.89~K& &1.35~K\\
\hline
\multicolumn{8}{c}{Energy levels and wave functions} \\
\cline{1-8}\\
$E (K)$ & & $|-5/2\rangle$ & $|-3/2\rangle$ & $|-1/2\rangle$ & $|+1/2\rangle$ & $|+3/2\rangle$ & $|+5/2\rangle$\\
0 & & 0 & -1 & 0 & 0 & 0 & 0\\
0 & & 0 & 0 & 0 & 0 & 1 & 0\\
19 & & 1 & 0 & 0 & 0 & 0 & 0\\
19 & & 0 & 0 & 0 & 0 & 0 & -1\\
61 & & 0 & 0 & -1 & 0 & 0 & 0\\
61 & & 0 & 0 & 0 & -1 & 0 & 0\\
\end{tabular*}
\caption{
CEF parameters, energy levels, and wave functions of CeAgBi$_{2}$ single crystals obtained from the best fits of the magnetic susceptibility data to the model described in the text.
}
\label{tab:1}
\end{table*}
```

We now further explore the role of anisotropic interactions and CEF effects in the magnetic properties of CeAgBi$_2$. To this end, we analyze the experimental data using a mean field model including two mean anisotropic interactions ($z_{\mathrm{AFM}}J_{\mathrm{AFM}}$ and $z_{\mathrm{FM}}J_{\mathrm{FM}}$) between nearest-neighbors, which do not contain directional information. We also take into account the tetragonal CEF Hamiltonian $H_{CEF}=B_{2}^{0}O_{2}^{0} + B_{4}^{0}O_{4}^{0} + B_{4}^{4}O_{4}^{4}$, where $z$ is the number of nearest neighbors, $B_{i}^{n}$ are the CEF parameters, and $O_{i}^{n}$ are the Stevens equivalent operators obtained from the angular momentum operators. A more detailed description of the model can be found in Ref. .

This model was used to simultaneously fit $\chi(T)$, $M(H)$ and $C_{mag}(T)/T$ data in the entire range of temperature. The best fit which reproduces the anisotropic susceptibility is shown by solid lines in Fig. b and the extracted parameters for the CEF scheme and the exchange interactions are given in Table . We find a $|J=5/2,J_z = \pm3/2\rangle$ Kramers doublet ground state, separated from other excited doublets by 19 K and 60 K. We also find, in fact, (i) the CEF splitting is much smaller as compared to other bismuthides and (ii) the ground state is mainly $|J=5/2,J_z = \pm3/2\rangle$ instead of the dominantly $|J=5/2,J_z = \pm5/2\rangle$ found in CeCuBi$_{2}$ and CeAuBi$_{2}$. Further, the dominant CEF parameter $B_{2}^{0}$ obtained from the fits is similar to the value obtained using the high-temperature expansion of $\chi$: $B_{2}^{0}=10(\theta_{\perp}-\theta_{||})/[3(2J-1)(2J+3)]=-1.48$ K. This result suggests that the effects of anisotropic interactions at high temperatures are smaller in CeAgBi$_2$ than in CeCuBi$_2$ and CeAuBi$_2$. We also note that although the magnetic anisotropy and $T_{N}$ along the $c$-axis are well reproduced by our simple model, the single ion CEF effect is not able to capture all field-induced transition in $M(H)$ data at 1.8 K.

Fig. c shows the temperature dependence of the specific heat of CeAgBi$_2$ in zero field. To determine the magnetic contribution to specific heat, the data obtained from non-magnetic reference compound LaAgBi$_2$ was subtracted from the result. There is a single peak corresponding to T$_N$. We note that this sample is from an earlier batch, so it has higher silver vacancies and a lower T$_N$. The top inset of Fig. 1c shows the integration of specific heat over temperature, i.e., the recovered magnetic entropy. At T$_N$, the recovered entropy is only about 80% the ground-state doublet (R$\ln{2}$). We note, however, that there is a broad feature centered around 8 K, which is consistent with a Schottky anomaly generated by the CEF splitting of 19 K. This result indicates that the first excited CEF state is already partially occupied at T$_N$, giving an additional entropy to the expected R$\ln{2}$. Therefore, the reduction of entropy is larger than 20%, although a precise calculation cannot be evaluated. It is unlikely that a reduction of entropy of more than 20% can be fully accounted for by classical magnetic fluctuations. For example, numerical calculations of the Ising model on a simple cubic lattice find an entropy reduction of just under 20% at T$_N$, with other three dimensional geometries showing an even smaller reduction. We thus attribute this reduction to both magnetic frustration and hybridization between Ce $4f$-electrons and conduction electrons. This conclusion is motivated by the enhanced effect masses found in dHvA experiments and the magnetic frustration that often arises from competing exchange interactions.

Due to the similar energy scales of the first excited CEF state ($19$ K) and the peak in resistance ($24$ K), we performed high-temperature resistance measurements in several different fields. The results are presented in inset of Fig. 1c. If the peak near 24 K were due to crystal field depopulation, then increasing the field should lead to an increase of the energy difference between the lowest Zeeman split state of the ground state and the upper Zeeman split state of the first excited CEF state. This in turn would lead to an increase of the temperature of the resistance peak as the field is increased, which is in fact observed experimentally. We note that, in heavy fermion compounds where the energy difference between Kondo coherence and $\Delta_{\mathrm{CEF}}$ is small, the onset of Kondo coherence ($T^{*}$) usually occurs at temperatures of the order of $\Delta_{\mathrm{CEF}}/2$. This suggests that $T^{*} \sim 10$ K in CeAgBi$_{2}$. However, the experimental lack of two distinct resistance maxima as is typically observed when the energy difference is large leads to an ambiguity as to whether the resistance peak is due to CEF splitting of Kondo coherence.

We now turn to the analysis of the low-temperature phase diagram as a function of an applied magnetic field along the $c$-axis. Fig. a shows magnetization and (transverse) magnetoresistance (MR) of CeAgBi$_{2}$ at 100 mK. As the field is swept from $0$ to $90$ kOe, we find a sequence of field-induced metamagnetic transitions near $H=34, 37, 54, 78,$ and $84$ kOe, indicating a complex phase diagram for $T<T_N$. Above the $84$ kOe transition, the magnetization saturates just below $1.9$ $\mu_{B}$/Ce, which is slightly lower than the saturation value of $2.1$ $\mu_{B}$/Ce previously reported, and may be due to a non-linearity between the deflection of the cantilever and the corresponding change in capacitance. Such large saturated value is somewhat surprising since the largest possible $c$-axis magnetization of a $|\pm3/2\rangle$ CEF ground state is $(3/2)g_L.\mu_B$, i.e., 1.29 $\mu_{B}$/Ce. Hence, our results indicate that the Zeeman effect induces a change in the CEF ground state from $|\pm3/2\rangle$ to $|\pm5/2\rangle$ when $H \sim 84$ kOe.

Of the observed transitions, the one near 78 kOe appears to be strongly first-order as shown by magnetization in Fig. b. As $T$ is lowered from 500 mK to 35 mK a sharpening of the transition is observed and the transition eventually becomes step-like. Field sweeps performed around this transition show signs of irreversibility, namely a $\sim4$ kOe wide hysteresis loop, providing further evidence of first-order behavior. This in contrast with other Ce-based bismuthides. For instance, other members of the isovalent series such as CeCuBi$_2$ ($T_{N} =  16$ K) and CeAuBi$_2$ ($T_{N} =  12$ K) show only a single clear spin-flop transition. We note, however, that multiple steps have been observed in previous reports on more deficient CeCuBi$_2$ ($T_{N} =  11$ K) and even CeCuBi$_2$ ($T_{N} =  16$ K) shows hints to a second transition just before the main one. It is also noteworthy that hysteresis loops show a significant difference ($\sim 2$ kOe) at the transition centered near $54$ kOe as well. The change in magnetization, however, is not nearly as abrupt as found in the 78 kOe transition. It is well-known that transport measurements in real materials are complicated by disorder. Thus, as this transition is clearly not strongly first-order, the slight amount of irreversibility is likely attributed to pinning due to crystallographic defects (e.g., inherent silver deficiency). In fact, a small amount of hysteresis (on the order of a hundred gauss) was also observed in magnetization and transport at the remaining three transitions.

The rich phase structure in CeAgBi$_2$ is likely due to the weak anisotropy combined with anisotropic exchange parameters with opposite signs, which lead to magnetic frustration. Further, the close energy scales of the CEF splitting ($\sim19$ K), Kondo coherence ($\sim 10$ K) and the AFM ($T_N\sim6.4$ K) generates a complex response of the physical properties to the application of magnetic field.

The MR also reflects the multi-step phase structure, tracking each metamagnetic transition with a sharp step, indicating the presence Kondo coupling between the Ce magnetic moments and itinerant $p$-electrons from Bi. The overall trend of MR is linear, a feature shared with the (non-magnetic) compounds LaAgBi$_2$ and LaAgSb$_2$, where it is attributed to an underlying Dirac dispersion for Bi/Sb itinerant electrons . A preliminary ab initio calculation  for CeAgBi$_{2}$ indicates that the Bi $p$-electrons also have a Dirac dispersion, suggesting a similar origin for the underlying linear MR. However, we note an alternative explanation for the increase in MR is an enhancement in spin-disorder scattering in AFM materials in an applied field . Note that the MR exhibits a sharp drop and hysteretic behavior near the $79$ kOe transition, lending further support that it is first-order.

Motivated by this evidence for coupling of itinerant electrons and local moments, we performed measurements of the Hall resistivity $\rho_{xy}$ to further elucidate the nature of the different phases. Like the MR, the Hall effect tracks each metamagnetic transition (Fig. ).

The decrease in $\rho_{xy}$ roughly parallels the increase in magnetization over the first two transitions, but then sharply deviates from this trend as the field is increased across the trio of transitions at 57, 78, and 84 kOe, even changing its sign between $\sim{}60-80$ kOe. Further insight is afforded by fitting the Hall resistivity to the standard form $\rho_{xy} (H) = R_H H + R_M M(H)$ that includes Hall effect contributions from both the applied field and the induced magnetization. Using the data up to $H\lesssim 38$ kOe to obtain the fit parameters, we find that this model fits the data extremely well for low fields $H\lesssim54$ kOe, whereupon the measured $\rho_{xy}$ strongly deviates from the expectation based on the model. While there is still some discrepancy between the fit and the data after the last transition ($H\gtrsim 84$ kOe), this is relatively small compared to the much larger deviation between fit and data in the regime $54$ kOe $\lesssim H\lesssim 84$ kOe. As the temperature is raised, the magnitude of the anomaly in $\rho_{xy}$ near 54 kOe is decreased, as shown in Fig. b. At present, we do not have a complete understanding of the origin of this Hall effect anomaly, but we comment on some possible scenarios below. We note that while there is an additional deviation from linear behavior of $\rho_{xy}$ above $\sim110$ kOe, a previous study on lower-quality samples using pulsed fields up to $400$ kOe  appears to rule out an additional transition.

# Discussion

Our results can be summarized in the form of a phase diagram of CeAgBi$_2$ with temperature and applied field, as shown in Fig. . The second $T=0$ phase (for 34 kOe $<H<$ 37 kOe) is not seen above the temperature $T\gtrsim$ 2.5 K, illustrating the fragility of this phase to thermal fluctuations. Note the strong agreement between the probes of specific heat, transport, and magnetic structure when the corresponding measurements overlap. Of the sequence of transitions between the different low-temperature phases, the penultimate transition near 79 kOe appears to be strongly first-order, whereas the remaining transitions appear continuous. As noted, the Hall effect is well-described by a simple model including effects of both the applied field and induced magnetization very well below $54$ kOe and reasonably well above $84$ kOe, but exhibits a strong deviation from this simple behavior in the intermediate field regime.

Heavy fermion compounds CeAuSb$_2$ and YbAgGe both have some commonalities in the low-temperature phase diagram. For instance, CeAuSb$_2$ orders AF and has at least one magnetic transition before the suppression of AF order to 0 K at 54 kOe. Recent work suggests that the first transition might bifurcate into two at lower temperatures, similar to CeAgBi$_2$. YbAgGe also has a similarly complicated phase diagram as a function of field which contains a region partially bordered by a first order transition . Interestingly, in this region YbAgGe exhibits anomalous behavior attributed to the influence of a quantum bicritical point . Unlike in YbAgGe, however, the set of transitions is only observed in CeAgBi$_2$ for $H||c$. Further, the large anomalies in the Hall resistivity appear to be a feature unique to CeAgBi$_2$.

We now comment on potential explanations for the physics at hand. First, we attribute the sequence of metamagnetic transitions to the frustration from competing anisotropic exchange interactions, as evinced by the susceptibility data. As in many HFs, the Hall data is more challenging to describe . Its evident sensitivity to the magnetization suggests a significant contribution from the induced magnetization at low fields, yet a simple phenomenological model incorporating this contribution seems unable to describe the behavior at intermediate fields. Here we note that quantum oscillation measurements reveal electron effective masses in the range of 5 to 7 m$_e$, suggesting heavy fermion physics plays a role in transport ; this could perhaps explain the anomaly, e.g. via a Fermi surface reconstruction. To explore this scenario, one would have to investigate, for instance, band-structure calculations in the presence of a magnetic field. We anticipate that future theoretical analysis, incorporating details of the conduction electron band structure, as well as a careful treatment of their coupling to local moments, will shed more light on these issues .

An alternative scenario involves the 'geometrical Hall effect' from a non-collinear spin texture; while more exotic, such physics is now quite well-established as a consequence of Dzyaloshinskii-Moriya interactions in ferromagnets and has been also proposed to occur in anisotropic exchange AFMs, and recently also in heavy-fermion metals . In this case, an anomalous Hall effect could be observed as a result of a non-vanishing Berry-phase curvature . To this end, neutron diffraction/scattering experiments would be valuable to unveil, for instance, the field regions where non-collinear spin structures are realized.

# Spin model and metamagnetic transitions

As mentioned above, our previous model was unable to capture all field-induced magnetic transitions. Finally, in the present section we introduce a spin model for the Ce local moments to discuss their field-induced response. Although more experimental investigation is clearly essential in order to clarify what type of AFM order is stabilized in CeAgBi$_2$ in each field region, here we sketch one possible scenario built on existing experimental data on the closely-related compound CeCuBi$_2$, where the zero-field magnetic structure has already been well-established experimentally . We defer a detailed analysis of the spin model to future work, and here only discuss a simple mean-field picture of the magnetic order.

For CeCuBi$_2$, X-ray magnetic diffraction reveals that the Ce local moments order antiferromagnetically with an ordering wave vector $(0,0,0.5)$ . This corresponds to "up-up-down-down" magnetic ordering along the crystallographic $c$-axis and ferromagnetic (FM) ordering across the basal $a$-$b$ plane. (See Fig. ). The magnetic susceptibility data shows a large anisotropy consistent with a strong easy-axis along the $c$-axis. The application of a $c$-axis magnetic field results in a single spin-flop transition at a field of $\sim 55$kOe; for fields in the $ab$ plane, no such transition was observed. Within a classical approximation, a minimal spin model that may be inferred from these experimental results incorporates a single-ion anisotropy term and dominant spin exchanges between neighboring sites: $${\mathcal H} = \sum_{ ij } J_{ij} {\boldsymbol S}_{i} \cdot  {\boldsymbol S}_{j} -  \sum_{i}  {\boldsymbol h} \cdot {\boldsymbol S}_{i} - \Delta \sum_{i} ({S_i}^z )^2.
\label{suppeq:1}$$ Here, ${\boldsymbol h}$ is the (applied) magnetic field and $\Delta>0$ is the magnitude of an easy-axis single-ion anisotropy along the $c$-axis. $J_{ij}$ represents competing spin exchanges between neighboring sites; FM and AFM (or AFM and FM) spin exchanges for two symmetry distinct neighbors along $c$ axis are denoted $J$ and $J'$, respectively, and a FM spin interaction in the $a$-$b$ plane is denoted as $J_\perp$, as seen in Fig. . The competition between alternating ferro- and antiferromagnetic spin exchanges along the $c$-axis stabilizes the "up-up-down-down" magnetic structure along $c$-axis, with a spin polarized state FM on each $a$-$b$ plane. In the limit $\Delta \gg J_{ij}$, the moments behave like $c$-axis Ising spins, and the presence of a $c$-axis field leads to a direct magnetic transition from AF ordering to a fully-spin-polarized, paramagnetic state. However, a smaller single ion anisotropy $\Delta \lesssim J_{ij}$ admits a spin-flop transition at finite $c$-axis field, resulting in AF spin canting. This is consistent with the experimental data. Given that the minimal picture of competing spin exchanges and single ion anisotropy provide a reasonable explanation of magnetic ordering and field effect of CeCuBi$_2$ material, we now turn to a similar analysis of CeAgBi$_2$.

Magnetic susceptibility measurements on CeAgBi$_2$ again show the presence of magnetic anisotropy but smaller than in CeCuBi$_2$, which indicates weaker Ising anisotropy of the localized moments. Furthermore, when Ag substitutes for Cu on the the transition metal site, spin superexchange between Ce moments occurs via the $4d$ or $5s$ Ag orbitals, as compared to the $3d$ or $4s$ electrons of Cu, thereby changing the effective superexchange couplings $J, J', J_\perp$. Moreover, the slight difference in lattice parameters and bond angles may also lead to changes in the RKKY interactions induced by coupling to the itinerant electrons of Bi. In general, this confluence of competing effects may stabilize very complicated magnetic ordering. We also note that further-neighbor interactions besides $J$, $J'$ and $J_\perp$ generally induce incommensurate *spiral* ordering, affording an even richer set of possibilities. For the purposes of a preliminary analysis, we focus on our minimal spin model keeping just $J$, $J'$ and $J_\perp$, and describing the possible ordered states as a function of the applied field. Notably, even this simple model exhibits a plethora of magnetic orders separated by a sequence of metamagnetic transitions.

A convenient approach to understand the magnetic order is to separately consider every two layers of Ce sites, corresponding to a single unit cell lattice spacing along the $c$ axis. These two layers of Ce sites then form a buckled square lattice, where the nearest and next-nearest neighbor spin-spin couplings are $J$ and $J_\perp$ respectively; we will refer to this as a 'bilayer'. Reasoning in analogy with the well-studied $J_1-J_2$ square-lattice Heisenberg model, we anticipate that FM, Néel and stripy phases can be stabilized in each bilayer depending on the relative signs and magnitudes of $J$ and $J_\perp$. We may then use these orders as a building block to construct the full three dimensional magnetic order by coupling the Ce bilayers via an interlayer coupling $J'$. We now briefly summarize how competing interactions and magnetic anisotropy can lead to several metamagnetic transitions within this approach, even if we restrict ourselves to working within a classical spin approximation. In order to have several metamagnetic phases, we require that $J$, $J'$, $J_\perp$ and $\Delta$ are all of the same order; this is also consistent with expectations based on experiments and a study of lattice parameters. To describe the ordered states stabilized with this parameter regime, it is convenient to work with a magnetic unit cell 8 times larger than the original unit cell (i.e., containing 16 Ce sites), and optimize the classical energy of this unit cell for a given set of parameters $J_{ij}$, $\Delta$ and $h$, we minimize the energy of Eq..

Fig.  shows the magnetization $m$ ($m=1$ is the fully polarized classical spin per site) as a function of $c$-axis magnetic field for specific values of parameters $J = -1$, $J'=2$, $J_\perp= 1.5$ and $\Delta =1$, which are the parameters that best describe the experimental data. In fact, we find six distinct phases over our field range. Note that the exact same argument can be applied when $J$ and $J'$ values are interchanged, due to site connectivity. The magnetic ordering pattern in each of these phases is illustrated in Fig. .

At a zero field, each buckled square lattice bilayer forms a stripy phase, as $|J| < |J_\perp |$ and the stripy phase is robust under a small and finite magnetic field. With the chosen parameters, this phase is degenerate in energy with the "up-up-down-down" like ordering (discussed for the case of CeCuBi$_2$ and illustrated in Fig. ). Further increase of magnetic field, however, induces a transition into a phase where spins in the lower two layers are polarized along the field direction and spins in the upper two layers are anti-aligned to the field but have a finite canting. The canted moments induce a finite magnetization, preventing perfect cancellation of magnetic moments between the upper and lower bilayers. At $h \approx 4$, a partial spin-flop transition occurs, yielding a finite magnetization jump. In this phase, spins in the upper two layers almost form a stripy phase, similar to the phase (I). In the lower two layers, however, all spins attempt to align along the field direction but four spins form a different canting angle with respect to the other four. By smoothly changing those canting angles, this phase reaches a half-magnetization plateau region, where spins in the upper bilayer form a stripy phase and spins in the lower bilayer are fully polarized. With stronger magnetic fields, there is another partial spin-flop transition exhibiting a finite jump in the magnetization. In phase (V), all 16 spins attempt to be aligned along the field direction but four of them have a different canting angle until their canting angle becomes zero and a fully polarized spin state (VI) is realized at a saturation field $H \approx 14$.

Based on the above analysis, we conclude that competing exchange interactions and anisotropy can in principle lead to several metamagnetic transitions with field. We exemplified this for a specific parameter regime in Eq. and found six distinct phases that may emerge taking into account the most dominant exchange interactions. Future neutron scattering studies will be valuable to confirm such scenario, and lay a firmer foundation for theoretical studies. More detailed theoretical and experimental analyses will be discussed in Ref. , which will also discuss scenarios beyond this minimal model and attempt to connect the magnetic ordering with transport.

# Conclusions

In conclusion, we have shown that high-quality CeAgBi$_{2}$ single crystals (${T_{N} = 6.4}$ K) present a rich field-tuned phase diagram with five metamagnetic transitions at $40$ mK. In contrast to other Ce$TX_{2}$ members, a strongly first-order transition is observed at $\sim 79$ kOe in the vicinity of the transition to the paramagnetic state. Remarkably, we unveil an anomalous Hall contribution for fields $H<54$ kOe which is drastically altered when $H$ is tuned through a trio of transitions at 57, 78, and 84 kOe, suggesting that the Fermi surface is reconstructed in a subset of the metamagnetic transitions. Our results shed light on hidden properties of CeAgBi$_{2}$ and open new avenues for both experimental and theoretical studies on non-collinear magnetic structures, quantum (bi-)criticality, and Fermi surface effects, to name a few.

[^1]: These authors contributed equally to this work.

[^2]: These authors contributed equally to this work.