# Abstract

Social media are becoming an increasingly important source of information about the public mood regarding issues such as elections, Brexit, stock market, etc. In this paper we focus on sentiment classification of Twitter data. Construction of sentiment classifiers is a standard text mining task, but here we address the question of how to properly evaluate them as there is no settled way to do so. Sentiment classes are ordered and unbalanced, and Twitter produces a stream of time-ordered data. The problem we address concerns the procedures used to obtain reliable estimates of performance measures, and whether the temporal ordering of the training and test data matters. We collected a large set of 1.5 million tweets in 13 European languages. We created 138 sentiment models and out-of-sample datasets, which are used as a gold standard for evaluations. The corresponding 138 in-sample datasets are used to empirically compare six different estimation procedures: three variants of cross-validation, and three variants of sequential validation (where test set always follows the training set). We find no significant difference between the best cross-validation and sequential validation. However, we observe that all cross-validation variants tend to overestimate the performance, while the sequential methods tend to underestimate it. Standard cross-validation with random selection of examples is significantly worse than the blocked cross-validation, and should not be used to evaluate classifiers in time-ordered data scenarios.

# Introduction

Online social media are becoming increasingly important in our society. Platforms such as Twitter and Facebook influence the daily lives of people around the world. Their users create and exchange a wide variety of contents on social media, which presents a valuable source of information about public sentiment regarding social, economic or political issues. In this context, it is important to develop automatic methods to retrieve and analyze information from social media.

In the paper we address the task of sentiment analysis of Twitter data. The task encompasses identification and categorization of opinions (e.g., negative, neutral, or positive) written in quasi-natural language used in Twitter posts. We focus on estimation procedures of the predictive performance of machine learning models used to address this task. Performance estimation procedures are key to understand the generalization ability of the models since they present approximations of how these models will behave on unseen data. In the particular case of sentiment analysis of Twitter data, high volumes of content are continuously being generated and there is no immediate feedback about the true class of instances. In this context, it is fundamental to adopt appropriate estimation procedures in order to get reliable estimates about the performance of the models.

The complexity of Twitter data raises some challenges on how to perform such estimations, as, to the best of our knowledge, there is currently no settled approach to this. Sentiment classes are typically ordered and unbalanced, and the data itself is time-ordered. Taking these properties into account is important for the selection of appropriate estimation procedures.

The Twitter data shares some characteristics of time series and some of static data. A time series is an array of observations at regular or equidistant time points, and the observations are in general dependent on previous observations . On the other hand, Twitter data is time-ordered, but the observations are short texts posted by Twitter users at any time and frequency. It can be assumed that original Twitter posts are not directly dependent on previous posts. However, there is a potential indirect dependence, demonstrated in important trends and events, through influential users and communities, or individual user's habits. These long-term topic drifts are typically not taken into account by the sentiment analysis models.

We study different performance estimation procedures for sentiment analysis in Twitter data. These estimation procedures are based on (**i**) cross-validation and (**ii**) sequential approaches typically adopted for time series data. On one hand, cross-validations explore all the available data, which is important for the robustness of estimates. On the other hand, sequential approaches are more realistic in the sense that estimates are computed on a subset of data always subsequent to the data used for training, which means that they take time-order into account.

Our experimental study is performed on a large collection of nearly 1.5 million Twitter posts, which are domain-free and in 13 different languages. A realistic scenario is emulated by partitioning the data into 138 datasets by language and time window. Each dataset is split into an in-sample (a training plus test set), where estimation procedures are applied to approximate the performance of a model, and an out-of-sample used to compute the gold standard. Our goal is to understand the ability of each estimation procedure to approximate the true error incurred by a given model on the out-of-sample data.

The paper is structured as follows. provides an overview of the state-of-the-art in estimation methods. In section  we describe the experimental setting for an empirical comparison of estimation procedures for sentiment classification of time-ordered Twitter data. We describe the Twitter sentiment datasets, a machine learning algorithm we employ, performance measures, and how the gold standard and estimation results are produced. In section  we present and discuss the results of comparisons of the estimation procedures along several dimensions. provide the limitations of our work and give directions for the future.

# Related work

In this section we briefly review typical estimation methods used in sentiment classification of Twitter data. In general, for time-ordered data, the estimation methods used are variants of cross-validation, or are derived from the methods used to analyze time series data. We examine the state-of-the-art of these estimation methods, pointing out their advantages and drawbacks.

Several works in the literature on sentiment classification of Twitter data employ standard cross-validation procedures to estimate the performance of sentiment classifiers. For example, Agarwal et al. and Mohammad et al. propose different methods for sentiment analysis of Twitter data and estimate their performance using 5-fold and 10-fold cross-validation, respectively. Bermingham and Smeaton produce a comparative study of sentiment analysis between blogs and Twitter posts, where models are compared using 10-fold cross-validation. Saif et al. asses binary classification performance of nine Twitter sentiment datasets by 10-fold cross validation. Other, similar applications of cross-validation are given in .

On the other hand, there are also approaches that use methods typical for time series data. For example, Bifet and Frank use the prequential (predictive sequential) method to evaluate a sentiment classifier on a stream of Twitter posts. Moniz et al. present a method for predicting the popularity of news from Twitter data and sentiment scores, and estimate its performance using a sequential approach in multiple testing periods.

The idea behind the $K$-fold cross-validation is to randomly shuffle the data and split it in $K$ equally-sized folds. Each fold is a subset of the data randomly picked for testing. Models are trained on the $K-1$ folds and their performance is estimated on the left-out fold. $K$-fold cross-validation has several practical advantages, such as an efficient use of all the data. However, it is also based on an assumption that the data is independent and identically distributed which is often not true. For example, in time-ordered data, such as Twitter posts, the data are to some extent dependent due to the underlying temporal order of tweets. Therefore, using $K$-fold cross-validation means that one uses future information to predict past events, which might hinder the generalization ability of models.

There are several methods in the literature designed to cope with dependence between observations. The most common are sequential approaches typically used in time series forecasting tasks. Some variants of $K$-fold cross-validation which relax the independence assumption were also proposed. For time-ordered data, an estimation procedure is sequential when testing is always performed on the data subsequent to the training set. Typically, the data is split into two parts, where the first is used to train the model and the second is held out for testing. These approaches are also known in the literature as the out-of-sample methods .

Within sequential estimation methods one can adopt different strategies regarding train/test splitting, growing or sliding window setting, and eventual update of the models. In order to produce reliable estimates and test for robustness, Tashman recommends employing these strategies in multiple testing periods. One should either create groups of data series according to, for example, different business cycles , or adopt a randomized approach, such as in . A more complete overview of these approaches is given by Tashman .

In stream mining, where a model is continuously updated, the most commonly used estimation methods are holdout and prequential . The prequential strategy uses an incoming observation to first test the model and then to train it.

Besides sequential estimation methods, some variants of $K$-fold cross-validation were proposed in the literature that are specially designed to cope with dependency in the data and enable the application of cross-validation to time-ordered data. For example, blocked cross-validation (the name is adopted from Bergmeir ) was proposed by Snijders . The method derives from a standard $K$-fold cross-validation, but there is no initial random shuffling of observations. This renders $K$ blocks of contiguous observations.

The problem of data dependency for cross-validation is addressed by McQuarrie and Tsai . The modified cross-validation removes observations from the training set that are dependent with the test observations. The main limitation of this method is its inefficient use of the available data since many observations are removed, as pointed out in . The method is also known as non-dependent cross-validation .

The applicability of variants of cross-validation methods in time series data, and their advantages over traditional sequential validations are corroborated by Bergmeir et al. . The authors conclude that in time series forecasting tasks, the blocked cross-validations yield better error estimates because of their more efficient use of the available data. Cerqueira et al. compare performance estimation of various cross-validation and out-of-sample approaches on real-world and synthetic time series data. The results indicate that cross-validation is appropriate for the stationary synthetic time series data, while the out-of-sample approaches yield better estimates for real-world data.

Our contribution to the state-of-the-art is a large scale empirical comparison of several estimation procedures on Twitter sentiment data. We focus on the differences between the cross-validation and sequential validation methods, to see how important is the violation of data independence in the case of Twitter posts. We consider longer-term time-dependence between the training and test sets, and completely ignore finer-scale dependence at the level of individual tweets (e.g., retweets and replies). To the best of our knowledge, there is no settled approach yet regarding proper validation of models for Twitter time-ordered data. This work provides some results which contribute to bridging that gap.

# Methods and experiments

The goal of this study is to recommend appropriate estimation procedures for sentiment classification of Twitter time-ordered data. We assume a static sentiment classification model applied to a stream of Twitter posts. In a real-case scenario, the model is trained on historical, labeled tweets, and applied to the current, incoming tweets. We emulate this scenario by exploring a large collection of nearly 1.5 million manually labeled tweets in 13 European languages (see subsection ). Each language dataset is split into pairs of the in-sample data, on which a model is trained, and the out-of-sample data, on which the model is validated. The performance of the model on the out-of-sample data gives an estimate of its performance on the future, unseen data. Therefore, we first compute a set of 138 out-of-sample performance results, to be used as a gold standard (subsection ). In effect, our goal is to find the estimation procedure that best approximates this out-of-sample performance.

Throughout our experiments we use only one training algorithm (subsection ), and two performance measures (subsection ). During training, the performance of the trained model can be estimated only on the in-sample data. However, there are different estimation procedures which yield these approximations. In machine learning, a standard procedure is cross-validation, while for time-ordered data, sequential validation is typically used. In this study, we compare three variants of cross-validation and three variants of sequential validation (subsection ). The goal is to find the in-sample estimation procedure that best approximates the out-of-sample gold standard. The error an estimation procedure makes is defined as the difference to the gold standard.

## Data and models

We collected a large corpus of nearly 1.5 million Twitter posts written in 13 European languages. This is, to the best of our knowledge, by far the largest set of sentiment labeled tweets publicly available. We engaged native speakers to label the tweets based on the sentiment expressed in them. The sentiment label has three possible values: negative, neutral or positive. It turned out that the human annotators perceived the values as ordered. The quality of annotations varies though, and is estimated from the self- and inter-annotator agreements. All the details about the datasets, the annotator agreements, and the ordering of sentiment values are in our previous study . The sentiment distribution and quality of individual language datasets is in Table . The tweets in the datasets are ordered by tweet ids, which corresponds to ordering by the time of posting.

| Language    |     | Negative | Neutral | Positive |     Total | Quality |
|:------------|:----|---------:|--------:|---------:|----------:|:--------|
| Albanian    | alb |    7,062 |  15,066 |   23,630 |    45,758 | poor    |
| Bulgarian   | bul |   14,374 |  28,961 |   19,932 |    63,267 | fair    |
| English     | eng |   23,250 |  38,457 |   25,721 |    87,428 | v.good  |
| German      | ger |   19,039 |  52,166 |   26,743 |    97,948 | fair    |
| Hungarian   | hun |    9,062 |  17,833 |   30,410 |    57,305 | good    |
| Polish      | pol |   59,027 |  48,658 |   84,245 |   191,930 | good    |
| Portuguese  | por |   56,008 |  53,026 |   43,009 |   152,043 | fair    |
| Russian     | rus |   30,249 |  37,401 |   25,671 |    93,321 | good    |
| Ser/Cro/Bos | scb |   58,796 |  61,265 |   73,766 |   193,827 | fair    |
| Slovak      | slk |   15,060 |  13,112 |   30,598 |    58,770 | good    |
| Slovenian   | slv |   34,164 |  48,458 |   30,210 |   112,832 | good    |
| Spanish     | spa |   27,675 |  88,481 |  117,048 |   233,204 | poor    |
| Swedish     | swe |   22,381 |  15,387 |   13,630 |    51,398 | good    |
| Total       |     |  376,147 | 518,271 |  544,613 | 1,439,031 |         |

Sentiment label distribution of Twitter datasets in 13 languages. The last column is a qualitative assessment of the annotation quality, based on the levels of the self- and inter-annotator agreement.

There are many supervised machine learning algorithms suitable for training sentiment classification models from labeled tweets. In this study we use a variant of Support Vector Machine (SVM) . The basic SVM is a two-class, binary classifier. In the training phase, SVM constructs a hyperplane in a high-dimensional vector space that separates one class from the other. In the classification phase, the side of the hyperplane determines the class. A two-class SVM can be extended into a multi-class classifier which takes the ordering of sentiment values into account, and implements ordinal classification . Such an extension consists of two SVM classifiers: one classifier is trained to separate the negative examples from the neutral-or-positives; the other separates the negative-or-neutrals from the positives. The result is a classifier with two hyperplanes, which partitions the vector space into three subspaces: negative, neutral, and positive. During classification, the distances from both hyperplanes determine the predicted class. A further refinement is a **TwoPlaneSVMbin** classifier. It partitions the space around both hyperplanes into bins, and computes the distribution of the training examples in individual bins. During classification, the distances from both hyperplanes determine the appropriate bin, but the class is determined as the majority class in the bin.

The vector space is defined by the features extracted from the Twitter posts. The posts are first pre-processed by standard text processing methods, i.e., tokenization, stemming/lemmatization (if available for a specific language), unigram and bigram construction, and elimination of terms that do not appear at least 5 times in a dataset. The Twitter specific pre-processing is then applied, i.e, replacing URLs, Twitter usernames and hashtags with common tokens, adding emoticon features for different types of emoticons in tweets, handling of repetitive letters, etc. The feature vectors are then constructed by the Delta TF-IDF weighting scheme .

In our previous study we compared five variants of the SVM classifiers and Naive Bayes on the Twitter sentiment classification task. TwoPlaneSVMbin was always between the top, but statistically indistinguishable, best performing classifiers. It turned out that monitoring the quality of the annotation process has much larger impact on the performance than the type of the classifier used. In this study we fix the classifier, and use TwoPlaneSVMbin in all the experiments.

## Performance measures

Sentiment values are ordered, and distribution of tweets between the three sentiment classes is often unbalanced. In such cases, *accuracy* is not the most appropriate performance measure . In this context, we evaluate performance with the following two metrics: Krippendorff's $\mathit{Alpha}$  , and $\overline{F_{1}}$  .

$\mathit{Alpha}$  was developed to measure the agreement between human annotators, but can also be used to measure the agreement between classification models and a gold standard. It generalizes several specialized agreement measures, takes ordering of classes into account, and accounts for the agreement by chance. $\mathit{Alpha}$  is defined as follows: $$\mathit{Alpha} = 1 - \frac{D_{o}}{D_{e}} \,$$ where $D_{o}$ is the observed disagreement between models, and $D_{e}$ is a disagreement, expected by chance. When models agree perfectly, $\mathit{Alpha}$`<!-- -->`{=html}$=1$, and when the level of agreement equals the agreement by chance, $\mathit{Alpha}$`<!-- -->`{=html}$=0$. Note that $\mathit{Alpha}$  can also be negative. The two disagreement measures are defined as: $$D_{o} = \frac{1}{N} \sum_{c,c'} N(c,c') \cdot \delta^2(c,c') \,,$$ $$D_{e} = \frac{1}{N(N-1)} \sum_{c,c'} N(c) \cdot N(c') \cdot \delta^2(c,c') \,.$$ The arguments, $N, N(c,c'), N(c)$, and $N(c')$, refer to the frequencies in a coincidence matrix, defined below. $c$ (and $c'$) is a discrete sentiment variable with three possible values: *negative* ($-1$), *neutral* (0), or *positive* ($+1$). $\delta(c,c')$ is a difference function between the values of $c$ and $c'$, for ordered variables defined as: $$\delta(c,c') = |c - c'| \;\;\;\; c,c'\in \{-1,0,+1\} \;.$$ Note that disagreements $D_{o}$ and $D_{e}$ between the extreme classes (*negative* and *positive*) are four times larger than between the neighbouring classes.

A coincidence matrix tabulates all pairable values of $c$ from two models. In our case, we have a $3$-by-$3$ coincidence matrix, and compare a model to the gold standard. The coincidence matrix is then the sum of the confusion matrix and its transpose. Each labeled tweet is entered twice, once as a $(c,c')$ pair, and once as a $(c',c)$ pair. $N(c,c')$ is the number of tweets labeled by the values $c$ and $c'$ by different models, $N(c)$ and $N(c')$ are the totals for each value, and $N$ is the grand total.

$\overline{F_{1}}$  is an instance of the $F$ score, a well-known performance measure in information retrieval and machine learning. We use an instance specifically designed to evaluate the 3-class sentiment models . $\overline{F_{1}}$  is defined as follows: $$\overline{F_1} = \frac{F_1(-1) + F_1(+1)}{2} \,.$$ $\overline{F_{1}}$  implicitly takes into account the ordering of sentiment values, by considering only the extreme labels, *negative* $(-1)$ and *positive* $(+1)$. The middle, *neutral*, is taken into account only indirectly. $F_{1}(c)$ is the harmonic mean of precision and recall for class $c$, $c\in \{-1,+1\}$. $\overline{F_{1}}$`<!-- -->`{=html}$=1$ implies that all negative and positive tweets were correctly classified, and as a consequence, all neutrals as well. $\overline{F_{1}}$`<!-- -->`{=html}$=0$ indicates that all negative and positive tweets were incorrectly classified. $\overline{F_{1}}$  does not account for correct classification by chance.

## Gold standard

We create the gold standard results by splitting the data into the in-sample datasets (abbreviated as in-set), and out-of-sample datasets (abbreviated as out-set). The terminology of the in- and out-set is adopted from Bergmeir et al. . Tweets are ordered by the time of posting. To emulate a realistic scenario, an out-set always follows the in-set. From each language dataset (Table ) we create $L$ in-sets of varying length in multiples of 10,000 consecutive tweets, where $L = \lfloor N/10000 \rfloor$. The out-set is the subsequent 10,000 consecutive tweets, or the remainder at the end of each language dataset. This is illustrated in Figure .

The partitioning of the language datasets results in 138 in-sets and corresponding out-sets. For each in-set, we train a TwoPlaneSVMbin sentiment classification model, and measure its performance, in terms of $\mathit{Alpha}$  and $\overline{F_{1}}$, on the corresponding out-set. The results are in Tables  and . Note that the performance measured by $\mathit{Alpha}$  is considerably lower in comparison to $\overline{F_{1}}$, since the baseline for $\mathit{Alpha}$  is classification by chance.

```latex
\resizebox{\textwidth}{!}{
    \begin{tabular}{rrrrrrrrrrrrr}
    \hline
    alb & bul & eng & ger & hun & pol & por & rus & scb & slk & slv & spa & swe \\
    \hline
    0.210 & 0.321 & 0.414 & 0.391 & 0.419 & 0.409 & 0.338 & 0.369 & 0.275 & 0.367 & 0.327 & 0.171 & 0.470 \\
    0.102 & 0.324 & 0.433 & 0.420 & 0.453 & 0.432 & 0.336 & 0.420 & 0.393 & 0.411 & 0.380 & 0.222 & 0.463 \\
    0.084 & 0.339 & 0.449 & 0.423 & 0.482 & 0.479 & 0.360 & 0.441 & 0.408 & 0.425 & 0.414 & 0.255 & 0.458 \\
    0.106 & 0.363 & 0.474 & 0.416 & 0.460 & 0.499 & 0.428 & 0.435 & 0.457 & 0.438 & 0.439 & 0.269 & 0.473 \\
          & 0.375 & 0.513 & 0.387 & 0.475 & 0.486 & 0.183 & 0.478 & 0.421 & 0.454 & 0.453 & 0.211 & 0.480 \\
          & 0.397 & 0.513 & 0.403 &       & 0.487 & 0.176 & 0.452 & 0.327 &       & 0.478 & 0.227 &  \\
          &       & 0.541 & 0.406 &       & 0.483 & 0.224 & 0.492 & 0.293 &       & 0.455 & 0.226 &  \\
          &       & 0.526 & 0.354 &       & 0.512 & 0.333 & 0.474 & 0.341 &       & 0.418 & 0.227 &  \\
          &       &       & 0.351 &       & 0.467 & 0.388 & 0.489 & 0.358 &       & 0.425 & 0.151 &  \\
          &       &       &       &       & 0.513 & 0.409 &       & 0.384 &       & 0.418 & 0.193 &  \\
          &       &       &       &       & 0.491 & 0.425 &       & 0.382 &       & 0.320 & 0.196 &  \\
          &       &       &       &       & 0.526 & 0.434 &       & 0.485 &       &       & 0.220 &  \\
          &       &       &       &       & 0.549 & 0.439 &       & 0.528 &       &       & 0.233 &  \\
          &       &       &       &       & 0.535 & 0.453 &       & 0.551 &       &       & 0.207 &  \\
          &       &       &       &       & 0.541 & 0.472 &       & 0.512 &       &       & 0.202 &  \\
          &       &       &       &       & 0.500 &       &       & 0.533 &       &       & 0.179 &  \\
          &       &       &       &       & 0.544 &       &       & 0.418 &       &       & 0.159 &  \\
          &       &       &       &       & 0.532 &       &       & 0.514 &       &       & 0.207 &  \\
          &       &       &       &       & 0.528 &       &       & 0.479 &       &       & 0.216 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.251 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.241 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.110 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.142 &  \\
    \hline
    \end{tabular}
    }
```

```latex
\resizebox{\textwidth}{!}{
    \begin{tabular}{rrrrrrrrrrrrr}
    \hline
    alb & bul & eng & ger & hun & pol & por & rus & scb & slk & slv & spa & swe \\
    \hline
    0.479 & 0.509 & 0.545 & 0.578 & 0.610 & 0.621 & 0.356 & 0.551 & 0.492 & 0.616 & 0.485 & 0.436 & 0.627 \\
    0.396 & 0.501 & 0.567 & 0.595 & 0.624 & 0.632 & 0.358 & 0.560 & 0.569 & 0.657 & 0.533 & 0.452 & 0.620 \\
    0.387 & 0.498 & 0.571 & 0.588 & 0.637 & 0.653 & 0.383 & 0.572 & 0.577 & 0.669 & 0.567 & 0.504 & 0.629 \\
    0.388 & 0.510 & 0.595 & 0.561 & 0.628 & 0.670 & 0.449 & 0.571 & 0.626 & 0.670 & 0.593 & 0.473 & 0.630 \\
          & 0.513 & 0.634 & 0.533 & 0.640 & 0.651 & 0.243 & 0.604 & 0.580 & 0.675 & 0.603 & 0.446 & 0.658 \\
          & 0.535 & 0.640 & 0.537 &       & 0.663 & 0.252 & 0.588 & 0.485 &       & 0.624 & 0.454 &  \\
          &       & 0.654 & 0.529 &       & 0.656 & 0.322 & 0.617 & 0.469 &       & 0.550 & 0.440 &  \\
          &       & 0.647 & 0.409 &       & 0.682 & 0.448 & 0.610 & 0.493 &       & 0.521 & 0.438 &  \\
          &       &       & 0.413 &       & 0.654 & 0.529 & 0.614 & 0.503 &       & 0.524 & 0.429 &  \\
          &       &       &       &       & 0.672 & 0.556 &       & 0.526 &       & 0.507 & 0.424 &  \\
          &       &       &       &       & 0.659 & 0.589 &       & 0.573 &       & 0.415 & 0.412 &  \\
          &       &       &       &       & 0.680 & 0.605 &       & 0.654 &       &       & 0.407 &  \\
          &       &       &       &       & 0.696 & 0.608 &       & 0.686 &       &       & 0.431 &  \\
          &       &       &       &       & 0.679 & 0.624 &       & 0.696 &       &       & 0.398 &  \\
          &       &       &       &       & 0.682 & 0.638 &       & 0.665 &       &       & 0.403 &  \\
          &       &       &       &       & 0.650 &       &       & 0.684 &       &       & 0.402 &  \\
          &       &       &       &       & 0.670 &       &       & 0.644 &       &       & 0.390 &  \\
          &       &       &       &       & 0.663 &       &       & 0.661 &       &       & 0.446 &  \\
          &       &       &       &       & 0.663 &       &       & 0.625 &       &       & 0.479 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.516 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.516 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.423 &  \\
          &       &       &       &       &       &       &       &       &       &       & 0.449 &  \\
    \hline
    \end{tabular}
    }
```

The 138 in-sets are used to train sentiment classification models and estimate their performance. The goal of this study is to analyze different estimation procedures in terms of how well they approximate the out-set gold standard results shown in Tables  and .

## Estimation procedures

There are different estimation procedures, some more suitable for static data, while others are more appropriate for time-series data. Time-ordered Twitter data shares some properties of both types of data. When training an SVM model, the order of tweets is irrelevant and the model does not capture the dynamics of the data. When applying the model, however, new tweets might introduce new vocabulary and topics. As a consequence, the temporal ordering of training and test data has a potential impact on the performance estimates.

We therefore compare two classes of estimation procedures. Cross-validation, commonly used in machine learning for model evaluation on static data, and sequential validation, commonly used for time-series data. There are many variants and parameters for each class of procedures. Our datasets are relatively large and an application of each estimation procedure takes several days to complete. We have selected three variants of each procedure to provide answers to some relevant questions.

First, we apply 10-fold cross-validation where the training:test set ratio is always 9:1. Cross-validation is *stratified* when the fold partitioning is not completely random, but each fold has roughly the same class distribution. We also compare standard *random* selection of examples to the *blocked* form of cross-validation , where each fold is a block of consecutive tweets. We use the following abbreviations for cross-validations:

-   **xval(9:1, strat, block)** - 10-fold, stratified, blocked;

-   **xval(9:1, no-strat, block)** - 10-fold, not stratified, blocked;

-   **xval(9:1, strat, rand)** - 10-fold, stratified, random selection of examples.

In sequential validation, a sample consists of the training set immediately followed by the test set. We vary the ratio of the training and test set sizes, and the number and distribution of samples taken from the in-set. The number of samples is 10 or 20, and they are distributed equidistantly or semi-equidistantly. In all variants, samples cover the whole in-set, but they are overlapping. See Figure  for illustration. We use the following abbreviations for sequential validations:

-   **seq(9:1, 20, equi)** - 9:1 training:test ratio, 20 equidistant samples,

-   **seq(9:1, 10, equi)** - 9:1 training:test ratio, 10 equidistant samples,

-   **seq(2:1, 10, semi-equi)** - 2:1 training:test ratio, 10 samples randomly selected out of 20 equidistant points.

# Results and discussion

We compare six estimation procedures in terms of different types of errors they incur. The error is defined as the difference to the gold standard. First, the magnitude and sign of the errors show whether a method tends to underestimate or overestimate the performance, and by how much (subsection ). Second, relative errors give fractions of small, moderate, and large errors that each procedure incurs (subsection ). Third, we rank the estimation procedures in terms of increasing absolute errors, and estimate the significance of the overall ranking by the Friedman-Nemenyi test (subsection ). Finally, selected pairs of estimation procedures are compared by the Wilcoxon signed-rank test (subsection ).

## Median errors

An estimation procedure estimates the performance  (abbreviated $Est$) of a model in terms of $\mathit{Alpha}$  and $\overline{F_{1}}$. The error it incurs is defined as the difference to the gold standard performance (abbreviated $Gold$): $Err = Est - Gold$. The validation results show high variability of the errors, with skewed distribution and many outliers. Therefore, we summarize the errors in terms of their medians and quartiles, instead of the averages and variances.

The median errors of the six estimation procedures are in Tables  and , measured by $\mathit{Alpha}$  and $\overline{F_{1}}$, respectively.

```latex
\resizebox{\textwidth}{!}{
    \begin{tabular}{l *{6}{S[table-format=-1.3]}}
    \hline
    Lang &
    \pbox{1.9cm}{xval(9:1,\\strat, block)} &
    \pbox{2.5cm}{xval(9:1,\\no-strat, block)}  &
    \pbox{1.8cm}{xval(9:1,\\strat, rand)} &
    \pbox{1.8cm}{seq(9:1,\\20, equi)} &
    \pbox{1.8cm}{seq(9:1,\\10, equi)} &
    \pbox{2.5cm}{seq(2:1,\\10, semi-equi)} \\
\hline
alb&0.052&0.036&0.206&0.001&0.001&0.001\tabularnewline
bul&0.009&0.013&0.046&-0.019&-0.025&-0.043\tabularnewline
eng&-0.016&-0.017&-0.010&-0.040&-0.042&-0.039\tabularnewline
ger&0.037&0.049&0.059&0.009&0.010&0.001\tabularnewline
hun&0.009&0.013&0.025&-0.011&-0.007&-0.007\tabularnewline
pol&0.011&0.016&0.054&-0.020&-0.017&-0.031\tabularnewline
por&-0.048&-0.048&-0.015&-0.040&-0.045&-0.085\tabularnewline
rus&0.008&0.008&0.029&-0.027&-0.029&-0.045\tabularnewline
scb&-0.046&-0.051&0.026&-0.047&-0.043&-0.069\tabularnewline
slk&0.018&0.015&0.055&-0.025&-0.023&-0.039\tabularnewline
slv&0.003&-0.004&0.040&-0.029&-0.026&-0.031\tabularnewline
spa&-0.008&0.031&0.070&0.012&0.011&-0.011\tabularnewline
swe&0.055&0.057&0.106&0.011&0.006&-0.028\tabularnewline
\hline
Median&0.009&0.013&0.046&-0.020&-0.023&-0.031\tabularnewline
\hline
\end{tabular}
}
```

```latex
\resizebox{\textwidth}{!}{%
    \begin{tabular}{l *{6}{S[table-format=-1.3]}}
\hline
    Lang &
    \pbox{1.9cm}{xval(9:1,\\strat, block)} &
    \pbox{2.5cm}{xval(9:1,\\no-strat, block)}  &
    \pbox{1.8cm}{xval(9:1,\\strat, rand)} &
    \pbox{1.8cm}{seq(9:1,\\20, equi)} &
    \pbox{1.8cm}{seq(9:1,\\10, equi)} &
    \pbox{2.5cm}{seq(2:1,\\10, semi-equi)} \\
\hline
alb&0.026&0.016&0.137&-0.014&-0.007&-0.009\tabularnewline
bul&0.020&0.024&0.047&0.003&-0.002&-0.019\tabularnewline
eng&-0.019&-0.020&-0.015&-0.027&-0.027&-0.028\tabularnewline
ger&0.056&0.058&0.072&0.025&0.028&0.014\tabularnewline
hun&0.022&0.022&0.030&-0.006&-0.009&-0.005\tabularnewline
pol&0.013&0.020&0.044&-0.001&0&-0.007\tabularnewline
por&-0.050&-0.045&-0.040&-0.049&-0.056&-0.092\tabularnewline
rus&0.008&0.010&0.025&-0.019&-0.018&-0.021\tabularnewline
scb&-0.034&-0.037&0&-0.030&-0.032&-0.050\tabularnewline
slk&0.005&0.008&0.025&-0.013&-0.015&-0.013\tabularnewline
slv&0.003&0&0.029&-0.022&-0.026&-0.032\tabularnewline
spa&-0.001&0.024&0.060&0.007&0.010&0.012\tabularnewline
swe&0.030&0.037&0.071&0.008&0.006&-0.011\tabularnewline
\hline
Median&0.008&0.016&0.030&-0.013&-0.009&-0.013\tabularnewline
\hline
\end{tabular}
}
```

Figure depicts the errors with box plots. The band inside the box denotes the median, the box spans the second and third quartile, and the whiskers denote 1.5 interquartile range. The dots correspond to the outliers. Figure shows high variability of errors for individual datasets. This is most pronounced for the Serbian/Croatian/Bosnian (scb) and Portuguese (por) datasets where variation in annotation quality (scb) and a radical topic shift (por) were observed. Higher variability is also observed for the Spanish (spa) and Albanian (alb) datasets, which have poor sentiment annotation quality (see for details).

The differences between the estimation procedures are easier to detect when we aggregate the errors over all language datasets. The results are in Figures and , for $\mathit{Alpha}$  and $\overline{F_{1}}$, respectively. In both cases we observe that the cross-validation procedures (xval) consistently overestimate the performance, while the sequential validations (seq) underestimate it. The largest overestimation errors are incurred by the random cross-validation, and the largest underestimations by the sequential validation with the training:test set ratio 2:1. We also observe high variability of errors, with many outliers. The conclusions are consistent for both measures, $\mathit{Alpha}$  and $\overline{F_{1}}$.

## Relative errors

Another useful analysis of estimation errors is provided by a comparison of relative errors. The relative error is the absolute error an estimation procedure incurs divided by the gold standard result: $RelErr = |Est - Gold| / Gold$. We chose two, rather arbitrary, thresholds of 5% and 30%, and classify the relative errors as small ($RelErr < 5\%$), moderate ($5\% \le RelErr \le 30\%$), and large ($RelErr > 30\%$).

Figure shows the proportion of the three types of errors, measured by $\mathit{Alpha}$, for individual language datasets. Again, we observe a higher proportion of large errors for languages with poor annotations (alb, spa), annotations of different quality (scb), and different topics (por).

Figures and aggregate the relative errors across all the datasets, for $\mathit{Alpha}$  and $\overline{F_{1}}$, respectively. The proportion of errors is consistent between $\mathit{Alpha}$  and $\overline{F_{1}}$, but there are more large errors when the performance is measured by $\mathit{Alpha}$. This is due to smaller error magnitude when the performance is measured by $\mathit{Alpha}$  in contrast to $\overline{F_{1}}$, since $\mathit{Alpha}$  takes classification by chance into account. With respect to individual estimation procedures, there is a considerable divergence of the random cross-validation. For both performance measures, $\mathit{Alpha}$  and $\overline{F_{1}}$, it consistently incurs higher proportion of large errors and lower proportion of small errors in comparison to the rest of the estimation procedures.

## Friedman test

The Friedman test is used to compare multiple procedures over multiple datasets . For each dataset, it ranks the procedures by their performance. It tests the null hypothesis that the average ranks of the procedures across all the datasets are equal. If the null hypothesis is rejected, one applies the Nemenyi post-hoc test on pairs of procedures. The performance of two procedures is significantly different if their average ranks differ by at least the critical difference. The critical difference depends on the number of procedures to compare, the number of different datasets, and the selected significance level.

In our case, the performance of an estimation procedure is taken as the absolute error it incurs: $AbsErr = |Est - Gold|$. The estimation procedure with the lowest absolute error gets the lowest (best) rank. The results of the Friedman-Nemenyi test are in Figures and , for $\mathit{Alpha}$  and $\overline{F_{1}}$, respectively.

For both performance measures, $\mathit{Alpha}$  and $\overline{F_{1}}$, the Friedman rankings are the same. For six estimation procedures, 13 language datasets, and the 5% significance level, the critical difference is $2.09$. In the case of $\overline{F_{1}}$  (Figure ) all six estimation procedures are within the critical difference, so their ranks are not significantly different. In the case of $\mathit{Alpha}$  (Figure ), however, the two best methods are significantly better than the random cross-validation.

## Wilcoxon test

The Wilcoxon signed-rank test is used to compare two procedures on related data . It ranks the differences in performance of the two procedures, and compares the ranks for the positive and negative differences. Greater differences count more, but the absolute magnitudes are ignored. It tests the null hypothesis that the differences follow a symmetric distribution around zero. If the null hypothesis is rejected one can conclude that one procedure outperforms the other at a selected significance level.

In our case, the performance of pairs of estimation procedures is compared at the level of language datasets. The absolute errors of an estimation procedure are averaged across the in-sets of a language. The average absolute error is then $AvgAbsErr = \sum |Est - Gold| / L$, where $L$ is the number of in-sets. The results of the Wilcoxon test, for selected pairs of estimation procedures, for both $\mathit{Alpha}$  and $\overline{F_{1}}$, are in Figure .

The Wilcoxon test results confirm and reinforce the main results of the previous sections. Among the cross-validation procedures, blocked cross-validation is consistently better than the random cross-validation, at the 1% significance level. Stratified approach is better than non-stratified, but significantly (5% level) only for $\overline{F_{1}}$. The comparison of the sequential validation procedures is less conclusive. The training:test set ratio 9:1 is better than 2:1, but significantly (at the 5% level) only for $\mathit{Alpha}$. With the ratio 9:1 fixed, 20 samples yield better performance estimates than 10 samples, but significantly (5% level) only for $\overline{F_{1}}$. We found no significant difference between the best cross-validation and sequential validation procedures in terms how well they estimate the average absolute errors.

# Conclusions

In this paper we present an extensive empirical study about the performance estimation procedures for sentiment analysis of Twitter data. Currently, there is no settled approach on how to properly evaluate models in such a scenario. Twitter time-ordered data shares some properties of static data for text mining, and some of time series data. Therefore, we compare estimation procedures developed for both types of data.

The main result of the study is that standard, random cross-validation should not be used when dealing with time-ordered data. Instead, one should use blocked cross-validation, a conclusion already corroborated by Bergmeir et al. . Another result is that we find no significant differences between the blocked cross-validation and the best sequential validation. However, we do find that cross-validations typically overestimate the performance, while sequential validations underestimate it.

The results are robust in the sense that we use two different performance measures, several comparisons and tests, and a very large collection of data. To the best of our knowledge, we analyze and provide by far the largest set of manually sentiment-labeled tweets publicly available.

There are some biased decisions in our creation of the gold standard though, which limit the generality of the results reported, and should be addressed in the future work. An out-set always consists of 10,000 tweets, and immediately follows the in-sets. We do not consider how the performance drops over longer out-sets, nor how frequently should a model be updated. More importantly, we intentionally ignore the issue of dependent observations, between the in- and out-sets, and between the training and test sets. In the case of tweets, short-term dependencies are demonstrated in the form of retweets and replies. Medium- and long-term dependencies are shaped by periodic events, influential users and communities, or individual user's habits. When this is ignored, the model performance is likely overestimated. Since we do this consistently, our comparative results still hold. The issue of dependent observations was already addressed for blocked cross-validation by removing adjacent observations between the training and test sets, thus effectively creating a gap between the two. Finally, it should be noted that different Twitter language datasets are of different sizes and annotation quality, belong to different time periods, and that there are time periods in the datasets without any manually labeled tweets.

# Data and code availability

All Twitter data were collected through the public Twitter API and are subject to the Twitter terms and conditions. The Twitter language datasets are available in a public language resource repository <span class="smallcaps">clarin.si</span> at <http://hdl.handle.net/11356/1054>, and are described in . There are 15 language files, where the Serbian/Croatian/Bosnian dataset is provided as three separate files for the constituent languages. For each language and each labeled tweet, there is the tweet ID (as provided by Twitter), the sentiment label (negative, neutral, or positive), and the annotator ID (anonymized). Note that Twitter terms do not allow to openly publish the original tweets, they have to be fetched through the Twitter API. Precise details how to fetch the tweets, given tweet IDs, are provided in Twitter API documentation <https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup>. However, upon request to the corresponding author, a bilateral agreement on the joint use of the original data can be reached.

The TwoPlaneSVMbin classifier and several other machine learning algorithms are implemented in an open source LATINO library . LATINO is a light-weight set of software components for building text mining applications, openly available at <https://github.com/latinolib>.

All the performance results, for gold standard and the six estimation procedures, are provided in a form which allows for easy reproduction of the presented results. The **R** code and data files needed to reproduce all the figures and tables in the paper are available at <http://ltorgo.github.io/TwitterDS/>.

# Acknowledgements

Igor Mozetič and Jasmina Smailović acknowledge financial support from the H2020 FET project DOLFINS (grant no. 640772), and the Slovenian Research Agency (research core funding no. P2-0103).

Luis Torgo and Vitor Cerqueira acknowledge financing by project "Coral - Sustainable Ocean Exploitation: Tools and Sensors/NORTE-01-0145-FEDER-000036", financed by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).

We thank Miha Grčar and Sašo Rutar for valuable discussions and implementation of the LATINO library.


abstract: The unequivocal demonstration of temporal or spatial variability in a fundamental constant of nature would be of enormous significance. Recent attempts to measure the variability of the fine structure constant $\alpha$ over cosmological time, using high resolution spectra of high redshift quasars observed with 10m class telescopes, have produced conflicting results. We use the Many Multiplet (MM) method with and lines on very high signal-to-noise, high resolution ($R = 72,000$) Keck HIRES spectra of eight narrow quasar absorption systems. We consider both systematic uncertainties in spectrograph wavelength calibration and also velocity offsets introduced by complex velocity structure in even apparently simple and weak narrow lines and analyze their effect on claimed variations in $\alpha$. We find no significant change in $\alpha$, $\Delta\alpha/\alpha =(0.43\pm0.34) \times 10^{-5}$, in the redshift range $z=0.7-1.5$, where this includes both statistical and systematic errors.
  .
  We also show that the scatter in measurements of $\Delta\alpha/\alpha$ arising from absorption line structure can be considerably larger than assigned statistical errors even for apparently simple and narrow absorption systems. We find a null result of $\Delta\alpha/\alpha = (-0.59\pm0.55) \times 10^{-5}$ in a system at $z=1.7382$ using lines of , and , whereas using and lines in a system at $z=1.6614$ we find a systematic velocity trend which, if interpreted as a shift in $\alpha$, would correspond to $\Delta\alpha/\alpha = (1.88\pm0.47) \times 10^{-5}$, where both results include both statistical and systematic errors. This latter result is almost certainly caused by varying ionic abundances in subcomponents of the line: using , and in the analysis changes the result to $\Delta\alpha/\alpha = (-0.47\pm 0.53) \times 10^{-5}$. Combining the and results with estimates based on , and gives $\Delta\alpha/\alpha = (-0.01\pm 0.26) \times 10^{-5}$. We conclude that spectroscopic measurements of quasar absorption lines are not yet capable of unambiguously detecting variation in $\alpha$ using the MM method.
author: A. Songaila & L. L. Cowie
title: Constraining the Variation of the Fine Structure Constant with Observations of Narrow Quasar Absorption Lines

# Introduction

There has been much interest over the last decade in the possibility that fundamental dimensionless constants may evolve or in some way be a product of the history of the region in which they lie . In the case of the fine-structure constant, $\alpha$, one group has claimed to see such time variation and spatial variation using the "many multiplet" (MM) method of analyzing quasar absorption line spectra, which they pioneered . However, this positive result is limited to one group and more stringent null results, using the same methodology, have been reported . The early results have been criticized by who claim that their quoted significance is too high though not, in the case of the Chand et al. result, consistent with evidence for varying $\alpha$ . Investigations of the change in $\alpha$ by comparing redshifted radio lines also do not show evidence of time variation . These results provide an alternative source of evidence for a non-varying $\alpha$ that has different systematic effects to the MM method, though the interpretation depends on assuming a low value for the change in $\mu \equiv m_e/m_p$ for which there is direct evidence at $z < 0.9$ . (See e.g. for a summary of results at higher redshift). The discrepancy in published results could be reconciled if systematic effects dominate the measurements, leading to underestimates of the errors. In the present paper we consider both systematic uncertainties in spectrograph wavelength calibration and also velocity offsets introduced by complex velocity structure in even apparently simple and weak narrow lines and analyze their effect on claimed variations in $\alpha$.

The current observational situation regarding variation in $\alpha$ is far from clear. Terrestrial laboratory and geophysical measurements show no sign of any time variation on both long and short time scales. Laboratory measurements of the stability of atomic clocks are consistent with no instantaneous time variation at the present time, while analysis of the OKLO natural fission reactor places a limit of $(-0.11 \le \Delta\alpha/\alpha \le 0.24) \times 10^{-7}$ over a time span of 2 Gyr . Astronomical measurements, on the other hand, have so far given both null results and also the first reported detections of variation. A number of groups have looked for variation in $\alpha$ over cosmic time in quasar absorption lines at high redshift using various methodologies. All such measurements based on comparing the relative wavelength separation of the two members of an alkaline doublet observed at high redshift in a quasar absorption system are consistent with no evolution . It was only with the advent of the more sensitive MM method that a significant change at the level of $\Delta\alpha/\alpha = (-0.57\pm0.11) \times 10^{-5}$ was claimed over the redshift range $0.2 < z < 4.2$ .

The enormous advantage of the MM method is its sensitivity: it is sensitive to $\alpha^{-1}\Delta\alpha$ at a level of about $10^{-5}$ in an individual quasar absorption line system, making it about an order of magnitude better than the alkaline doublet method. On the other hand, this sensitivity is bought at the price of much greater required control of systematic effects, among the most important of which are the accuracy of the wavelength calibrations in the echelle spectrographs being used for the measurements, and the assumptions of uniform velocity structure in the various ion species being compared and uniform spatial abundance patterns . The wavelength calibration is particularly critical since, in contrast to the alkaline doublet method, the lines being compared in the MM method can have wide wavelength separations. While these systematic effects have been extensively investigated and modeled , it is still likely that they do in fact dominate the measurement error and are responsible for the discrepancies in published results.

A suitable choice of ionic species can minimize or exclude some of the potential systematic errors associated with the MM method. In particular, concerns about systematics in the wavelength calibration can be reduced by using lines of singly ionised Cr and Zn to measure individual offsets since these lines are neighboring and interspersed with one another (Figure ). Since the effect of a change in $\alpha$ is large and in the opposite sense for the two species , these two ions also provide an optimally sensitive measurement. For the neighboring $2026~\rm\AA$ and the $2026~\rm\AA$ lines, a change of $\Delta\alpha /\alpha = 10^{-5}$ produces a relative offset of $0.48~{\rm km\
s}^{-1}$ . This is large enough that with a simple sharp absorption line we can hope to measure the offset directly. A crucially important estimate of the magnitude of systematic errors in the MM method comes from having the required sensitivity and spectral resolution to fully analyze a *single* system at one redshift and map the velocity shifts from ion to ion . This is what we do here for two such systems, at $z = 1.7382$ in the quasar HS1946+7658 and at $z=1.6614$ in the quasar HE1104$-$`<!-- -->`{=html}1805A.

However, suitable Cr and Zn systems are rare and most of the MM measurements have been made using and lines, which have two disadvantages. First, the offset between $2600~\rm\AA$ and $2342~\rm\AA$ and the $2800~\rm\AA$ doublet is smaller ($0.21~{\rm km\ s}^{-1}$ for $\Delta\alpha /\alpha = 10^{-5}$) than for the Cr and Zn lines; and second, these systems have a much wider wavelength separation. We shall show here that it is extremely hard using the HIRES spectrograph on the Keck I telescope to obtain the necessary wavelength accuracy for this type of measurement.

# Data and Data Reduction

For the present study we chose three quasars from our existing data base of high resolution quasar spectra which appeared optimal for measuring the fine structure variation. These are HE1104-1805A ($z_{em} = 2.31$), HS1700+6416 ($z_{em} = 2.73$), and HS1946+7658 ($z_{em} = 3.02$). These three quasars are very bright, allowing us to obtain high signal-to-noise observations even at very high spectral resolution ($R = 72,000$) and their spectra all contain narrow, strong (but not saturated) lines with corresponding strong . In addition HE1104-1805A and HS1946+7658 contain damped Lyman $\alpha$ (DLA) absorbers with simple sharp and unsaturated metal lines that are ideal for measuring the fine structure variation with multiple lines and, in particular, ZnII and CrII. HE1104-1805 is a gravitationally lensed system with multiple images , of which the brightest (image "A") is extremely luminous. The $z=1.6614$ DLA absorber has intermediate strength and lines in its spectrum ($N({\rm CrII}) \sim 7 \times 10^{12}\ {\rm cm}^{-2}$ and $N({\rm ZnII}) \sim 2.4 \times 10^{12}\ {\rm cm}^{-2}$; Figure ). Furthermore, the dominant feature in the weaker singly-ionized absorption lines of the $z=1.6614$ system is extremely sharp ($b \sim 4\ {\rm km\ s}^{-1}$) and was only marginally resolved in our earlier moderately high resolution ($R = 35,000$) observations. These earlier observations already suggested there was a velocity offset between the and lines in this system. HS1946+7658 also has an isolated narrow system corresponding to the DLA absorber at $z=1.7382$ which is seen in the , and lines, though this system is weaker than that in HE1104-1805A and some useful lines lie in the Ly$\alpha$ forest.

The upgrade of the CCDs in HIRES spectrograph on the Keck I 10m telescope in 2004 provided finer pixel sampling and allowed the instrument to achieve proper sampling of the high spectral resolution that could be obtained with narrow slits. We therefore began an intensive observing program on these three quasars using a slit width of $0.574^{\prime\prime}$, corresponding to a spectral resolution of $R = 72,000$, in which we paid careful attention to the wavelength calibration. The seeing was generally larger than the slit width so the slit illumination should be uniform. We will discuss this in detail in §. For each observation, 80– or 100–minute exposures were obtained in two 40– or 50–minute segments to allow for processing to remove cosmic rays. ThAr calibration lamps and in some case Iodine cell exposures of a white dwarf standard were taken before and after each pair of quasar exposures (i.e. every 80 minutes). The observations are summarized in Table . Each quasar was observed in multiple grating settings to provide complete wavelength coverage. This also allowed us to test how reproducible wavelengths were in different configurations of the instrument. The total exposure times are 340 minutes for HE1104-1805A, 280 minutes for HS1700+6416, and 600 minutes for HS1946+7658.

```latex
\begin{deluxetable}{lcll}
\tablecaption{Log of Observations \label{tab_obs}}
\tablehead{\colhead{Quasar}  &
\colhead{\ \ \ $z_{em}$\ \ } &
\colhead{Date (UT)} &
\colhead{\quad Exp. (s)}
}
\startdata
HE 1104$-$1805A & 2.31  & 2005 Dec 27 & \quad $3 \times 2400$\\
                              &           & 2009 Jan 31  & \quad $1 \times 2400$\\
                              &          & 2005 Dec 27 & \quad $2 \times 2400$\\
                              &           & 2008 Jan 3   & \quad $2 \times 2400$\\
HS 1700+6416      & 2.73  & 2005 July 2  & \quad $4 \times 2400$\\
                              &          & 2005 July 3  & \quad $3 \times 2400$\\
HS 1946+7658      & 3.02  & 2004 Sep 10  & \quad $6 \times 2400$\\
                              &          & 2005 July 3  & \quad $2 \times 2400$\\
                              &          & 2006 Aug 29 & \quad $3 \times 2400$\\
                              &          & 2005 July 2  & \quad $2 \times 2400$\\
                              &           & 2006 Aug 30 & \quad $2 \times 2400$
\enddata
\end{deluxetable}
```

The observations were obtained with a 2-point binning in the spatial direction (corresponding to a binned pixel size of $0.24^{\prime\prime}$) and no binning in the spectral direction. Each spectrum was extracted by order using an IDL procedure written for the purpose. The images were initially flattened using normalized exposures of a quartz flat lamp observed in the same spectrograph configuration. In each order the spatial distortion of the spectrum was measured from white dwarf calibration stars taken in the same configuration; these were also used to remove the blaze function of the instrument. Sky subtraction was performed by interpolating the sky measured on each side of the spectrum and a vector corresponding to the sky level at the position of the spectrum was stored for each order. Each spectrum was then extracted using a 5 binned spatial pixel window centered on the distorted spatial position at that wavelength to obtain the total counts at that position. Cosmic rays were rejected by comparing the spectra corresponding to the sub-exposures and a final summation of the spectra and the corresponding sky was then made and stored. The ThAr calibration spectra were measured over the same positions used to extract the spectra and also stored by order. The raw counts in the spectrum plus sky were used to compute the expected noise at each pixel assuming a read noise of 6 electrons per pixel and a gain setting of 2.4. The final spectral resolution was measured from the sky lines and is consistent with the nominal value of $R = 72,000$. We show a Gaussian fit to a sample sky line at 6241 Å in Figure . The full width half maximum corresponds to 3.2 spectral pixels, giving a reasonable sampling. In the following analysis we use the laboratory wavelengths of relevant ions summarized in .

# Wavelength calibration

The accuracy of the wavelength calibration is a crucial constraint in achieving the sensitivity necessary to measure evolution in the fundamental constants, particularly if widely separated lines such as and are being used, and requires the adoption of very careful observing techniques and attention to systematics. This has previously been discussed for the HIRES spectrograph by , who used iodine cell exposures to evaluate the systematics of ThAr lamp calibration techniques, and , who presented a calibration procedure for the VLT/UVES spectrograph that removes systematics previously noted in the pipeline procedure by . To obtain the wavelength calibration, we made ThAr lamp exposures before and after each object exposure, and also took a number of exposures each night of white dwarf standards with the iodine cell in place. Since Griest et al. have made a thorough study of the iodine cell calibration, we will not discuss this issue any further here but concentrate on how well the calibration can be made using only ThAr lines. It should be noted that many of the archival observations used in previous analyses of the variation in the fine-structure constant did not aim for precision calibration. The present discussion relates to observations specifically designed to obtain the most accurate calibrations possible with the ThAr lamp. We first discuss the drift in wavelength calibration exhibited by the before and after ThAr lamp spectra, and then analyze the accuracy of the best possible ThAr calibrations using night sky lines that are present in the object spectra, and also using the multiple measurements of the various lines furnished by the independent observations, and comparing the velocities of doublets seen in the spectra.

## ThAr lamp

A preliminary wavelength calibration was made using the ThAr calibration lamp exposures made before and after each quasar exposure (i.e. with a typical 80 minute separation). In each order we fitted Gaussians to between 15 and 20 lines uniformly covering the wavelength range. We then used a fourth-order polynomial to obtain the wavelength fit. The typical dispersion in the ThAr lines relative to the wavelength solution is 0.0016Å.

In order to measure the drift in the calibration we measured the shifts in individual ThAr lines between the before and after calibrations. Examples of these measurements are shown in Figure . The most pronounced effect is an overall wavelength offset, up to 0.025Å at 6000Å over an 80-minute observing period in an extreme case though the more typical value is $\sim 0.005$Å. In Figure  we show that the effect is smoothly increasing as a function of time for a sequence of exposures, suggesting that it is caused by a regular change in the optical path, possibly driven by changes in the environment of the instrument. These overall shifts have no effect on the measurement of $\Delta \alpha/\alpha$, which depends on the relative offset of line pairs. However, the shifts also result in differential offsets along the orders and in the cross-dispersed direction (Figure ). The relative offset of the orders as a function of wavelength is shown in Figure  for all of our ThAr pairs. Both negative and positive absolute offsets are seen and the corresponding gradient is linearly related to the absolute offsets as $d({\delta \lambda} )/d{\lambda} \propto 1.5 \times 10^{-6}\, \delta \lambda\ (6000\rm\AA )$. Typical end-to-end wavelength shifts are $\sim 0.0015$Å along the orders though in the most extreme case the value rises to $\sim 0.0050$Å.

Under most conditions these offsets will scatter equally to negative and positive values and contribute only to the systematic errors. However, depending on the cause of the drift and the observer procedure, they could result in systematic effects. For example, if the offset were driven by temperature changes, which are more often negative through the night, and if observers normally take ThAr exposures prior to the on-sky observation, then there could be a systematic shift in $\Delta \alpha$. However we assume here that the offsets simply add to the error budget.

The actual errors will depend on the observing procedure but if we assume that the data are being calibrated with a ThAr exposure taken 80 minutes away from the observation (which may be fairly typical) then we can use the current data to estimate the systematic error. For a system at $z = 1$, the dispersion in the measured offset between 2344Å and 2800Å corresponds to an r.m.s. error of $4.5 \times 10^{-6}$ in $\Delta \alpha/\alpha$ for an individual measurement.

## Sky lines

We can also use the night sky lines in the spectra of the quasars to test the accuracy of the final wavelength solution and to check if there are residual distortions in the solution arising from the different optical path followed by the ThAr calibration lamp versus sky objects illuminating the slit. Since we used a very narrow slit (in all cases narrower than the typical seeing) the slit was approximately uniformly illuminated, and the sky lines should provide a good calibration. For our ThAr wavelength scale we used the average of the wavelength solutions computed from the before and after ThAr wavelength fits, which should remove much of the drift discussed in the previous subsection.

We selected appropriate night sky lines from the list tabulated by in the wavelength range $5500$ – $7500~\rm\AA$. To obtain sufficient accuracy, we recalculated the wavelengths of all lines from the term values in . We fitted Gaussians to all sky lines using the line-fitting program MPFIT . Examples of the Gaussian fits are shown as the blue solid lines in Figure , plotted over the spectrum (black squares and error bars), shown in the observed air wavelength frame corresponding to the ThAr calibration taken immediately before the object exposure. The OH sky lines are doublets owing to $\Lambda$-splitting. The theoretical wavelengths of the $\Lambda$-split components are shown by the red lines. We fitted Gaussians with the full width of the lines fixed to the measured instrument resolution of $R = 72,000$ and the wavelength offset between the $\Lambda$ - split components fixed to the theoretical separation. Figure  shows examples of sky lines with $\Lambda$-splitting ranging from just resolved to about four times the instrumental resolution. The offsets of the ThAr-measured wavelengths of the sky lines from the true position range from $10^{-3}~\rm\AA$ to $7 \times 10^{-3}~\rm\AA$. These offsets are significantly larger than the statistical errors expected in the fits, which are typically in the range $5 \times 10^{-4}$ – $10^{-3}$Å.

We selected sky lines from this list with intensity $> 50$ counts whose $\Lambda$-split components are resolved at our instrumental resolution and have roughly equal intensities. We rejected lines whose FWHM from the Gaussian fit appeared to be too narrow, indicating that the line might be spurious. This selection provided 20 – 30 sky lines in each spectrum.

Figure  shows the measured offsets in ${\rm km\ s}^{-1}$ for these sky lines in all the spectra as a function of wavelength. The offset is measured from the wavelength position expected from the final wavelength solution, as described above. The black squares with error bars show the individual offsets, the red triangles the average offset in cases with four or more observations of a given sky line, and the blue diamonds show the offsets averaged in 500 Å bins. The solid line is a linear fit to all the points. Over the range 5500 – 7500Å, where there are large numbers of sky lines, we find that there is a significant gradient, with $d(\Delta v)/d\lambda = (-2.91 \pm 0.66) \times 10^{-5}\ {\rm km\ s}^{-1}{\rm \AA}^{-1}$. At $z = 1.5$, using 2344Å and 2800Å, this would result in an offset in the $\Delta\alpha /\alpha$ measurement of $-1.7 \times 10^{-6}$, a significant correction to measured offsets. Therefore a measurement of $\Delta\alpha /\alpha$ with HIRES using and lines and ThAr calibration will show a negative bias. have also found large systematic errors in the HIRES wavelength calibration between 5000 Å and 6000 Å, using the Iodine cell to track wavelength shifts, and , using similar methods, report similar though smaller systematics for the VLT/UVES spectrograph. By comparing solar and asteroid spectra, found wavelength-dependent offsets for UVES, though with the opposite slope to our result for HIRES.

In what follows we used the average ThAr line measurement for the wavelength solution. In Section  we then apply the correction to the wavelength solution based on the sky line fits to correct the measured results so that we can see the comparative size of the effect. The wavelengths were then converted to vacuum heliocentric which is used, except where specified otherwise, in the subsequent text.

## Testing the measurement errors

Error estimation for both Voigt fitting and cross-correlation is complex and there is considerable room for systematic effects. It is our assessment that many of the previous analyses have assigned surprisingly, and possibly unrealistically, small errors to their results. We estimated errors on our fitting in three ways. In the first we measured independent observations of the lines to compare the dispersion with the formal statistical error. In the second, we measured the scatter in an individual order using measurements of doublets in the spectra. In the third we placed the fitted model for a given line at a large number of neighboring random positions in the spectrum and then profile-fitted these simulated lines.

The formal statistical error will not include systematics, such as relative errors in the wavelength determination, uncertainties in the absorption line template, and so on. We therefore refitted the lines, separating the data by night of observation, and also separating systems that appeared in more than one order of the echelle spectrum. A comparison of the individual observations for a absorption line in HS1946+7658 with the final combined profile is shown in Figure . Up to eight independent measurements of the offsets were obtained in this way for some of the absorption lines, and these can be used to make an independent error estimate which should include nearly all of the these random systematic effects associated with the wavelength calibration and profile fitting (though not systematic effects that are present in all the observations such as the linear offset from the ThAr line calibration noted in the previous section).

In Figure  we show an example of the measured velocity offsets in a single system relative to the value measured in the combined spectrum. We show the doublet (black squares) and the 2344, 2382 and 2600 lines as colored symbols. For the weaker lines, with their higher statistical errors, the measured dispersion is fully consistent with the statistical error. However for the strong lines the statistical error is small and is dominated by the systematic error which is approximately $0.1~{\rm km\ s}^{-1}$ in each individual measurement. This has little effect on any of the measurements discussed in the next sections since the differential errors are always dominated by weaker lines.

Griest et al.'s Iodine cell observations suggest that much of the velocity uncertainty of the Th Ar calibration arises in individual orders (the sawtooth pattern seen in their Figure 4). This is of considerable interest for methods such as the comparison of Cr and Zn lines discussed in section . We therefore tested this by measuring the doublet separations of , , and lines in our data. We used only lines where both members of the doublet were in a single order, where lines appeared to be a single symmetric component, and where both members of the doublet were strong but unsaturated. We show the measured offsets as a function of the wavelength separation of the doublet at the observed redshift in Figure . The highest S/N measurements again show a larger scatter than would be expected based on statistical errors, with a typical maximum error of $\sim 0.3\ {\rm km\ s}^{-1}$, consistent with the value expected from the Griest et al. result. This error is expected to be random and should average out with a large enough number of measurements. The systematic error contributes a dispersion of about $0.17~{\rm km\ s}^{-1}$ in addition to the statistical error, which is large enough to substantially increase the error estimates.

The simulation technique, which was also used by Chand et al. (2004), has the advantage of automatically including the effects of continuum fitting and line contamination in the spectrum as well as the effects of non-Gaussian noise, and is therefore considerably superior in these respects to estimates that rely on only the local statistical noise to estimate the error. We expect *a priori* that this method will produce a larger estimate of the noise because of these effects. It also allows us to quantify the fraction of times a line will suffer a major error because of contamination by another absorption line feature or other structures in the the spectrum. We find that this type of analysis gives an average increase in the noise by a factor of 1.17 over the formal statistical errors, with a range from 0.9 – 1.3. This is slightly smaller than the factor of 1.5 seen by . Most contamination by other lines is easily recognized and would be excluded in the actual measurements, but significant velocity offsets ($> 4\sigma$) which would not be easily recognized are seen in about 1% of the simulated lines.

# ,  and  measurements

As we have discussed above, measurements based on the , and lines can avoid some of the wavelength calibration problems though they are still subject to the in-order errors. However, as we discuss here, variability in relative abundances in the substructure of the line can cause serious problems in analyzing even apparently weak and simple systems. A portion of the spectrum of HE1104-1805A covering the and lines of interest in an absorption system at $z = 1.6614$ is shown in Figure  and an expanded region around the 2062.236 and 2062.660 lines is shown in Figure . The expanded spectrum demonstrates that the wavelength structure of this absorption is indeed very simple in the weaker singly-ionized lines and consists of a single sharp feature which is only marginally resolved, together with very weak broader absorption to the red, which extends only to about $50~{\rm km\
s}^{-1}$ and is nearly fully separated from the sharp feature. It is the sharp singly-ionized line which we focus on here. The velocity offset between the and lines can already be seen by visual inspection of Figure .

We first fitted Voigt profiles to all of the useful singly-ionized lines that were not saturated. We used our own customized IDL routines, based on the MPFIT programs of . (We note that a cross-correlation analysis gives basically identical results). A two-component fit based on the strong 1808 line was used. The principal component, containing the bulk of the material, is the narrow feature with a $b$-value of $4.1~{\rm km\ s}^{-1}$; a single broad component was used to model the weaker red wing. We fitted each of the lines by holding the $b$-values fixed but allowing the column densities of all the components, and the overall velocity, to vary. The redshifts of the sharp component were measured and the corresponding velocity offsets determined. The offsets are not sensitive at any significant level to the choice of cloud model, and the results are almost unchanged if we allow the column density, velocity and $b$-value to vary independently in the two components. We note that the velocities are consistent between independent measurements of a given ion, suggesting that line contamination and line saturation are not problems.

We show the relation between the offset expected from a change in $\alpha$ and the measured velocity offset for the $z=1.6614$ system in HE1104-1805A in Figure . A linear regression analysis gives $(0.74\pm0.20) \times 10^{-5}$ (statistical error only) for the required $\Delta\alpha /\alpha$. The best fit is shown by the solid line in the figure. However, the fit has a very poor $\chi^2$ because of the offset between the and lines. We have also measured the offset which would be obtained from restricted subsets of the lines. If we include only the and lines we obtain $(1.87\pm0.24) \times 10^{-5}$, whereas if we use the , and lines this reduces to a null result of $(-0.43\pm0.24) \times 10^{-5}$. Including the systematic errors discussed in and Section  would significantly increase the errors from $0.24 \times 10^{-5}$ to $0.46 \times 10^{-5}$ but still leave a highly significant difference between the fits based on the and lines.

This result shows that there can be serious systematic problems in determining $\Delta\alpha$ using measurements of just and . It strongly suggests that, even in a very simple narrow system like the present one, abundance variation in substructure in the sharp feature (the most likely systematic) can produce an apparent offset between the species. Zn and Cr are well known to have very different abundance patterns, an effect which has traditionally been ascribed to dust depletion but may also reflect variations in nucleosynthesis . If a positive subcomponent of the narrow absorption line had a high Cr/Zn ratio relative to a negative subcomponent this could pull the relative offsets of and to negative values. This is consistent with the observed velocity pattern: the weakly refractory and have the most negative velocities, shown by the filled symbols in Figure , while the refractory elements (, , and ) have more consistent positive velocities. It appears that using with is more secure than using with since both and are refractory elements.

Changes in the isotopic mix are unlikely to produce significant changes in the results. The isotopic effect may be largest in the lines but even here the maximum shift which could be obtained by changing the isotope mix from the local uniform mix of isotopes to being predominantly $^{64}$Zn is only about $0.1~{\rm km\ s}^{-1}$, which is small compared with the measured shifts. As long as $^{52}$Cr remains the dominant isotope we expect the lines to have relatively small isotopic shifts .

The $z=1.7382$ system has a very similar structure and we adopted the same fitting procedure. The measured offsets shown in Figure  yield a consistent solution for both and relative to , albeit with larger errors for this weaker system. The regression analysis gives $\Delta\alpha/\alpha = (-0.94\pm0.24) \times 10^{-5}$ with an acceptable $\chi^2$, and consistent results between and and and . Again, including the systematic error from Section  raises the error from $0.24 \times 10^{-5}$ to $0.59 \times 10^{-5}$. Adopting the values based on , and ions and correcting for the systematic errors, we consider that the most reliable measurements are $\Delta\alpha/\alpha = (-0.47 \pm 0.53) \times 10^{-5}$ for the $z = 1.6614$ system in HE1104$-$`<!-- -->`{=html}1805A and $\Delta\alpha/\alpha = (-0.79 \pm 0.62) \times 10^{-5}$ for the $z = 1.7382$ system in HS1946+7658. However, systematic errors remain a concern even when using the more consistent ions and in particular abundance variations may cause velocity shifts. Large numbers of systems would be required to average out the latter effect.

#  measurements

We next analysed the eight narrow absorption systems seen in the three quasars. These are drawn from five independent complexes, two of which contain two or three useful isolated narrow features. The and lines in each of the systems are shown in Figure 12. In all cases the lines are separated from neighboring absorption features and at least some of the lines are strong enough for accurate measurements. The doublet ratio can be used to estimate the degree of saturation, and there are four systems in which the lines appear unsaturated and four in which at least the lines show saturation effects. We profile-fitted each of the lines using a model based on the 2803 line and varying only the individual column densities of the components and the overall velocity of the system. The results are shown in Figure .

Figure  shows that the spread of these systems, which range in redshift from $z=0.7$ to $z=1.5$, is consistent with the assigned errors. Including only statistical errors, the error-weighted $\Delta\alpha/\alpha$ is $(0.10 \pm 0.24) \times 10^{-5}$. When we correct for the wavelength bias discussed in Section  and for the systematic errors discussed in Section  this becomes $\Delta\alpha/\alpha = (0.43 \pm 0.34)
 \times 10^{-5}$. The small offsets for relative to also show that change in the isotopic mix of from the local value is not large in these systems and is not weighted toward the heavier isotopes. However, the effects of the isotopic mix in some of the quasar absorption lines can be large and could contribute to the spread seen here.

# Summary

MM measurements of the evolution of the fine structure constant are generally based on either comparison of and lines or (at higher redshifts) on measurements of , , and lines. Both have significant problems which have to be dealt with in order to obtain an accurate limit.

In the case of the widely separated and lines, extreme care must be taken in the wavelength calibration. As has previously been discussed in , we see drift in the HIRES wavelength calibration with time together with a significant systematic error in the ThAr calibration of each order of the spectrum and, using the sky lines as a calibrator, a significant linear deviation between the sky line based calibration and the ThAr line calibration. The latter effect will produce a negative deviation in the measurement of $\Delta\alpha/\alpha$ for HIRES observations. The exact shift depends on the redshift and line pair used but is around $1.7 \times10^{-6}$ for typical observations and would reduce the significance of the evolution from $5\sigma$ to about $3.5\sigma$, assuming that the sky offset on HIRES remains at a fixed level over the course of time and has not changed between the observations of and those presented here. Similar effects have been noted for the UVES spectrograph on the VLT . In particular, found wavelength-dependent offsets of a similar magnitude to those discussed here for HIRES by comparing solar and asteroid spectra in the wavelength range 3300Å– 3900Å. However, these offsets vary with wavelength in the opposite sense to the slope we find for HIRES, and so will produce a positive deviation. If a consistent negative slope is found in further tests of the HIRES wavelength accuracy, the opposite signs of the HIRES and UVES observations could provide a natural explanation of the apparent spatial variation in $\alpha$ claimed by .

We have measured the variation using our present high S/N and high resolution observations of eight systems in and , obtaining a value of $\Delta\alpha/\alpha = (0.43 \pm 0.34) \times 10^{-5}$ when we correct for the systematic bias and allow for systematic errors in the line fitting. This null result is consistent with most previous measurements using a variety of techniques.

Measurements based on the neighboring lines of , etc. occur in higher column density systems where abundance variation in unresolved subcomponents can cause measurement errors. In the present work we have argued that , and may give the most robust results for $\Delta\alpha/\alpha$ and that and measurements should be avoided. Using the and measurements for two weak and narrow systems we found limits of $(-0.47 \pm 0.53) \times 10^{-5}$ and $(-0.79 \pm 0.62) \times 10^{-5}$ for $\Delta\alpha/\alpha$ when we allow for systematic wavelength calibration errors.

Combining the and results with the and results gives us a final null result of $\Delta\alpha/\alpha =
(0.01 \pm 0.26) \times 10^{-5}$. We conclude that spectrosopic measurements, made using these methods, are not yet capable of unambiguously detecting variation in $\alpha$.

We gratefully acknowledge support of this work through the National Science Foundation. We would like to thank Amy Barger for drawing our attention to work on the spatial variation of the fine structure constant and to acknowledge Maria Pereira's early contributions to our cross correlation analysis of the quasar absorption lines. We would also like to thank the anonymous referee and Michael Murphy, John Webb, and Sergei Levshakov for comments on the first draft of the paper.


abstract: The evolution of the magnetization in (Cd,Mn)Te quantum wells after a short pulse of magnetic field was determined from the giant Zeeman shift of spectroscopic lines. The dynamics in absence of magnetic field was found to be up to three orders of magnitude faster than that at 1 T. Hyperfine interaction and strain are mainly responsible for the fast decay. The influence of a hole gas is clearly visible: at zero field anisotropic holes stabilize the system of Mn ions, while in a magnetic field of 1 T they are known to speed up the decay by opening an additional relaxation channel.
author: M. Goryca$^{1}$; D. Ferrand$^{2}$; P. Kossacki$^{1,2}$; M. Nawrocki$^{1}$; W. Pacuski$^{1,2}$; W. Maślana$^{1,2}$; J.A. Gaj$^{1}$; S. Tatarenko$^{2}$; J. Cibert$^{2}$; T. Wojtowicz$^{3}$; G. Karczewski$^{3}$
date: 2024-10-03
title: Magnetization dynamics down to zero field in dilute (Cd,Mn)Te quantum wells

The quest for carrier-induced ferromagnetism, and for systems appropriate to spin manipulation in the context of quantum information processing, stimulates intense studies of Diluted Magnetic Semiconductors (DMS) . In the DMS family, (Cd,Mn)Te appears as particularly suitable for studies of low dimensional structures, from quantum wells (QWs) exhibiting carrier-induced ferromagnetic interactions , to quantum dots (QDs) containing a single magnetic impurity . This flexibility is due to the isoelectronic character of Mn in CdTe, offering the possibility of modulation doping and electrical biasing of nanostructures , and to the broad range of optical methods available, which yield direct information on the magnetic configuration of the Mn system through the so called giant Zeeman effect. While the static properties of these nanostructures are essentially well understood, new questions arise about the magnetization dynamics. In particular, understanding spin dynamics in absence of any applied magnetic field is urgently needed.

Spin dynamics of Mn in (Cd,Mn)Te has been studied, for years, almost exclusively in the presence of a magnetic field. Typical measurements involve a short heat pulse used to drive the sample out of thermal equilibrium, the evolution of the magnetization being measured with a pick-up coil or extracted from the giant Zeeman effect in photoluminescence (PL) . Faraday rotation following the creation of electron-hole pairs by a laser pulse reveals the transverse relaxation time $T_2$ . $T_2$ was also deduced from the width of the Electron Paramagnetic Resonance (EPR) line, in the range of Mn content where it exhibits exchange narrowing.

The relaxation of the transverse component of magnetization is an adiabatic process, which was ascribed to anisotropic spin-spin interactions . It is fast, and accelerates with the Mn content. The relaxation of the longitudinal component involves a transfer of the Zeeman energy to the lattice (spin-lattice relaxation). For isolated Mn$^{2+}$ ions, this is a slow process, due to the absence of spin-orbit coupling within the orbital singlet which forms the ground state. In DMSs, the spin-lattice relaxation rate increases with the Mn content, as a result of a fast transfer via the Mn spins towards "killer centers", in particular Mn clusters which statistically exist within the random Mn distribution. For compositions where both have been measured, the spin-spin time is faster than the spin-lattice relaxation time, by 2 to 3 orders of magnitude (see Fig. , adapted from Ref. ). The general picture that emerges, is that fast processes are expected in a DMS with a high Mn content, while strongly diluted or isolated Mn spins should exhibit long relaxation times. More recently , the role of free carriers has been stressed: they form an efficient channel to transfer the Mn Zeeman energy to the lattice.

Experimental data at zero field are scarce. In Ref. , the magnetization of a bulk (Cd,Mn)Te sample, in the presence of a static field, was changed by the collinear, AC field of a coil, and monitored by Faraday rotation. The spin-lattice relaxation time was clearly identified and exhibits a dramatic decrease at low field. In principle, this method can be used down to zero field but the frequency response of the set-up was limited to a few microseconds, so that the static field had to be kept above a few tenths of tesla. By contrast, $T_2$ slowly increases when decreasing the applied field in Zn$_{0.9}$Mn$_{0.1}$Se or (Cd,Mn)Te .

In this paper we describe the magnetization dynamics after a short magnetic pulse created by a small coil, down to zero field and with a temporal resolution of a few ns, in (Cd,Mn)Te QWs with low Mn content, and various densities of electrons or holes.

Samples contain a single 8 nm-wide, *p*-type or *n*-type, (Cd,Mn)Te QW, with 0.2% to 1.5% Mn, and (Cd,Mg,Zn)Te barriers. The whole structure was grown coherently, by molecular beam epitaxy, on a Cd$_{0.96}$Zn$_{0.04}$Te or Cd$_{0.88}$Zn$_{0.12}$Te substrate or a CdTe buffer layer grown on GaAs. Holes were introduced into the QW either by modulation doping with Nitrogen or by tunneling from surface electron traps . Electrons were provided by modulation doping with Iodine . The carrier density ranged from $1\times10^{10}$ to $3\times10^{11}$ cm$^{-2}$.

Pulses of magnetic field were produced with a small coil (28 turns, internal diameter 400 $\mu$m) mounted directly on the sample surface, fed by a coaxial transmission line terminated with a matched resistor. A 2 A current produces $\approx$ 40 mT, as calibrated on PL spectra of samples with a high Mn content , with rise and fall times $\approx$ 7 ns (see Fig. ). Illumination (a cw linearly polarized diode laser) and collection of PL signal (time resolved photon counting system) were done through the aperture of the coil, along its axis, and in some cases a static magnetic field was added along the same axis (Faraday configuration). Fig. a displays PL spectrum of a *p*-type sample, obtained under typical conditions used in the present study. It features a single line related to the charged exciton $X^+$ (at this carrier density, the neutral exciton is not visible). Application of a magnetic field results in a giant Zeeman shift opposite in the two circular polarizations, proportional to the magnetization change. If the Zeeman shift induced by the field pulse is much smaller than the linewidth, by measuring the intensity difference between the two polarizations, we obtain a signal proportional to the Mn magnetization (Fig. b). A higher dynamics (3 orders of magnitude) but a slightly lower time resolution ($\approx$`<!-- -->`{=html}20 ns) was obtained using transmission or reflectivity. Then, both the charged and neutral exciton lines are visible, so that the field pulse induces not only a shift, but also intensity transfers . The setup for transmission was the same as that for PL, but with a halogen lamp. In reflectivity, we used light pulses synchronized with the field pulse, either from a cw Ti:sapphire laser modulated to pulses shorter than 20 ns using an acousto-optic modulator, or from a pulsed Ti:sapphire laser. An optical bridge was used to monitor the difference between the two circular polarizations of the reflected light.

Fig. c shows an example of the PL signal observed without applied static field. A bi-exponential decay convoluted with the current profile describes well the magnetization decay. The faster characteristic time is about 20 ns, and it is longer than the experimental resolution ($\approx$`<!-- -->`{=html}7 ns). It is smaller than the values obtained in magnetic field by about three orders of magnitude. It shows almost no dependence on the Mn content (Fig. ), as expected for isolated Mn ions.

Several interactions may be responsible for the fast zero-field decay observed: the hyperfine interaction with the nuclear spin of the Mn ion itself, the superhyperfine interaction with the nuclear spin of neighbor Cd atoms, dipolar interaction with nuclear spins or spins of other Mn ions, the single ion anisotropy due to the cubic crystal field and to strain in the epitaxial structure, fluctuations in the interaction with carriers, and so on...

It is well known, from EPR spectroscopy of Mn diluted in CdTe- and ZnTe-MnTe superlattices , that the Mn electronic spin $\overrightarrow{S}$ interacts strongly with its nuclear spin $\overrightarrow{I}$ (both 5/2), with a sizable crystal field, and with strain. The Hamiltonian assessed by EPR is

$$\begin{aligned}
\label{eq:eq1}
\mathcal{H}=g\mu_B\overrightarrow{B}.\overrightarrow{S}+A\overrightarrow{I}.\overrightarrow{S}
+D[S_{z}^{2}-\frac{S(S+1)}{3}] \nonumber
\\
+\frac{a}{6}[S_{x}^{4}+S_{y}^{4}+S_{z}^{4}-\frac{S(S+1)(3S^{2}+3S-1)}{5}]
\end{aligned}$$

with $A=~$`<!-- -->`{=html}680 neV for the hyperfine coupling, $a=~$`<!-- -->`{=html}320 neV for the cubic crystal field. The dominant strain component in epitaxial layers grown pseudomorphically on substrates with 4% and 12% Zn is biaxial, with $\varepsilon_{xx}=\varepsilon_{yy}=-(c_{11}/2c_{12})\varepsilon_{zz}=-2.3\times10^{-3}$ and $-7\times10^{-3}$, respectively. Its effect is described by the $DS_{z}^{2}$ term, with values of $D$ deduced from Ref. . A typical energy diagram is shown in Fig. .

Strong anticrossings appear at low values of the static field (Fig. b): therefore sweeping down the magnetic field at the end of the pulse produces a change not only in the Zeeman energies but also in the state of the Mn. The evolution of the magnetization can be viewed as a series of Landau-Zener processes; it is easily obtained by solving numerically the Schrödinger equation of the density matrix, with an initial value corresponding to a thermalized system. As shown by the dashed line in Fig. c, this calculation of the evolution of the coupled Mn electronic and nuclear spins, averaged over a thermal distribution but with no additional effect of the environment, well describes the first fast drop in the experimental data.

The hyperfine coupling has non-zero matrix elements only between states with $\Delta S_z=\pm1$. So several levels show no anticrossing, and the spin states remain eigenstates when varying the magnetic field. This results in the presence of a persistent magnetization which is clearly visible in Fig. .

The fast decay due to the hyperfine field and the persistent magnetization are not much changed when varying slightly the value of the biaxial strain, which in our samples remains quite low. A larger value of the strain would induce a splitting of the $S=5/2$ sextuplet into three $\pm m$ doublets, see Fig. 2 in Ref. . In particular, around zero field, the $\pm
5/2$ and $\pm3/2$ exhibit no anticrossing, only the $\pm1/2$ doublet does. Such a strong strain exists in CdTe QD grown on ZnTe , with the $\pm1/2$ as the ground state.

A more complete decay is calculated when introducing additional mechanisms which mix states with different values of $S_z$. Within the single Mn system, this is simulated when considering strain components which are anisotropic within the QW plane. For instance, the effect of a strain $\varepsilon_{xy}=4\times10^{-3}$, calculated using Ref. , is shown in Fig. c as a solid line. It shows a good agreement with the experimental data. Such a strain in a QW could be caused by local fluctuations or dislocations, but its estimated value is surprisingly large: x-ray diffraction data gives a 4 times smaller estimate for similar structures . This suggests that mechanisms related to the environment of the Mn impurity should be considered.

Note also that even for the best fit, the calculated evolution leaves a small but nonzero persistent magnetization. The experimental data in Fig. c and a ($B = 0$) shows indeed such a long-time component, which decays with characteristic times in sub-$\mu$s range. Upon application of an external, static magnetic field, the characteristic time and the amplitude of this long-time component increase. A precise study of this effect is beyond the scope of the present work.

It is worth to note, however, that at field of 1 T the signal is mono-exponential, with a characteristic time in good agreement with previous data of spin-lattice relaxation in (Cd,Mn)Te, see Fig. . In a *p*-type sample, the decay is faster for a larger hole density (which is easily controlled by additional illumination ), in agreement with Ref. . Also the dependence of the decay at 1 T on the Mn content (Fig. ) agrees with previous observations . Actually, at 1 T, the pulse field merely changes the Zeeman energy of the six sublevels of the $S=5/2$ multiplet, so that the populations have to re-adjust through spin-lattice relaxation  .

As examplified in Fig. b, at the zero-field dynamics is systematically slower in the presence of holes than in the presence of electrons. We ascribe this effect to the anisotropy introduced by holes. Due to confinement, these are heavy holes with projections $\pm3/2$ of their moment along the growth direction. Due to the strong exchange coupling between holes and Mn spins, the $\pm5/2$ doublet of the Mn spin (with the proper orientation of the holes) forms the ground state, while the $\pm3/2$ and $\pm1/2$ ones are at higher energy. A polarized hole gas of density in the $10^{11}~$cm$^{-2}$ range induces an exchange field of the order of 0.1 T, *i.e.*, a splitting of $10~\mu$eV between the $\pm 5/2$ and $\pm3/2$ doublets. This results in an effective anisotropy similar to that occurring in molecular magnets, where it gives rise to a slow dynamics of the magnetization . Thus the effect of holes on the dynamics of the Mn spin at zero field (slowing down due to an effective anisotropy) is opposite to that at high field (additional relaxation channel). It was also pointed out that in a ferromagnetic (Cd,Mn)Te QW, this coupling results in a softening of the Mn resonance when approaching the critical temperature , and a slow decay in the ferromagnetic phase .

To sum up, the use of a fast magnetic pulse reveals strong peculiarities of the Mn spin dynamics in very dilute DMSs. At zero magnetic field, a fast dynamics is observed. This dynamics is driven by hyperfine coupling with the nuclear spin of the Mn ion; it is highly sensitive to the presence of an anisotropy, particularly that introduced by holes and by strain. In the present study of an ensemble of Mn spins, with a thermal distribution of nuclear spins, this results in a fast decay of the magnetization when the field is swept down to zero. These mechanisms are expected to strongly influence the behavior of a single Mn spin, with different consequences for the dephasing time observed in time-averaged measurements and the coherence time in correlation experiments.