INTRODUCTION
Despite recent advances in the eld of speech emotion recognition, learning representations for natural speech segments that can be used e ciently under noisy and unconstrained conditions still represents a signi cant challenge. Obtaining large, labelled human emotion datasets 'in the wild' is hindered by a number of di culties. First, since labelling naturalistic speech segments is extremely expensive, most datasets consist of elicited or acted speech. Second, as a consequence of the subjective nature of emotions, labelled datasets often su er from low human annotator agreement, as well as the use of varied labelling schemes (i.e., dimensional or categorical) which can require careful alignment [46] . Finally, cost and time prohibitions often result in datasets with low speaker diversity, making it di cult to avoid speaker adaptation. Fully supervised techniques trained on such datasets hence often demonstrate high accuracy for only intra-corpus data, with a natural propensity to over t [42] .
INTRODUCTION
In light of these challenges, we pose the following question: is it possible to learn a representation for emotional speech content for natural speech, from unlabelled audio-visual speech data, simply by transferring knowledge from the facial expression of the speaker?
INTRODUCTION
Given the recent emergence of large-scale video datasets of human speech, it is possible to obtain examples of unlabelled human emotional speech at massive scales. Moreover, although it is challenging to assess the accuracy of emotion recognition models precisely, recent progress in computer vision has nevertheless enabled deep networks to learn to map faces to emotional labels in a manner that consistently matches a pool of human annotators [1] . We show how to transfer this discriminative visual knowledge into an audio network using unlabelled video data as a bridge. Our method is based on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker.
INTRODUCTION
Our work is motivated by the following four factors. First, we would like to learn from a large, unlabelled collection of 'talking faces' in videos as a source of free supervision, without the need for any manual annotation. Second, evidence suggests that this is a possible source of supervision that infants use as their visual and audio capabilities develop [30] . Newborns look longer at face-like stimuli and track them farther than non-face-like stimuli (Goren et al. [29] ; Johnson et al. [38] ), and combining these facial stimuli together with voices, detect information that later may allow for the discrimination and recognition of emotional expressions. Our third motivation is that we would like to be able to handle ambiguous emotions gracefully. To this end, we seek to depart from annotation that relies on a single categorical label per segment, but instead incorporate a measure of uncertainty into the labelling scheme, building on prior work by [66] and [32] . Finally, accepting that the relationship between facial and vocal emotion will be a noisy one, we would like to make use of the remarkable ability of CNNs to learn e ectively in the presence of label noise when provided with large volumes of training data [45, 59] .
INTRODUCTION
We make the following contributions: (i) we develop a strong model for facial expression emotion recognition, achieving state of the art performance on the FERPlus benchmark (section 3.1), (ii) we use this computer vision model to label face emotions in the VoxCeleb [50] video dataset -this is a large-scale dataset of emotionunlabelled speaking face-tracks obtained in the wild (section 4); (iii) we transfer supervision across modalities from faces to a speech, and then train a speech emotion recognition model using speaking facetracks (section 5); and, (iv) we demonstrate that the resulting speech model is capable of classifying emotion on two external datasets (section 5.2). A by-product of our method is that we obtain emotion annotation for videos in the VoxCeleb dataset automatically using the facial expression model, which we release as the E V C dataset.
RELATED WORK
Teacher-student methods. Teaching one model with another was popularised by [12] who trained a single model to match the performance of an ensemble, in the context of model compression. E ective supervision can be provided by the "teacher" in multiple ways: by training the "student" model to regress the pre-softmax logits [7] , or by minimising cross entropy between both models' probabilistic outputs [43] , often through a high-temperature softmax that softens the predictions of each model [19, 34] . In contrast to these methods which transfer supervision within the same modality, cross-modal distillation obtains supervision in one modality and transfers it to another. This approach was proposed for RGB and depth paired data, and for RGB and ow paired data by [31] . More recent work [3, 5, 6, 53] has explored this concept by exploiting the correspondence between synchronous audio and visual data in teacher-student style architectures [5, 6] , or as a form of "selfsupervision" [3] where networks for both modalities are learnt from scratch (an idea that was previously explored in the neuroscience community [9] ). Some works have also examined cross-modal relationships between faces and voices in order to learn identity representations [39, 48, 49] . Di erently from these works, our approach places an explicit reliance on the correspondence between the facial and vocal emotions emitted by a speaker during speech, discussed next. Links between facial and vocal emotion. Our goal is to learn a representation that is aware of the emotional content in speech prosody, where prosody refers to the extra-linguistic variations in speech (e.g. changes in pitch, tempo, loudness, or intonation), by transferring such emotional knowledge from face images extracted synchronously. For this to be possible, the emotional content of speech must correlate with the facial expression of the speaker. Thus in contrast to multimodal emotion recognition systems which seek to make use of the complementary components of the signal between facial expression and speech [15] , our goal is to perform cross-modal learning by exploiting the redundancy of the signal that is common to both modalities. Fortunately, given their joint relevance to communication, person perception, and behaviour more generally, interactions between speech prosody and facial cues have been intensively studied (Cvejic et al. [21] ; Pell [56] ; Swerts and Krahmer [61] ). The broad consensus of these works is that during conversations, speech prosody is typically associated with other social cues like facial expressions or body movements, with facial expression being the most 'privileged' or informative stimulus [58] . Deep learning for speech emotion recognition. Deep networks for emotional speech recognition either operate on hand-crafted acoustic features known to have a signi cant e ect on speech prosody, (e.g. MFCCs, pitch, energy, ZCR, ...), or operate on raw audio with little processing, e.g. only the application of Fourier transforms [20] . Those that use handcrafted features focus on global suprasegmental/prosodic features for emotion recognition, in which utterance level statistics are calculated. The main limitation of such global-level acoustic features is that they cannot describe the dynamic variation along an utterance [2] . Vocal emotional expression is shaped to some extent by di erences in the temporal structure of language and emotional cues are not equally salient throughout the speech signal [41, 58] . In particular, there is a well-documented propensity for speakers to elongate syllables located in word-or phrase-nal positions [52, 55] , and evidence that speakers vary their pitch in nal positions to encode gradient acoustic cues that refer directly to their emotional state (Pell [55] ). We therefore opt for the second strategy, using minimally processed audio represented by magnitude spectrograms directly as inputs to the network. Operating on these features can potentially improve performance "in the wild" where the encountered input can be unpredictable and diverse [40] . By using CNNs with max pooling on spectrograms, we encourage the network to determine the emotionally salient regions of an utterance. Existing speech emotion datasets. Fully supervised deep learning techniques rely heavily on large-scale labelled datasets, which are tricky to obtain for emotional speech. Many methods rely on using actors [13, 14, 44, 47] (described below), and automated methods are few. Some video datasets are created using subtitle analysis [25] . In the facial expression domain, labels can be generated through reference events [1] , however this is challenging to imitate for speech. A summary of popular existing datasets in given in Table 1 . We highlight some common disadvantages of these datasets below, and contrast these with the VoxCeleb dataset that is used in this paper:
RELATED WORK
(1) Most speech emotion datasets consist of elicited or acted speech, typically created in a recording studio, where actors read from written text. However, as [27] points out, full-blown emotions very rarely appear in the real world and models trained on acted speech rarely generalise to natural speech. Furthermore there are physical emotional cues that are di cult to consciously mimic, and only occur in natural speech. In contrast, VoxCeleb consists of interview videos from YouTube, and so is more naturalistic.
RELATED WORK
(2) Studio recordings are also often extremely clean and do not su er from 'real world' noise artefacts. In contrast, videos in the VoxCeleb dataset are degraded with real world noise, consisting of background chatter, laughter, overlapping speech and room acoustics. The videos also exhibit considerable variance in the quality of recording equipment and channel noise.
RELATED WORK
(3) For many existing datasets, cost and time prohibitions result in low speaker diversity, making it di cult to avoid speaker adaptation. Since our method does not require any emotion labels, we can train on VoxCeleb which is two orders of magnitude larger than existing public speech emotion datasets in the number of speakers.
RELATED WORK
Note that for any machine learning system that aims to perform emotion recognition using vision or speech, the ground truth emotional state of the speaker is typically unavailable. To train and assess the performance of models, we must ultimately rely on the judgement of human annotators as a reasonable proxy for the true emotional state of a speaker. Throughout this work we use the term "emotion recognition" to mean accurate prediction of this proxy.
CROSS MODAL TRANSFER
The objective of this work is to learn useful representations for emotion speech recognition, without access to labelled speech data. Our approach, inspired by the method of cross modal distillation [31] , is to tackle this problem by exploiting readily available annotated data in the visual domain.
CROSS MODAL TRANSFER
Under the formulation introduced in [31], a "student" model operating on one input modality learns to reproduce the features of a "teacher" model, which has been trained for a given task while operating on a di erent input modality (for which labels are available). The key idea is that by using a su ciently large dataset of modality paired inputs, the teacher can transfer task supervision to the student without the need for labelled data in the student's modality. Importantly, it is assumed that the paired inputs possess the same attributes with respect to the task of interest.
CROSS MODAL TRANSFER
In this work, we propose to use the correspondence between the emotion expressed by the facial expression of a speaker and the emotion of the speech utterance produced synchronously. Our approach relies on the assumption that there is some redundancy in the emotional content of the signal communicated through the concurrent expression and speech of a speaker. To apply our method, we therefore require a large number of speaking face-tracks, in which we have a known correspondence between the speech audio and the face depicted. Fortunately, this can be acquired, automatically and at scale using the recently developed SyncNet [18] . This method was used to generate the large-scale VoxCeleb dataset [50] for speaking face-tracks, which forms the basis of our study.
CROSS MODAL TRANSFER
As discussed in Sec. 2, there are several ways to "distill" the knowledge of the teacher to the student. While [31] trained the student by regressing the intermediate representations at multiple layers in the teacher model, we found in practice that the approach introduced in [34] was most e ective for our task. Speci cally, we used a cross entropy loss between the outputs of the networks after passing both both sets of predictions through a softmax function with temperature T to produce a distribution of predictions:
CROSS MODAL TRANSFER
where x i denotes the logit associated with class i and p i denotes the corresponding normalised prediction. A higher temperature softmax produces a "softer" distribution over predictions. We experimented with several values of T to facilitate training and found, similarly to [34] , that a temperature of 2 was most e ective. We therefore use this temperature value in all reported experiments.
The Teacher
This section describes how we obtain the teacher model which is responsible for classifying facial emotion in videos. Frame-level Emotion Classi er. To construct a strong teacher network (which is tasked with performing emotion recognition from face images), training is performed in multiple stages. We base our teacher model on the recently introduced Squeeze-and-Excitation architecture [35] (the ResNet-50 variant). The network is rst pretrained on the large-scale VGG-Face2 dataset [16] (≈ 3.3 million faces) for the task of identity veri cation. The resulting model is then netuned on the FERplus dataset [10] for emotion recognition. This dataset comprises the images from the original FER dataset (≈ 35k images) [28] together with a more extensive set of annotations (10 human annotators per image). The emotions labelled in the dataset are: neutral, happiness, surprise, sadness, anger, disgust, fear and contempt. Rather than training the teacher to predict a single correct emotion for each face, we instead require it to
Corpus Speakers
Naturalness Labelling method Audio-visual AIBO [11] 51 Natural Manual Audio only EMODB [13] 10 Acted Manual Audio only ENTERFACE [47] 43 Acted Manual LDC [44] 7 Acted Manual Audio only IEMOCAP [14] 10 Both † Manual AFEW 6.0♠ [25] unknown + Acted Subtitle Analysis RML 8 Acted Manual EmoVoxCeleb 1,251 Natural Expression Analysis match the distribution of annotator labels. Speci cally, we train the network to match the distribution of annotator responses with a cross entropy loss:
Corpus Speakers
where p (n) i represents the probability of annotation n taking emotion label i, averaged over annotators, and q (n) i denotes the corresponding network prediction.
Corpus Speakers
During training, we follow the data augmentation scheme comprising a ne distortions of the input images introduced in [63] to encourage robustness to variations in pose. To verify the utility of the resulting model, we evaluate on the FERPlus benchmark, following the test protocol de ned in [10] , and report the results in Table 2 . To the best of our knowledge, our model represents the current state of the art on this benchmark. From Frames to Face-tracks. Since a single speech segment typically spans many frames, we require labels at a face-track level in order to transfer knowledge from the face domain to the speech domain. To address the fact that our classi er has been trained on individual images, not with face-tracks, we take the simplest approach of considering a single face-track as a set of individual frames. A natural consequence of using still frames extracted from video, however, is that the emotion of the speaker is not captured with equal intensity in every frame. Even in the context of a highly emotional speech segment, many of the frames that correspond to transitions between utterances exhibit a less pronounced facial expression, and are therefore often labelled as 'neutral' (see Figure 2 for an example track). One approach that has been proposed to address this issue is to utilise a single frame or a subset of frames known as peak frames, which best represent the emotional content of the face-track [57, 64] . The goal of this approach is to select the frames for which the dominant emotional expression is at its apex. It is di cult to determine which frames are the key frames, however, while [57] select these frames manually, [64] add an extra training step which measures the 'distance' of the expressive face from the subspace of neutral facial expressions. This method also relies on the implicit assumption that all facial parts reach the peak point at the same time.
Corpus Speakers
We adopt a simple approximation to peak frame selection by representing each track by the maximum response of each emotion across the frames in the track, an approach that we found to work well in practice. We note that prior work has also found simple average pooling strategies over frame-level predictions [8, 36] to be e ective (we found average pooling to be slightly inferior, though not dramatically di erent in performance). To verify that max-pooling represents a reasonable temporal aggregation strategy, we applied the trained SENet Teacher network to the individual frames of the AFEW 6.0 dataset, which formed the basis of the 2016 Emotion Recognition in the Wild (EmotiW) competition [24] . Since our objective here is not to achieve the best performance by specialising for this particular dataset (but rather to validate the aggregation strategy for predicting tracks), we did not ne-tune the parameters of the teacher network for this task. Instead, we applied our network directly to the default face crops provided by the challenge organisers and aggregated the emotional responses over each video clip using max pooling. We then treat the predictions as 8-dimensional embeddings and use the AFEW training set to t a single a ne transformation (linear transformation plus bias), followed by a softmax, allowing us to account for the slightly di erent emotion categorisation (AFEW does not include a contempt label). By evaluating the resulting re-weighted predictions on the validation set we obtained an accuracy of 49.3% for the 7-way classi cation task, strongly outperforming the baseline of 38.81% released by the challenge organisers. Figure 2 : An example set of frames accompanying a single speech segment in the VoxCeleb dataset illustrating the neutral transition-face phenomenon exhibited by many face tracks: the facial expression of the speaker, as predicted by the static image-based face classi er often takes a 'neutral' label while transitioning between certain phonemes.
The Student
The student model, which is tasked with performing emotion recognition from voices, is based on the VGG-M architecture [17] (with the addition of batch normalization). This model has proven e ective for speech classi cation tasks in prior work [50] , and provides a good trade-o between computational cost and performance. The architectural details of the model are described in section 5.1.
Time-scale of transfer
The time-scale of transfer determines the length of the audio segments that are fed into the student network for transferring the logits from face to voice. Determining the optimal length of audio segment for which emotion is discernable is still an open question. Ideally, we would like to learn only features related to speech prosody and not the lexical content of speech, and hence we do not want to feed in audio segments that contain entire sentences to the student network. We also do not want segments that are too short, as this creates the risk of capturing largely neutral audio segments. Rigoulot, 2014 [58] studied the time course for recognising vocally expressed emotions on human participants, and found that while some emotions were more quickly recognised than others (fear as opposed to happiness or disgust), after four seconds of speech emotions were usually classi ed correctly. We therefore opt for a four second speech segment input. Where the entire utterance is shorter than four seconds, we use zero padding to obtain an input of the required length.
EMOVOXCELEB DATASET
We apply our teacher-student framework on the VoxCeleb [50] dataset, a collection of speaking face-tracks, or contiguous groupings of talking face detections from video. The videos in the Vox-Celeb dataset are interview videos of 1,251 celebrities uploaded to YouTube, with over 100,000 utterances (speech segments). The speakers span a wide range of di erent ages, nationalities, professions and accents. The dataset is roughly gender balanced. The audio segments also contain speech in di erent languages. While the identities of the speakers are available, the dataset has no emotion labels, and the student model must therefore learn to reason about emotions entirely by transferring knowledge from the face network. The identity labels allow us to partition the dataset into three splits: Train, Heard-Val and Unheard-Val. The Heard-Val split contains held out speech segments from the same identities in the training set, while the Unheard-Val split contains identities that are disjoint from the other splits 2 . Validating on unheard identities allows us to ascertain whether the student model is exploiting identity as a bias to better match the predictions of the teacher model. The identity labels may also prove useful for researchers tackling other tasks, for example evaluating the e ect of emotional speech on speaker veri cation, as done by [54] . The total size of each partition is given in Table 3 .
EMOVOXCELEB DATASET
By applying the teacher model to the frames of the VoxCeleb dataset as described in section 3.1, we automatically obtain emotion labels for the face-tracks and the speech segments. These labels take the form of a predicted distribution over eight emotional states that were used to train the teacher model: neutral, happiness, surprise, sadness, anger, disgust, fear and contempt. These frame-level predictions can then be directly mapped to synchronous speech segments by aggregating the individual prediction distributions into a single eight-dimensional vector for each speech segment. For all experiments we perform this aggregation by max-pooling across frames. However, since the best way to perform this aggregation remains an open topic of research, we release the frame level predictions of the model as part of the dataset annotation. The result is a large-scale audio-visual dataset of human emotion, which we call the E V C dataset. As a consequence of the automated labelling technique, it is reasonable to expect that the noise associated with the labelling will be higher than for a manually annotated dataset. We validate our labelling approach by demonstrating quantitatively that the labels can be used to learn useful speech emotion recognition models (Sec. 5.2). Face-track visualisations can be seen in Figure 3 , and audio examples are available online 3 . Distribution of emotions. As noted above, each frame of the dataset is annotated with a distribution of predictions. To gain an estimate of the distribution of emotional content in E V C , we plot a histogram of the dominant emotion (the label with the strongest prediction score by the teacher model) for each extracted frame of the dataset, shown in Figure 4 . While we see that the dataset is heavily skewed towards a small number of emotions (particularly neutral, as discussed in Sec. 3), we note that it still contains some diversity of emotion. For comparison, we also illustrate the distribution of emotional responses of the teacher model on 'Afew 6.0' [25] , an emotion recognition benchmark. The Afew dataset was collected by selecting scenes in movies for which the subtitles contain highly emotive content. We see the distribution of labels is signi cantly more balanced but still exhibits a similar overall trend to E V C . Since this dataset has been actively sampled to contain good diversity of emotion, we conclude that the coverage of emotions in E V C may still prove useful, given that no such active sampling was performed. We note that Afew does not contain segments directly labelled with the contempt emotion, so we would therefore not expect there to be frames for which this is the predicted emotion. It is also worth noting that certain emotions are rare in our dataset. Disgust, fear and contempt are not commonly exhibited during natural speech, particularly in interviews and are therefore rare in the predicted distribution. Data Format. As mentioned above, we provide logits (the presoftmax predictions of the teacher network) at a frame level which can be used to directly produce labels at an utterance level (using max-pooling as aggregation). The frames are extracted from the 3 http://www.robots.ox.ac.uk/~vgg/research/cross-modal-emotions face tracks at an interval of 0.24 seconds, resulting in a total of approximately 5 million annotated individual frames.
EXPERIMENTS
To investigate the central hypothesis of this paper, namely that it is possible to supervise a speech emotion recognition model with a model trained to detect emotion in faces, we proceed in two stages. First, as discussed in Sec. 4, we compute the predictions of the SENet Teacher model on the frames extracted from the VoxCeleb dataset. The process of distillation is then performed by randomly sampling segments of speech, each four seconds in duration, from the training partition of this dataset. While a xed segment duration is not required by our method (the student architecture can process variable-length clips by dynamically modifying its pooling layer), it leads to substantial gains in e ciency by allowing us to batch clips together. We experimented with sampling speech segments in a manner that balanced the number of utterance level emotions seen by the student during training. However, in practice, we found that it did not have a signi cant e ect on the quality of the learned student network and therefore, for simplicity, we train the student without biasing the segment sampling procedure.
EXPERIMENTS
For each segment, we require the student to match the response of the teacher network on the facial expressions of the speaker that occurred during the speech segment. In more detail, the responses of the teacher on each frame are aggregated through max-pooling to produce a single 8-dimensional vector per segment. As discussed in Section 3, both the teacher and student predictions are passed through a softmax layer before computing a cross entropy loss. Similarly to [34] , we set the temperature of both the teacher and student softmax layers to 2 to better capture the con dences of the teacher's predictions. We also experimented with regressing the pre-softmax logits of the teacher directly with an Euclidean loss (as done in [7] ), however, in practice this approach did not perform as well, so we use cross entropy for all experiments. As with the predictions made by the teacher, the distribution of predictions made by the student are dominated by the neutral class so the useful signal is primarily encoded through the relative soft weightings of each emotion that was learned during the distillation process. The student achieves a mean ROC AUC of 0.69 over the teacher-predicted emotions present in the unheard identities (these include all emotions except disgust, fear and contempt) and a mean ROC AUC of 0.71 on validation set of heard identities on the same emotions.
Implementation Details
The student network is based on the VGGVox network architecture described in [50] , which has been shown to work well on spectrograms, albeit for the task of speaker veri cation. The model is based on the lightweight VGG-M architecture, however the fully connected fc6 layer of dimension 9 ×n (support in both dimensions) is replaced by two layers -a fully connected layer of 9 × 1 (support in the frequency domain) and an average pool layer with support 1 × n, where n depends on the length of the input speech segment (for example for a 4 second segment, n = 11). This allows the network to achieve some temporal invariance, and at the same time keeps the output dimensions the same as those of the original fully Data size  conv1  7×7  1  96  2×2  254×198  mpool1  3×3  --2×2  126×99  conv2  5×5  96  256  2×2  62×49  mpool2  3×3  --2×2  30×24  conv3  3×3  256  256  1×1  30×24  conv4  3×3  256  256  1×1  30×24  conv5  3×3  256  256  1×1  30×24  mpool5  5×3  --3×2  9×11  fc6  9×1  256  4096  1×1  1×11  apool6  1×n  --1×1  1×1  fc7  1×1  4096  1024  1×1  1×1  fc8  1×1  1024  1251 1×1 1×1 Table 4 : The CNN architecture for the student network. The data size up until fc6 is depicted for a 4-second input, but the network is able to accept inputs of variable lengths. Batchnorm layers are present after every conv layer. cropped from the source frame to include only the face region (we use the face detections provided by the VoxCeleb dataset) resized to 224 × 224, followed by mean subtraction. The input to the student network is a short-term amplitude spectrogram, extracted from four seconds of raw audio using a Hamming window of width 25ms and step (hop) 10ms, giving spectrograms of size 512 × 400. At train-time, the four second segment of audio is chosen randomly from the entire speaking face-track, providing an e ective form of data augmentation. Besides performing mean and variance normalisation on every frequency bin of the spectrogram, no other speech-speci c processing is performed, e.g. silence removal, noise ltering, etc. (following the approach outlined in [50] ). While randomly changing the speed of audio segments can be useful as a form of augmentation for speaker veri cation [50] , we do no such augmentation here since changes in pitch may have a signi cant impact on the perceived emotional content of the speech. Training Details. The network is trained for 50 epochs (one epoch corresponds to approximately one full pass over the training data where a speech segment has been sampled from each video) using SGD with momentum (set to 0.9) and weight decay (set to 0.0005). The learning rate is initially set to 1E − 4, and decays logarithmically to 1E − 5 over the full learning schedule. The student model is trained from scratch, using Gaussian-initialised weights. We monitor progress on the validation set of unheard identities, and select the nal model to be the one that minimises our learning objective on this validation set.
Results on external datasets
To evaluate the quality of the audio features learned by the student model, we perform experiments on two benchmark speech emotion datasets. RML: The RML emotion dataset is an acted dataset containing 720 audiovisual emotional expression samples with categorical labels: anger, disgust, fear, happiness, sadness and surprise. This database is language and cultural background independent. The video samples were collected from eight human subjects, speaking six di erent languages (English, Mandarin, Urdu, Punjabi, Persian, Italian). To further increase diversity, di erent accents of English and Chinese were also included. eNTERFACE [47] : The eNTERFACE dataset is an acted dataset (in English) recorded in a studio. Forty-two subjects of fourteen nationalities were asked to listen to six successive short stories, each of which was designed to elicit a particular emotion. The emotions present are identical to those found in the RML dataset.
Results on external datasets
Both external datasets consist of acted speech, and are labelled by human annotators. Since the external datasets are obtained in a single recording studio, they are also relatively clean, in contrast to the noisy segments in E V C . We choose the RML dataset for evaluation speci cally to assess whether our embeddings can generalise to multilingual speech. Both datasets are class-balanced. Table 5 : Comparison of method accuracy on RML and eN-TERFACE using the evaluation protocol of [51] . Where available, the mean ± std. is reported.
Results on external datasets
We do not evaluate the predictions of the student directly, for two reasons: rst, the set of emotions used to train the student di er from those of the evaluation test set, and second, while the predictions of the student carry useful signal, they skew towards neutral as a result of the training distribution. We therefore treat the predictions as 8-dimensional embeddings and adopt the strategy introduced in Sec. 3.1 of learning a map from the set of embeddings to the set of target emotions, allowing the classi er to re-weight each emotion prediction using the class con dences produced by the student. In more detail, for each dataset, we evaluate the quality of the student model embeddings by learning a single a ne transformation (comprising a matrix multiply and a bias) followed by a softmax to map the 8 predicted student emotions to the target labels of each dataset. Although our model has been trained using segments of four seconds in length, its dynamic pooling layer allows it to process variable length segments. We therefore use the full speech segment for evaluation.
Results on external datasets
To assess the student model, we compare against the following baselines: the expected performance at chance level by a random classi er; and the performance of the teacher network, operating on the faces modality. We also compare with the recent work of [51] , whose strongest speech classi er consisted of a random forest using a combination of 88 audio features inc. MFCCs, Zero Crossings Density (ZCD), lter-bank energies (FBE) and other pitch/intensityrelated components. We report performance using 10-fold cross validation (to allow comparison with [51] ) in Table 5 . While it falls short of the performance of the teacher, we see that the student model performs signi cantly better than chance. These results indicate that, while challenging, transferring supervision from the facial domain to the speech domain is indeed possible. Moreover, we note that the conditions of the evaluation datasets di er signicantly from those on which the student network was trained. We discuss this domain transfer problem for emotional speech in the following section.
Discussion
Evaluation on external corpora: Due to large variations in speech emotion corpora, speech emotion models work best if they are applied under circumstances that are similar to the ones they were Session: FF-2 MM'18, October 22-26, 2018, Seoul, Republic of Korea trained on [60] . For cross-corporal evaluation, most methods rely heavily on domain transfer learning or other adaptation methods [22, 23, 65] . These works generally agree that cross-corpus evaluation works to a certain degree only if corpora have similar contexts. We show in this work that the embeddings learnt on the E V C dataset can generalise to di erent corpora, even with di erences in nature of the dataset (natural versus acted) and labelling scheme. While the performance of our student model falls short of the teacher model that was used to supervise it, we believe this represents a useful step towards the goal of learning useful speech emotion embeddings that work on multiple corpora without requiring speech annotation. Challenges associated with emotion distillation: One of the key challenges associated with the proposed method is to achieve a consistent, high quality supervisory signal by the teacher network during the distillation process. Despite reaching state-of-the-art performance on the FERplus benchmark, we observe that the teacher is far from perfect on both the RML and eNTERFACE benchmarks. In this work, we make two assumptions: the rst is that distillation ensures that even when the teacher makes mistakes, the student can still bene t, provided that there is signal in the uncertainty of the predictions. The second is a broader assumption, namely that deep CNNs are highly e ective at training on large, noisy datasets (this was recently explored in [45, 59] , who showed that despite the presence of high label noise, very strong features can be learned on large datasets). To better understand how the knowledge of the teacher is propagated to the student, we provide confusion matrices for both models on the RML dataset in Figure 5 . We observe that the student exhibits reasonable performance, but makes more mistakes than the teacher for every emotion except sadness and anger. There may be several reasons for this. First, E V C used to perform the distillation may lack the distribution of emotions required for the student to fully capture the knowledge of the teacher. Second, it has been observed that certain emotions are easier to detect from speech than faces, and vice versa [15] , suggesting that the degree to which there is a redundant emotional signal across modalities may di er across emotions. Limitations of using interview data: Speech as a medium is intrinsically oriented towards another person, and the natural contexts in which to study it are interpersonal. Interviews capture these interpersonal interactions well, and the videos we use exhibit real world noise. However, while the interviewees are not asked to act a speci c emotion, i.e. it is a 'natural' dataset, it is likely that celebrities do not act entirely naturally in interviews. Another drawback is the heavily unbalanced nature of the dataset where some emotions such as contempt and fear occur rarely. This is an unavoidable artefact of using real data. Several works have shown that the interpretation of certain emotions from facial expressions can be in uenced to some extent by contextual clues such as body language [4, 33] . Due to the "talking-heads" nature of the data, this kind of signal is typically not present in interview data, but could be incorporated as clues into the teacher network. Student Shortcuts: The high capacity of neural networks can sometimes allow them to solve tasks by taking "shortcuts" by exploiting biases in the dataset [26] . One potential for such a bias in E V C is that interviewees may often exhibit consistent emotions which might allow the student to match the teacher's predictions by learning to recognise the identity, rather than the emotion of the speaker. As mentioned in Sec. 5, the performance of the student on the heardVal and unheardVal splits is similar (0.71 vs 0.69 mean ROC AUC on a common set of emotions), providing some con dence that the student is not making signi cant use of identity as a shortcut signal. Extensions/Future Work: First, we note that our method can be applied as is to other mediums of unlabelled speech, such as lms or TV shows. We hope to explore unlabelled videos with a greater range of emotional diversity, which may help to improve the quality of distillation and address some of the challenges discussed above. Second, since the act of speaking may also exert some in uence on the facial expression of the speaker (for example, the utterance of an "o" sound could be mistaken for surprise) we would also like to explore the use of proximal non-speech facial expressions as a supervisory signal in future work. Proximal supervision could also address the problem noted in Section 3, that speaking expressions can tend towards neutral. Finally, facial expressions in video can be learnt using self-supervision [62] , and this o ers an alternative to the strong supervision used for the teacher in this paper.
CONCLUSIONS
We have demonstrated the value of using a large dataset of emotion unlabelled video for cross-modal transfer of emotions from faces to speech. The bene t is evident in the results -the speech emotion model learned in this manner achieves reasonable classi cation performance on standard benchmarks, with results far above random. We also achieve state of the art performance on facial emotion recognition on the FERPlus benchmark (supervised) and set benchmarks for cross-modal distillation methods for speech emotion recognition on two standard datasets, RML and eNTERFACE.
CONCLUSIONS
The great advantage of this approach is that video data is almost limitless, being freely available from YouTube and other sources. Future work can now consider scaling up to larger unlabelled datasets, where a fuller range of emotions should be available.

{'paper_id': '52018690', 'title': 'Emotion Recognition in Speech using Cross-Modal Transfer in the Wild', 's2_url': 'https://api.semanticscholar.org/CorpusID:52018690', 'source': 'S2ORC'}