abstract: In this paper, we investigate how to learn rich and robust feature representations for audio classification from visual data and acoustic images, a novel audio data modality. Former models learn audio representations from raw signals or spectral data acquired by a single microphone, with remarkable results in classification and retrieval. However, such representations are not so robust towards variable environmental sound conditions. We tackle this drawback by exploiting a new multimodal labeled action recognition dataset acquired by a hybrid audio-visual sensor that provides RGB video, raw audio signals, and spatialized acoustic data, also known as acoustic images, where the visual and acoustic images are aligned in space and synchronized in time. Using this richer information, we train audio deep learning models in a teacher-student fashion. In particular, we distill knowledge into audio networks from both visual and acoustic image teachers. Our experiments suggest that the learned representations are more powerful and have better generalization capabilities than the features learned from models trained using just single-microphone audio data.
author: Andrés F. Pérez^1^Valentina Sanguineti^1,2^Pietro Morerio^1^Vittorio Murino^1,3,4^
^1^Pattern Analysis & Computer Vision - Istituto Italiano di Tecnologia, ^2^Università degli Studi di Genova, Italy,
^3^Computer Science Department - Università di Verona, Italy, ^4^Huawei Technologies Ltd., Ireland Research Center
`firstname.lastname@example.com {valentina.sanguineti, pietro.morerio, email@example.com`
bibliography: perez2019.bib
title: Audio–Visual Model Distillation Using Acoustic Images

# Introduction

Humans experience the world through a number of simultaneous sensory observation streams. The co-occurrence of these streams provides a useful learning signal to understand the environment surrounding us . There is in fact evidence that audio-visual mirror neurons play a central role in the recognition of actions given their temporal synchronization . Furthermore, it was found that many neurons with receptive fields spatially aligned across modalities show a super-additive response to coincident and co-localized multimodal stimulations .

In this paper, motivated by these findings, we investigate whether and how visual and acoustic data *synchronized in time* and *aligned in space* can be exploited for scene understanding. We take advantage of a recent audio-visual sensor, called DualCam, composed by an optical camera and a 2D planar array of microphones (see Figure ), able to provide spatially localized acoustic data aligned with the corresponding optical image (see Figure , right) . Specifically, by combining the raw signals acquired by 128 microphones (by beamforming ), this sensor is able to output an acoustic image where each pixel represents the imprint of the sound coming from the corresponding pixel location in the optical image. Using this sensor, we generate a new multimodal dataset depicting different subjects performing several actions in multiple scenarios. By exploiting spatialized audio information coupled to the related visual data and designing suitable multimodal deep learning models, we aim at generating more discriminant and robust features, likely resulting in a better description of the scene content for robust audio classification. Figure  shows the multispectral acoustic image used as input data, which has 512 frequency bins and an example of visualization of an acoustic image overlaid upon an optical image.

The idea of leveraging the co-occurrence of visual and audio events as supervisory signal is not new. Former approaches in the pre deep-learning era combined visual and auditory signals in rather simplistic ways. For instance, in a neural network was trained to predict the auditory signal given the visual input. A particularly relevant earlier work is , which introduced a self-supervised learning algorithm for jointly training audio and visual networks by minimizing codebook disagreement. Another interesting work is , which presented an algorithm based on canonical correlation analysis (CCA) to detect pixels associated to the sound, while filtering out other dynamic (but silent) pixels.

Several recent works address audio-related tasks such as natural sound recognition , speech separation and enhancement , audio event classification or sound source localization , either by directly modeling raw audio signals with 1D convolutions or, most popularly, by modeling intermediate sound representations such as spectrograms or cochleograms . Nevertheless, none of the past works tried to exploit spatially localized acoustic data to assess the potentialities of such richer information source.

In our work, we claim that it is possible to train audio deep learning models to face an action recognition problem in a more robust way across different scenarios utilizing a teacher-student framework able to distill knowledge from state-of-the-art vision network models and from a novel architecture that operates on the spatialized acoustic data. Similarly to , our intuition is to learn better features for a given modality assuming the availability of other complementary modalities at training time. We leverage video and multispectral acoustic image sequences aligned in space/time as side information at training, and predict actions given only a raw audio signal acquired by a single microphone at test, in a cross-scenario setting, where the environmental noise conditions are significantly different. Current methods, even best deep learning models, lead to very low classification accuracies in such conditions.

Hence, in essence, in this work we try to answer the following question: *Does spatialized data allow to learn more discriminant features for single-microphone audio classification?* In this respect, our main contributions can be summarized as follows.

1.  We propose a thorough study to assess whether visual and acoustic data aligned in space and synchronized in time bring advantage for single-microphone audio classification.

2.  We introduce a new multimodal dataset consisting in 14 action classes, in which acoustic and visual data are spatially aligned. This type of multi-sensory data has no counterpart in the literature and may lead to further studies by the scientific community.

3.  We develop a deep teacher-student model to deal with such new data, showing that it is indeed possible to extract semantically richer representations for improving audio classification from single microphone. In particular, we distill knowledge learned from spatialized audio-visual modalities to a single-microphone model.

It is worth to note that we are the first to propose an algorithm in which the transfer of knowledge involves teacher models considering 2 different modalities (2D audio and 2D visual data) and the student model is devised for a different modality (1D audio signal), when typically the student deals with the task of one of the teacher models.

We validate our approach 1) on the proposed action dataset, and 2) by transferring learned representations on a standard sound classification benchmark dataset, demonstrating remarkable capabilities and the usefulness of distillation for cross-scenario learning.

The remainder of this paper is organized as follows. We first discuss the related work in Section , mainly focusing on audio-visual models and benchmark datasets. In Section , we describe our new acquired multimodal action dataset, and in Section , we describe acoustic image pre-processing and we propose the network architecture to deal with acoustic images. In Section , we present our distillation-based approach to deal with multispectral acoustic data, and in Section , we extensively validate our proposed framework by devising a set of experiments in order to assess the soundness of the learned representations. Finally, we draw conclusions in Section .

# Related Work

We briefly review related work in the areas of multimodal learning, video and sound self-supervision, and transfer learning. We also review already existing audio and audio-visual datasets.

**Multimodal learning.** Multimodal learning concerns relating information from multiple data modalities. Such data provides complementary semantic information due to correlations in between them . We consider the cross-modality learning setting, in which data from multiple modalities is available only during training, while only data from a single modality is provided at testing phase. In the authors learn shared representations from multimodal aligned data and use them for cross-modal retrieval. for instance considers three major natural modalities: vision, sound and language, while considers five weakly aligned modalities: natural images, sketches, clip art, spatial text, and descriptions. Other works such as utilize RGB video images and depth information to learn feature representations through modality hallucination. In our work instead, we consider RGB video images, raw audio and acoustic images for training phase, and only raw audio at testing time.

**Video and sound self-supervision.** There has been increased interest in using deep learning models for multimodal fusion of auditory and visual signals to improve the performance of visual models or solve various speech-related problems, such as speech separation and enhancement.

First approaches trained single networks on one modality using the other one to derive some sort of supervisory signal . For example train an audio network to correlate with visual outputs using pre-trained visual networks as a teacher. Others such as train a visual network to generate sounds by solving a regression problem consisting in mapping a sequence of video frames to a sequence of audio features. In instead, they learn visual models using ambient sounds as scene labels.

More recent works train both visual and audio networks aiming at learning multimodal representations useful for many applications, such as cross-modal retrieval, speech separation, sound source localization, action recognition, and on/off-screen audio source separation. For instance in they learn aligned audio-visual representations, using an audio-visual correspondence task. In they train an early-fusion multisensory network to predict whether video frames and audio are temporally aligned. In a two-stream network structure is trained utilizing an attention mechanism guided by sound information to localize the sound source.

They key factor in all these works is that they exploit the natural synchronization between auditory and visual signals by training in a self-supervised manner. Although we address our problem in a pseudo-supervised manner using a combination of hard and soft labels, we notice that the natural spatial alignment and time synchronization of the data produced by the DualCam sensor opens the door to also train models through self-supervision.

**Transfer learning.** Our work is strongly related to transfer learning which deals with sharing information from one task to another. In particular we transfer knowledge between networks operating on different data modalities (see Section ). We perform transferring with the aid of the generalized distillation framework which proposes to use the teacher-student approach from the distillation theory to extract knowledge from a privileged information source , also called a teacher. In our case the privileged information leveraged at training time is represented by the additional modalities, i.e. video and acoustic images. A rather simple transfer mechanism is that of which proposes a teacher-student self-supervised training procedure based on the Kullback-Leibler divergence to transfer knowledge from a vision model into sound modality using unlabeled video as a bridge. This mechanism resembles the generalized distillation framework, however they only rely on the teacher soft labels which are in general less reliable than hard labels. An interesting work is which introduces a novel technique for incorporating additional information, in the form of depth images, at training time to improve test time RGB only detection models. We draw inspiration from which addresses action recognition by distilling knowledge from a depth network into a vision network. They accomplish this by training a hallucination network that learns to distill depth features. It is worth noticing that although works with different data modalities, it is the closest to ours since they transfer knowledge with the aid of the generalized distillation framework.

**Audio-visual datasets.** Due to recent interest in audio-visual and multimodal learning, several audio and audio-visual datasets have emerged. Here we summarize some of the most prominent ones.

*Flickr-SoundNet* is a large unlabelled dataset of completely unconstrained videos from Flickr, compiled by searching for popular tags and dictionary words. It contains over 2 million videos which total for over one year of continuous natural sound and video.

*Kinetics-Sounds* comprises a subset of the Kinetics dataset , which contains YouTube videos manually annotated for human actions, and cropped to 10 seconds around the action. The subset contains 19k video clips formed by filtering the Kinetics dataset for 34 human action classes, which have been chosen to be potentially manifested visually and aurally.

*FAIR-Play* is an unlabelled video dataset with binaural audio that mimics human hearing. It consists of 1.871 short clips of 10 seconds long musical performances, totaling 5.2 hours. It depicts different combinations of people playing musical instruments including cello, guitar, drum, ukelele, harp, piano, trumpet, upright bass, and banjo, in a large music room, in solo, duet, and multiplayer performances.

*Environmental Sound Classification (ESC-50)* is a labeled collection of 2.000 environmental audio recordings manually extracted from Freesound. It consists of 5 seconds long recordings organized into 50 semantical classes loosely arranged into five major categories: animals, natural soundscapes & water sounds, human non-speech sounds, interior/domestic sounds, and exterior/urban noise.

*Detection and Classification of Acoustic Scenes and Events (DCASE)* is a dataset consistent of of recordings from various acoustic scenes. It was recorded in six large European cities, in different locations for each scene class. For each recording location there are 5 to 6 minutes of audio split into segments of 10 seconds.

The closest dataset to ours is *FAIR-Play* because of its size and the nature of its data, since binaural audio is a form of spatial audio. Similarly to *Kinetics-Sounds* we propose a dataset of human actions, but with data in multiple modalities which try to describe more realistic conditions.

# Audio-Visually Indicated Action Dataset

We introduce a new multimodal dataset comprised of visual data as RGB image sequences and acoustic data as raw audio signals acquired from 128 microphones. The latter signals, opportunely combined by a beamforming algorithm, compose a multispectral acoustic image volume, which is aligned in space and time with the optical images (see Figure ). The following 14 actions were chosen:

For the acquisition, we acknowledge the participation of 9 people performing the aforementioned actions recorded in three different scenarios, with increasing and varying noise conditions, namely, an anechoic room, an indoor open space area, and a terrace outdoor. We name them scenario 1, 2, and 3, respectively. In our dataset, the same action is performed by different subjects in distinct places, so allowing to show the equivariance properties of the multispectral acoustic images across subjects, scenarios and position in the scene, which are exploited when learning audio features from an acoustic teacher model. In the end, the dataset consists of 378 audio-visual video sequences (27 per action) between 30 and 60 seconds depicting different people individually performing a set of actions producing a characteristic sound in each scenario. Figure  shows representative samples of our dataset for the 3 considered scenarios.

We acquired the dataset using the DualCam acoustic-optical camera described in . The sensor captures both audio and video data using a $0.45m\times0.45m$ planar array of 128 low-cost digital MEMS microphones located according to an optimized aperiodic layout, and a video camera placed at the device center as depicted in Figure .

The device is capable of acquiring audio data in the range 200 Hz – 10 kHz and audio-video sequences at a frame rate of 12 fps. In our acquisition setup the camera was static looking at the scene, while the subjects moved around within its field of view at a minimum distance of 2 meters from the device.

After collecting the dataset, audio and video data had to be synchronized since they were acquired in an interleaved way at different frame rates.

The data provided by the sensor consists in RGB video frames of $640\times480$ pixels, raw audio data from 128 microphones acquired at a frequency of 12 kHz, and $36\times48\times512$ multispectral acoustic images obtained from the raw audio signals of all the microphones using beamforming, which summarize the per-direction audio information in the frequency domain. This means that each acoustic pixel corresponds to 13,3 visual pixels, in fact acoustic resolution is lower than optical one. Among the raw audio waveforms, we choose the one of just one microphone for testing single microphone audio networks.

# Learning with Acoustic Images

In this section, we describe acoustic images representation, their pre-processing and the network architecture we proposed for modelling this novel type of data.

**Acoustic Images Pre-processing.** Multispectral acoustic images are generated with the frequency implementation of the filter-and-sum beamforming algorithm , aimed at producing a volume of size $36\times48$, with 512 channels corresponding to the frequency bins which represent the frequency information. Full details of the algorithm can be found in .

Handling input acoustic images with 512 channels is a computationally expensive task and typically the majority of information in our dataset is contained in the low frequencies. Consequently, we decided to compress the acoustic images using Mel-Frequency Cepstral Coefficients (MFCC), which consider audio human perception characteristics . Therefore, we compute 12 MFCC, going from from $36\times48\times512$-D volumes to $36\times48\times12$-D volumes, retaining the most important information and reducing consistently the computational complexity and the memory footprint.

**DualCamNet Architecture.** Acoustic images provide a small temporal support which is generally not enough for discriminating information over time intervals of several seconds. For this reason, we feed to our network a set of 12 consecutive $36\times48\times12$ acoustic images corresponding to 1 second of audio data. We deem that 1 second of acoustic images is a reasonable trade-off between sound information content and processing cost.

In order to train a model able to discriminate information from acoustic images, we explicitly model both the spatial and the temporal relationships among them. To this end, we propose the architecture structure shown in Figure  which utilizes 3D convolutions as commonly done in visual action recognition , where the spatial and temporal convolutions are decoupled.

We follow the LeNet design style, with $5\times5$ convolutional filters, and $2\times2$ max-pooling layers with stride 1 and zero-padding to keep the spatial resolution. The network includes 3 blocks of convolutional layers plus a block of 3 fully convolutional layers which produces the output prediction.

The first block consists of a single 1D convolutional layer over time followed by a ReLU nonlinearity. The aim of this layer is modeling the temporal relationship of consecutive acoustic images by aggregating them. In particular, we apply a filter of size 7 with stride 1 and zero-padding to keep the temporal resolution. We experimented with several filters sizes finding 7 to be the best one.

The second and third blocks model the spatial equivariance of the acoustic images and consist of a 2D convolutional layer followed by max-pooling. We go from the 12 channels of the input to 32 channels and then double it to 64. Each convolutional layer is followed by batch normalization and ReLU nonlinearity.

The final block comprises 3 fully convolutional layers with ReLU in between. It converts the input feature map into a 14-D classification vector as output, namely the predicted class probabilities, using intermediate features size of 1024-D and 1000-D.

This model will be used as teacher network in our validation experiments.

# Model Distillation

In this section, we describe the utilized network architectures and the knowledge transfer procedure.

## Architectures

Similarly to , we utilize data from multiple modalities at training phase, and only data from a single modality at testing phase. We leverage either RGB video images or multispectral acoustic images in training as side information, and we test only on audio data from a single microphone.

We want to emphasize here that, to the best of our knowledge, this is the first time that model distillation is performed from modalities different from those utilized in testing. Specifically, we train on 2-dimensional spatialized audio and video data, to improve accuracy on a model working on mono-dimensional audio signals only as input. As further original aspect, trains one ResNet-50 network per stream, while we use different network architectures for each stream of our model.

**Teacher Networks.** For the visual stream, we experimented with two models, ResNet-50 and its variation including 3D temporal convolutions introduced in , here called Temporal ResNet-50. We choose ResNet-50 respect to Temporal ResNet-50 as it provides a good compromise between network size and accuracy. On the other hand, Temporal ResNet-50 stands as a strong action recognition model dealing with action dynamics with the aid of temporal connections between residual units. It has also been selected since it constitutes a powerful baseline model to compare with. DualCamNet, explained earlier in Section , will be used as teacher model as well in the following.

**Student Networks.** Regarding the raw audio waveform stream, we experimented two models that capture different characteristics of audio data. The first one is SoundNet , which operates over time domain signals. We preferred the 5-layer version over the 8-layer one, as our dataset is not big enough to allow SoundNet to grasp the underlying data patterns. We used the exact same architecture described in , adding 3 fully convolutional layers at the bottom of the network with 1024, 1000 and 14 filters, respectively. To avoid further confusion, we named our version *OurSoundNet*.

The second model is a network based on the sound sub-network presented in , called from here on, *HearNet*. Its architecture is shown in Figure . This network operates on amplitude spectrograms obtained from an audio waveform of 5 seconds, upsampled to 22 kHz. Such spectrogram was produced by computing the STFT [^1] considering a window length of 20 ms with half-window overlap. This produces 500 windows with 257 frequency bands. The resulting $500\times1\times257$ spectrogram is interpreted as a 257-dimensional signal over 500 time steps.

HearNet processes spectrograms with 3 1D convolutions using kernel sizes 11, 5, 3, and 128, 256, 256 filters, respectively, with stride 1. The last convolutional layers are fully convolutional and use 1024, 1024, 1000 and 14 filters to obtain the class predictions. We applied zero-padding in all layers except conv4 in order to keep the spatial resolution. The chosen activation function is ReLU. After each of the first 3 convolutional layers, we downsampled with one-dimensional max-pooling by a factor of 5.

## Training procedure

Following the *generalized distillation* framework , we first learn a teacher function $f_t \in \mathcal{F}_t$ by solving a classification problem and, second, we compute the teacher soft labels $s_i$. As third step, we distill $f_t \in \mathcal{F}_t$ into $f_s \in \mathcal{F}_s$ by using both the hard and soft labels. The knowledge transfer procedure is graphically illustrated in Figure .

In particular we transfer knowledge between multiple modality network streams by using Hinton's distillation loss to extract knowledge from privileged representations. More formally, we distill the teacher learned representation $f_t \in \mathcal{F}_t$ into $f_s \in \mathcal{F}_s$ as follows: $$\small
f_s = \mathop{\mathrm{arg\,min}}_{f \in \mathcal{F}_s} \frac{1}{n} \sum_{i=1}^n (1-\lambda)\ell(y_i, \sigma(f(x_i)))+\lambda\ell(s_i, \sigma(f(x_i))),$$ where $s_i = \sigma(f_t(x_i^*/T)) \in \delta^c$ are the soft labels derived from the teacher about the training data, $\mathcal{F}_t$ and $\mathcal{F}_s$ are classes of functions described by the teacher and student models , respectively, $\sigma$ is the softmax operator, and $y_i$ are the ground truth hard labels. The imitation parameter $\lambda \in [0,1]$ allows to balance the weight of soft labels with respect to the true hard labels $y_i$. The temperature parameter $T>0$ allows to smoothen the probability vector predicted by the teacher network $f_t$.

# Experimental Results

Our goal is to learn feature representations for raw audio data by transferring knowledge across networks operating on different data modalities. To evaluate how well our method addresses this problem we perform two sets of experiments with the objectives of 1) showing the improvement brought by distilling knowledge from different data modality networks and 2) assessing the quality of the distilled representations on a standard sound classification benchmark.

## Acoustic Features Transfer

In this first set of experiments, we evaluate the performance of the teacher and student networks on the task of action recognition on our dataset. We train both the teacher and student networks in a fully supervised manner using action labels as ground truth, and only the student networks following the distillation procedure described in Section . In all cases we trained for 100 epochs[^2] with batches of 32 elements using the Adam optimizer with learning rates of 1 × 10^−3^ and 1 × 10^−4^ (see details in Supplementary Material). In order to measure the generalization capabilities of the learned representations, we evaluate the accuracy of our trained models on a cross-scenario setting, i.e. when the model is trained on certain scenario, it is tested on the other two scenarios using all the available data. In all cases the data was split by assigning 80% of them for training, 10% for validation and 10% for test.

**Teachers Networks.** First, we train our DualCamNet model and the two proposed visual networks, ResNet-50 and Temporal ResNet-50, as they constitute our baselines. Table  shows their performance. We observe that our DualCamNet convincingly outperforms the visual networks in all combination of scenarios. This indicates that most of the actions in our dataset are better distinguishable aurally than visually. One possible explanation for this, is that in the majority of the cases the "object" involved in the action execution, e.g. mouth, mouse or hammer, is not easily visible but has a characteristic sound signature.

A comparison of the two visual networks reveals that they achieve similar results throughout all configurations, indicating that motion is not a key factor to model the actions performed in our dataset. Consequently, we choose ResNet-50 over Temporal ResNet-50 as visual teacher for the rest of the experiments since the former one has a simpler structure.

Additionally, we have designed a hybrid network which combines the output of the DualCamNet and ResNet-50, to check whether modality fusion brings any performance improvement. We do so by concatenating the 1024 feature volumes of the two networks and processing them with two fully convolutional layers of 1000 and 14 filters, respectively. This network achieves a 7.1% improvement in accuracy with respect to DualCamNet when trained over all scenarios. It is important to note that it also consistently improves the testing accuracy in all cross-scenario configurations (see Table , AV column). These findings indicate some benefits brought by modality fusion that can be further explored in future research.

| **Train set** | **Test set** | **D**  | **R**  | **T**  | **AV** |
|:--------------|:-------------|:------:|:------:|:------:|:------:|
| Scenario 1    | Scenario 1   | 0.8470 | 0.6965 | 0.7117 | 0.8775 |
|               | Scenario 2   | 0.2938 | 0.2955 | 0.2616 | 0.3490 |
|               | Scenario 3   | 0.1471 | 0.1355 | 0.1410 | 0.1528 |
| Scenario 2    | Scenario 1   | 0.2986 | 0.1918 | 0.1844 | 0.3060 |
|               | Scenario 2   | 0.7600 | 0.5838 | 0.4987 | 0.7418 |
|               | Scenario 3   | 0.1504 | 0.1486 | 0.1243 | 0.2049 |
| Scenario 3    | Scenario 1   | 0.2309 | 0.1479 | 0.1571 | 0.2767 |
|               | Scenario 2   | 0.2032 | 0.1229 | 0.1063 | 0.2182 |
|               | Scenario 3   | 0.6736 | 0.2240 | 0.3013 | 0.5708 |
| All           | All          | 0.7702 | 0.6335 | 0.6393 | 0.8412 |
| scenarios     | scenarios    |        |        |        |        |

Test accuracy for teacher models. D: DualCamNet. R: ResNet-50 . T: Temporal ResNet-50 . AV: AVNet.

**Student Networks.** In order to measure the improvement brought by distillation, we need to look first at the performance of the two proposed student networks when trained only from hard labels only. Column G from Tables  and  show the accuracy results for OurSoundNet and HearNet, respectively. It can be observed that both networks perform well, with HearNet achieving a higher accuracy in all scenarios settings.

This result is impressive considering that OurSoundNet was fine-tuned from SoundNet-5 which was trained on the Flickr-SoundNet dataset, while HearNet instead was trained from scratch on our dataset. A reasonable explanation for this is that shallow networks such as HearNet perform better under small data regimes.

| **Train set** | **Test set**  | **G**  | **D**  | **R**  |
|:--------------|:--------------|:------:|:------:|:------:|
| Scenario 1    | Scenario 1    | 0.4881 | 0.6071 | 0.5238 |
|               | Scenario 2    | 0.4114 | 0.4669 | 0.4378 |
|               | Scenario 3    | 0.1958 | 0.2844 | 0.1958 |
| Scenario 2    | Scenario 1    | 0.4339 | 0.3598 | 0.4220 |
|               | Scenario 2    | 0.3333 | 0.3810 | 0.2619 |
|               | Scenario 3    | 0.1931 | 0.1799 | 0.1786 |
| Scenario 3    | Scenario 1    | 0.3796 | 0.4352 | 0.3955 |
|               | Scenario 2    | 0.2513 | 0.3386 | 0.2725 |
|               | Scenario 3    | 0.3690 | 0.3452 | 0.2619 |
| All scenarios | All scenarios | 0.4102 | 0.5299 | 0.4145 |
|               |               |        |        |        |

Test accuracy for OurSoundNet trained with distinct supervisory information. G: Ground truth hard labels. D: DualCamNet soft labels. R: ResNet-50 soft labels.

| **Train set** | **Test set**  | **G**  | **D**  | **R**  |
|:--------------|:--------------|:------:|:------:|:------:|
| Scenario 1    | Scenario 1    | 0.6548 | 0.7857 | 0.7262 |
|               | Scenario 2    | 0.4286 | 0.4325 | 0.4960 |
|               | Scenario 3    | 0.1627 | 0.1825 | 0.2989 |
| Scenario 2    | Scenario 1    | 0.4100 | 0.5542 | 0.5106 |
|               | Scenario 2    | 0.3214 | 0.2619 | 0.4524 |
|               | Scenario 3    | 0.1627 | 0.1825 | 0.1799 |
| Scenario 3    | Scenario 1    | 0.3307 | 0.3770 | 0.4405 |
|               | Scenario 2    | 0.2976 | 0.3056 | 0.2765 |
|               | Scenario 3    | 0.5000 | 0.6190 | 0.6071 |
| All scenarios | All scenarios | 0.6966 | 0.7009 | 0.6282 |
|               |               |        |        |        |

Test accuracy for HearNet trained with distinct supervisory information. G: Ground truth hard labels. D: DualCamNet soft labels. R: ResNet-50 soft labels.

**Teacher-Student Networks.** Finally, we compare the performance of the student networks when trained by distilling knowledge from the teacher networks. These results are shown in columns D and R from Tables  and .

We observe that whenever we perform training by transferring either from DualCamNet or ResNet-50 using data from scenario 1, we obtain better results and good generalization. When transferring from DualCamNet, the improvement can be ascribed to the fact that data acquired in the anechoic room is cleaner than in other scenarios. Similarly, when transferring from ResNet-50, there is little clutter in the scene, allowing the network to easily capture the objects involved in the action execution.

Such ideal conditions do not occur in scenarios 3 and (especially) in scenario 2, which are considerably more acoustically noisy and visually cluttered. In particular the worst case is scenario 2, where echoes are even more disturbing than background noise of scenario 3. In fact, in scenario 3, distilling from DualCamNet improves accuracy in all cases except for the OurSoundNet student tested on scenario 3. We are not able to improve results in all testing scenarios when training on scenario 2 as echoes introduce too much noise. ResNet-50 soft labels, on the contrary, do not always guarantee an increase in accuracy neither in scenario 2 nor in scenario 3.

Overall we thus notice that distillation from DualCamNet provides higher improvement over distillation from ResNet-50, especially when training on all scenarios. In some exceptional cases, the teacher is able to help the student even though it could not achieve a good accuracy. For instance, ResNet-50 trained on the scenario 3 achieves a 22.4% test accuracy and HearNet on the same setting reaches 50.0% but, when transferring from the ResNet-50 to HearNet the accuracy improves up to 60.71%.

We validated the chosen hyper-parameters, and found $T=1$ and $\lambda=0.5$ to be the best temperature value and imitation parameter, respectively. This means that we keep the teacher predictions unchanged and give them equal importance than to the hard labels. Interestingly our finding about $\lambda$ is consistent with that of .

In summary, these results show that knowledge distillation allows learning more robust features given there is not much noise corrupting the data.

## Acoustic Features Quality Assessment

Finally, we tested our student networks trained through distillation on a simple classification task on a standard sound benchmark, the DCASE-2018 dataset . Specifically, we performed both k-NN and SVM classification on the features extracted with our distilled student networks to verify whether the learned representations were general enough to perform well in a different audio domain.

Table  reports our results in comparison with those obtained from established baselines which were trained on DCASE-2018 and that of SoundNet-5 pre-trained on Flickr-SoundNet.

For OurSoundNet, we employed features both from the fc1 and conv4 layers, in order to be comparable with the conv5 and conv4 of the original SoundNet-5. We notice that OurSoundNet/conv4 outperforms OurSoundNet/fc1 by around 14%. This might be because fc1 has learned feature which are very specific for our dataset. On the other hand, conv4 layer features for both models perform similarly, because they captures less class-specific information, so also OurSoundNet has more general features.

Finally, for HearNet we considered the features from fc1 and some convolutional layers. We observe that lower layer learn features which are more general and thus transfer better to DCASE-2018. This is reasonable since higher layer encode more label-specific information, and this network was trained from scratch on our dataset. Performances of Hearnet lower layers are similar to those of OurSoundNet/fc1, which was pre-trained on Flickr-SoundNet, but was then adapted to our dataset. In conclusion, features learned with our dataset, which comprises 3 hours of videos, transfer reasonably well to DCASE if compared to features learned from the huge Flickr-SoundNet (2M videos).

| **Features**       | **Training Dataset** | **Test accuracy** |        |
|:-------------------|:--------------------:|:-----------------:|:------:|
| Mesaros *et al.*   |      DCASE-2018      |       0.597       |        |
| Liping *et al.*    |      DCASE-2018      |       0.798       |        |
| Golubkov *et al.*  |      DCASE-2018      |       0.801       |        |
| HearNet/fc1        |         Ours         |      0.2419       | 0.2609 |
| HearNet/conv5      |         Ours         |      0.2488       | 0.2740 |
| HearNet/conv4      |         Ours         |      0.2631       | 0.2967 |
| HearNet/conv3      |         Ours         |      0.2754       | 0.3100 |
| HearNet/conv2      |         Ours         |      0.2810       | 0.3403 |
| OurSoundNet/fc1    | Flickr-SoundNet+Ours |      0.2746       | 0.3014 |
| OurSoundNet/conv4  | Flickr-SoundNet+Ours |      0.4067       | 0.4420 |
| SoundNet-5/conv5   |   Flickr-SoundNet    |      0.4180       | 0.4643 |
| SoundNet-5/conv4   |   Flickr-SoundNet    |      0.4184       | 0.4275 |

Dataset transfer results for DCASE-2018 . Feature extracted by the models distilled from DualCamNet presented in Section  are fed into k-NN (*left*) and SVM (*right*) classifiers. The number of nearest neighbours is validated on the validation set.

# Conclusions

In this work, we investigate whether and how it is possible to transfer knowledge from visual data and spatialized sound, namely, acoustic images, in order to improve audio classification from single microphone. To this end, we take advantage of a special sensor, DualCam, an acoustic-optical camera that provides in output audio-visual data synchronized in time and spatially aligned. Using this sensor, we acquired a novel audio-visually indicated action dataset in 3 different scenarios, from which we aim at extracting information useful for audio classification.

The peculiar nature of the generated acoustic images synchronized with optical frames, never studied before, led to the design of deep learning models in the context of the teacher-student paradigm, in order to assess if this information was transferable and indeed useful for single-channel audio classification. We highlight here that the proposed teacher-student framework is the first able to distill from 2D visual data and acoustic images to a model taking as input a 1D modality, namely, audio signals.

On a set of experiments, in which we learnt from visual data and acoustic images separately, we found out that the distilled models are effective in the audio classification task. Future work aims at further exploring the capabilities of this sensor for detection, recognition, self-supervised learning, sound source localization and cross-modal retrieval.

```latex
\begin{appendix}
\appendixpage

%-------------------------------------------------------------------------
\section{Data Preparation}\label{section:augmentation}
We implemented all of our networks and our data processing pipeline using TensorFlow. In particular we store our dataset in multiple compressed TFRecord files, each of which contains 1 second of synchronized data from the three modalities, video images, raw audio waveforms, and acoustic images. We use the \texttt{tf.data} API to retrieve this data and compose at runtime variable length sequences. We grouped contiguous TFRecord files into full audio-video sequences and then randomly sampled shorter length sequences, e.g. we compose a full audio-video sequence of 30 seconds and sample from it 10 sequences of 5 seconds.

%-------------------------------------------------------------------------
\section{Dataset Splitting}
In Section 3 of the paper, we mentioned our dataset consists of 378 audio-video sequences from 30 to 60 seconds each. However we did not comment on how it was split for training purposes. Since only a few sequences were longer than 30 seconds, and in order to keep a balanced dataset, we cropped all the sequences up to 30 seconds and assign 80\% of them for training, 10\% for validation and 10\% for test.

Splitting the dataset this way accounts for 302 training sequences, 39 validation sequences, and 37 test sequences.
%We additionally perform data augmentation as described earlier in Section~\ref{section:augmentation}, to extract sequences of the required length.
We then extracted sequences of the desired length. In case that the required length was 1 second we extracted 30 samples, while in case the required length was 5 seconds we extracted 6 samples. Extracting more samples would result in a high load of data repeated. Finally to keep some consistence across the experiments, we used a fixed seed for random crops extraction and the epoch number as seed for data shuffling.

%-------------------------------------------------------------------------
\section{Hyperparameter Optimization}
In Section 6 of the paper, we presented the obtained experimental results and mentioned that in some cases we used a different learning rate. Basically we considered only two values, \num{1e-3} and \num{1e-4}. Table~\ref{table:learning_rates} shows the values used throughout all the experiments. For all teacher networks we used a learning rate of \num{1e-4} except for \textit{DualCamNet} which required a bigger value. For the student networks (\textit{OursSoundNet} and \textit{HearNet}) we used a mix of both considered values, and almost always the same across all scenarios settings, except for \textit{HearNet} when trained from \textit{DualCamNet} soft labels on first scenario which required a smaller learning rate.
\begin{table}[h]
    \centering
    \footnotesize
    \begin{tabular}{l|c}
        \multicolumn{1}{c|}{\textbf{Network}} & \textbf{Learning rate} \\
        \hline
        \multirow{1}{*}{DualCamNet} & \multirow{1}{*}{\num{1e-3}} \\
        \multirow{1}{*}{ResNet-50} & \multirow{1}{*}{\num{1e-4}} \\
        \multirow{1}{*}{Temporal ResNet-50} & \multirow{1}{*}{\num{1e-4}}\\
        \multirow{1}{*}{AVNet} & \multirow{1}{*}{\num{1e-4}} \\
        \multirow{1}{*}{HearNet (G)} & \multirow{1}{*}{\num{1e-4}} \\
        \multirow{1}{*}{HearNet (D)} & \multirow{1}{*}{\num{1e-3} and \num{1e-4}} \\
        \multirow{1}{*}{HearNet (R)} & \multirow{1}{*}{\num{1e-3} } \\
        \multirow{1}{*}{OurSoundNet (G)} & \multirow{1}{*}{\num{1e-4}} \\
        \multirow{1}{*}{OurSoundNet (D)} & \multirow{1}{*}{\num{1e-3}} \\
        \multirow{1}{*}{OurSoundNet (R)} & \multirow{1}{*}{\num{1e-4}} \\
    \end{tabular}
    \caption{\small Training learning rates. Supervision is indicated as follows: (G): from ground truth hard labels, (D): from DualCamNet soft labels, (R): from ResNet-50 soft labels.}
    \label{table:learning_rates}
\end{table}

It is worth mentioning that in all cases when training our student networks with distillation, we performed hyperparameters optimization using grid search by cross-validation on the held-out validation set. We basically looked at three hyperparameters, learning rate ($lr$), temperature value ($T$), and imitation parameter ($\lambda$). %These results were not included here since we consider that they did not provide very relevant information.

Finally, regarding the transfer learning results, also presented in Section 6 of the paper, we validated the considered number of nearest neighbors $k$. We computed accuracy with odd values between 7 and 15 included for validation set, choose on it best $k$ and use that value for the testing accuracy which we report.
%, and we found that the best results in the validation set were obtained with $k=9$ except for \textit{SoundNet-5/conv5}, pre-trained on Flickr-SoundNet dataset, where $k=5$ proved to be the best value.

%-------------------------------------------------------------------------
\section{Dataset Qualitative Analysis}
%In this section we present some observations from our dataset that clarify several statements we made on it, but could not be included in the paper due to space limitations.
\label{section:analysis}
In this section we provide additional qualitative insights on the proposed dataset, which may clarify some statements made in the paper.
We first illustrate the problem of visual clutter mentioned in Section 6 of the paper. Figure~\ref{fig:visual_clutter} shows three examples of actions performed over all three scenarios with varying conditions of visual clutter.
Comparing scenarios 1 and 3, it can be observed that on the first case the object involved on the action execution is well visible in the foreground, making easier for the visual models to identify the corresponding action.
With scenario 2 the difficulty is that often other people appear on the background or non-related objects are present on the foreground, thus making it harder to identify the action.
\begin{figure}[h]
\centering
\begin{subfigure}[t]{.32\linewidth}
\includegraphics[width=\linewidth]{images_supplementary/stick_dropping_1}
\includegraphics[width=\linewidth]{images_supplementary/stick_dropping_2}
\includegraphics[width=\linewidth]{images_supplementary/stick_dropping_3}
\caption{\small Stick dropping}\label{fig:1}
\end{subfigure}
\begin{subfigure}[t]{.32\linewidth}
\includegraphics[width=\linewidth]{images_supplementary/clicking_1}
\includegraphics[width=\linewidth]{images_supplementary/clicking_2}
\includegraphics[width=\linewidth]{images_supplementary/clicking_3}
\caption{\small Clicking}\label{fig:2}
\end{subfigure}
\begin{subfigure}[t]{.32\linewidth}
\includegraphics[width=\linewidth]{images_supplementary/crumpling_1}
\includegraphics[width=\linewidth]{images_supplementary/crumpling_2}
\includegraphics[width=\linewidth]{images_supplementary/crumpling_3}
\caption{\small Plastic crumpling}\label{fig:4}
\end{subfigure}
\caption{\small Comparison of three actions performed on all scenarios. From top to bottom, scenario 1 on the first row, scenario 2 on the second row, and scenario 3 on the third row.}
\label{fig:visual_clutter}
\end{figure}

A key finding on the paper was that models based on acoustic data achieved better classification results than models based on visual data.
Here we illustrate the difficulty of identifying actions from visual data in contrast to identifying actions from audio data.
Figure~\ref{fig:similar_actions} shows two subjects on the third scenario performing three different actions each.
It can be seen that some actions involving the same subject are visually similar although they depict completely different actions, but they are distinguishable by their acoustic signature.
\begin{figure}[h]
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_video}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_spectrogram}
        \caption{Speaking}\label{fig:8}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/ripping_video}
        \includegraphics[width=\linewidth]{images_supplementary/ripping_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/ripping_spectrogram}
        \caption{Paper ripping}\label{fig:9}
    \end{subfigure}
        \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/shaking_video}
        \includegraphics[width=\linewidth]{images_supplementary/shaking_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/shaking_spectrogram}
        \caption{Paper shaking}\label{fig:10}
    \end{subfigure}
    \begin{center}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/clapping_video}
        \includegraphics[width=\linewidth]{images_supplementary/clapping_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/clapping_spectrogram}
        \caption{Clapping}\label{fig:11}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/snapping_video}
        \includegraphics[width=\linewidth]{images_supplementary/snapping_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/snapping_spectrogram}
        \caption{Snapping fingers}\label{fig:12}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/whistling_video}
        \includegraphics[width=\linewidth]{images_supplementary/whistling_waveform}
        \includegraphics[width=\linewidth]{images_supplementary/whistling_spectrogram}
        \caption{Whistling}\label{fig:12a}
    \end{subfigure}
    \end{center}
    \caption{\small Comparison of six actions visually similar but distinguishable from audio. All six actions where performed on the third scenario corresponding to the terrace.}
    \label{fig:similar_actions}
\end{figure}

Looking more closely at Figure~\ref{fig:similar_actions}, it can be seen that some actions have a visually distinguishable pattern. For instance, ``clapping'' and ``snapping fingers'' have a periodic pattern and concentrate on the low frequencies rather than on the high ones. Such patterns are more difficult to grasp from raw waveform. This lead us to think that spectrograms are better audio representations since they summarize the scene acoustic content in a better way when compared to raw waveform. This observation gives some more clues into why \textit{HearNet} performs better than \textit{OurSoundNet} in many cases.

Figure~\ref{fig:subjects_knocking} shows the spectrograms for the same action performed by three different subjects on the third location. There can be seen that the same pattern of multiple events spaced at short time intervals with the energy concentrated on the low frequencies, repeats across different subject executions.
\begin{figure}[h]
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_1}
        \caption{Subject 3}\label{fig:13}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_2}
        \caption{Subject 4}\label{fig:14}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_3}
        \caption{Subject 5}\label{fig:15}
    \end{subfigure}
    \caption{\small Comparison of the spectrograms for the ``knocking'' action performed by three distinct subjects on the third scenario.}
    \label{fig:subjects_knocking}
\end{figure}

Figure~\ref{fig:scenarios} compares the spectrograms of the audios of three different actions performed by the same subject on the three considered scenarios. Here we also see that the audios for the same actions  share a visual pattern when visualized as a spectrogram, even when performed across locations.
Interestingly, the cleanest spectrograms are those from actions performed at first scenario, while for second and third scenarios there are two different kinds of noise. In second scenario the noise is mainly due to indoor echoes, while for third scenario it is due to ambient noise.
\begin{figure}[h]
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_4}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_5}
        \includegraphics[width=\linewidth]{images_supplementary/knocking_spectrogram_6}
        \caption{Knocking}\label{fig:22}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_spectrogram_1}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_spectrogram_2}
        \includegraphics[width=\linewidth]{images_supplementary/speaking_spectrogram_3}
        \caption{Speaking}\label{fig:23}
    \end{subfigure}
    \begin{subfigure}[t]{.32\linewidth}
        \includegraphics[width=\linewidth]{images_supplementary/kendama_spectrogram_1}
        \includegraphics[width=\linewidth]{images_supplementary/kendama_spectrogram_2}
        \includegraphics[width=\linewidth]{images_supplementary/kendama_spectrogram_3}
        \caption{Playing kendama}\label{fig:24}
    \end{subfigure}
    \caption{\small Comparison of the spectrograms of three actions performed by the same subject at the three considered scenarios. From top to bottom, scenario 1 on the first row, scenario 2 on the second row, and scenario 3 on the third row.}
    \label{fig:scenarios}
\end{figure}

%-------------------------------------------------------------------------
\section{Dataset Quantitative Analysis}
We report here the confusion matrices for all the student and teacher models, in order to get a deeper understanding of the dataset's challenges.
%with the purpose of understanding which classes are confused with each other, in and to see which networks perform better.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_hearnet_exp7}
    \caption{Hearnet trained on all scenarios confusion matrix.}
    \label{fig:hearnet_cm}
\end{figure}

For HearNet (Figure~\ref{fig:hearnet_cm}) we notice that Hammering is often confused with Knocking, Clicking with Typing, Paper shaking with Plastic crumpling. All the three pairs of classes, in fact, are very similar aurally.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_soundnet_exp7}
    \caption{OurSoundNet trained on all scenarios confusion matrix.}
    \label{fig:soundnet_cm}
\end{figure}

Regarding OurSoundNet (Figure~\ref{fig:soundnet_cm}) many classes are confused with Playing kendama and Stick dropping. Hammering and Knocking, Paper shaking and Stick dropping are confused with each other, Peanut breaking is always misclassified, probably because of its feeble audio pattern. As stated before, HearNet superior performance may be ascribed to its more powerful input representation (spectrogram).
%We can argue that HearNet is better than OurSoundNet and as stated before this can be because of the spectrogram representations.

We now consider the teachers confusion matrices. DualCamNet  (Figure~\ref{fig:dualcamnet_cm}) and AVNet (Figure~\ref{fig:avenet_cm}) confusion matrices have diagonal elements with very high values, indicating high accuracy (they are good teachers indeed).
Temporal ResNet-50 in Figure~\ref{fig:tempresnet_cm} and ResNet-50 in Figure~\ref{fig:resnet_cm} confuse many classes with Clapping and Clicking. Whistling is always misclassified.
%There are also other classes that are wrongly guessed.
As already certified by higher accuracy, we can conclude that are DualCamNet and AVNet are better teacher.
%many classes can be discernible by the spectral signature.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_dualcamnet_exp7}
    \caption{DualCamNet trained on all scenarios confusion matrix.}
    \label{fig:dualcamnet_cm}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_resnet_exp7}
    \caption{ResNet-50 trained on all scenarios confusion matrix.}
    \label{fig:resnet_cm}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_temp_resnet_exp7}
    \caption{Temporal ResNet-50 trained on all scenarios confusion matrix.}
    \label{fig:tempresnet_cm}
\end{figure}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_avenet_exp7}
    \caption{AVNet trained on all scenarios confusion matrix.}
    \label{fig:avenet_cm}
\end{figure}
Finally we can see in detail ResNet-50 confusion matrices when trained and tested on scenario 1 in Figure ~\ref{fig:resnet_1}, scenario 2 in Figure ~\ref{fig:resnet_2} and in scenario 3 in Figure ~\ref{fig:resnet_3}. We notice that when trained and tested on scenario 1, ResNet-50 presents higher accuracies for all classes. In scenario 2 many classes are confused with Clapping, in scenario 3 with Knocking.
In particular, we see in scenario 1 that Snapping fingers, Speaking and Plastic Crumpling are the more difficult to recognize.
In scenario 2 Speaking, Snapping fingers, Playing kendama and Paper shaking have low accuracies.
In scenario 3 many classes have low results, for e.g. Clapping and Snapping fingers. As a matter of fact these classes are visually similar to other ones or sometimes the visual part of the images to recognize the action are occluded or there are other objects  and they can be misunderstood. This confirms the hypothesis made before in Section~\ref{section:analysis} in Figure~\ref{fig:similar_actions} are true.
%but also other classes different from what mentioned can be confused with each other.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_resnet_exp1_location1}
    \caption{ResNet-50 trained and tested on scenario 1 confusion matrix.}
    \label{fig:resnet_1}
\end{figure}
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_resnet_exp2_location2}
    \caption{ResNet-50 trained and tested on scenario 2 confusion matrix.}
    \label{fig:resnet_2}
\end{figure}
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\columnwidth]{images_supplementary/confusion_matrix_resnet_exp3_location3}
    \caption{ResNet-50 trained and tested on scenario 3 confusion matrix.}
    \label{fig:resnet_3}
\end{figure}

%-------------------------------------------------------------------------
\section{Reproducibility}

To enable reproducibility of our results and to motivate further research on deep learning for acoustic images, our code\footnote{https://github.com/afperezm/acoustic-images-distillation}, data, and models are publicly available.

\end{appendix}
```

[^1]: Short-Time Fourier Transform

[^2]: The number of iterations varies with the size of the training set.


author: $^{\; (1)\;}$, T. Di Girolamo$^{\;(2,3)\;}$, F. Longo$^{\;(4,5)\;}$, P. Vallania$^{\;(6,7)\;}$, C. Vigorito$^{\;(7,8)\;}$ $\;\;\;\;$for the CTA Consortium[^1]
$^{(1)}$ Istituto Nazionale di Fisica Nucleare, Sezione di Bari, Bari, Italy
$^{(2)}$ Dipartimento di Fisica, Università degli Studi di Napoli, Napoli, Italy
$^{(3)}$ Istituto Nazionale di Fisica Nucleare, Sezione di Napoli, Napoli, Italy
$^{(4)}$ Dipartimento di Fisica, Università degli Studi di Trieste, Trieste, Italy
$^{(5)}$ Istituto Nazionale di Fisica Nucleare, Sezione di Trieste-Udine, Trieste, Italy
$^{(6)}$ Osservatorio Astrofisico di Torino dell'Istituto Nazionale di Astrofisica, Torino, Italy
$^{(7)}$ Istituto Nazionale di Fisica Nucleare, Sezione di Torino, Torino, Italy
$^{(8)}$ Dipartimento di Fisica, Università degli Studi di Torino, Torino, Italy
E-mail: `email@example.com`
title: Prospects for Gamma-Ray Bursts detection by the Cherenkov Telescope Array

# Introduction

Gamma–Ray Bursts represent a very interesting case study in astrophysics, mainly due to their multi–disciplinary nature. At present time, a GRB can trigger one or more of the dedicated instruments based on several satellites orbiting around the Earth, such as Swift, Fermi, MAXI or INTEGRAL. The observed keV-MeV prompt emission may be accompanied by X–ray, optical or radio emission. Rapid follow–up of this keV–MeV emission is possible thanks to communication through the Gamma-ray Coordinates Network (GCN), where the GRB position is spread out in real time to all other observatories. This includes all currently operative Imaging Atmospheric Cherenkov Telescopes (IACTs) like MAGIC, H.E.S.S., and VERITAS. Unfortunately, none of them ever succeeded in capturing a high-energy signal from a GRB, but several upper limits from a single or from a sample of bursts were published by each collaboration over the last years. In this contribution, we aim to investigate the possibility by the Cherenkov Telescope Array (CTA) to detect such elusive GRB emission.

# CTA configurations

CTA is a worldwide project aiming to build and operate a third generation of IACTs. In its current design, two huge arrays for a total amount of more than 100 telescopes are foreseen, one in each hemisphere, to extend the energy range of currently operating IACTs mainly to higher energies and to improve the sensitivity of about one order of magnitude with better angular and energy resolutions.

In order to cover the energy range from 20 GeV to more than 100 TeV, three kinds of telescopes are envisaged: Large Size Telescopes (LSTs, 20 GeV $\div$ 100 GeV), Medium Size Telescopes (MSTs, 100 GeV $\div$ 10 TeV), and Small Size Telescopes (SSTs, 10 TeV $\div$ \> 100 TeV). This configuration is driven by the features of the Cherenkov signal at different energies: near the threshold, the number of source photons is relatively high but the Cherenkov image is poor, so few huge telescopes (3 or 4, with a diameter of 23 m) are used to collect the faint showers. In this region, the challenge is to distinguish between Cherenkov and Night Sky Background (NSB) photons, and to discriminate primary gammas from the overwhelming flux of cosmic rays hadrons. On the other hand, the effective area is not an issue, so a small number of big telescopes close to each other is the best configuration. At higher energies, the Cherenkov signal starts to increase and the flux from the source is rapidly fading, so a compromise array of $\sim$ 25 telescopes of 12 m diameter scattered over an area of around 3 km$^2$ is the best choice. At the highest energies the situation is completely different: the Cherenkov signal is very strong, but the steep spectra reduce the signal to a handful of primary gammas. To overcome this, a huge effective area up to 6-7 km$^2$, covered by $\sim$ 70 SSTs (4 m diameter) is needed. We notice that for energies $\gtrsim$ 10 TeV the Cherenkov images are so rich that it is not only easy to discard the NSB photons, but also to discriminate gammas from hadrons, approaching a background-free working mode for which the sensitivity is proportional to the effective area and not to its square root. Due to the Extragalactic Background Light (EBL) absorption, that strongly depresses the high–energy spectra of distant sources detected on Earth (the limit being E $\lesssim$ 1 TeV for z $\geq$ 0.1) and to the fact that the Galactic plane is mostly visible from the Southern hemisphere, the SST array is not expected to be built in the Northern observatory.

# High–energy GRB observations

Due to their elusive nature, GRBs represent a very interesting candidate for future observations by CTA, as highlighted also in . Prospects for VHE GRB observations by CTA were also presented in detail by . Most of these analyses relied on extrapolations taken (1) from the GRB spectral parameters published in the catalogs of the Burst and Transient Source Experiment (BATSE, 20 keV–2 MeV) or of the *Swift* Burst Alert Telescope (BAT, 15–150 keV); and (2) from some very energetic GRBs detected by Fermi before 2012.

Since June 2008, the Fermi mission is constantly enhancing the number of GRB detections. The Fermi Gamma-Ray Burst Monitor (GBM, ) is sensitive in the energy rate bewteen 8 keV to 40 MeV and its field of view covers almost 4 $\pi$ sr. The GBM trigger rate lies at $\sim$`<!-- -->`{=html}250 GRB/yr (i.e. higher than BAT with $\sim$`<!-- -->`{=html}100 GRB/yr) and the second GBM catalog (covering 4 years of operation) includes almost 1000 bursts . Furthermore, thanks to the LAT onboard Fermi, the number of high–energy GRBs dramatically increased with respect to the bunch of events seen in the '90s by the Energetic Gamma–ray Experiment Telescope (EGRET, 20 MeV–30 GeV). LAT operates at energies between 100 MeV and $>$`<!-- -->`{=html}300 GeV and its first GRB catalog already included $\sim$`<!-- -->`{=html}30 GRBs observed during the first three years of operation above 100 MeV with the standard event reconstruction analysis. This number gets even larger when considering dedicated low–energy techniques (LLE) exploring the energy region between 10 and 100 MeV, where the LAT and GBM energy ranges overlap.

The current number of official LAT GRB detections is constantly kept up–to–date on the LAT Public Table website and lists almost 100 GRBs at the time of this writing, that is after seven years into the Fermi mission. However, this number is meant to grow as soon as the new event reconstruction algorithm (the so–called *Pass 8*) will be released and the catalog will be updated (see for more details).

Fermi's GRBs exhibit dozen of GeV photons with unprecedented high energies. The current record holder is GRB 130427A, which emitted a 95 GeV photon and was detected at a low redshift of 0.34. It was followed–up by a very large number of telescopes, including the Veritas array . Unfortunately, Veritas' observations only began almost 20 hours post trigger, leading to no GRB detection. The upper limit (UL) was placed at 3.3 $\times\,10^{\,-12}$ erg cm$^{\,-2}$ s$^{\,-1}$.

In the near future, we plan to revisit the latest prospects for VHE GRB observations in the light of the newest simulations produced by the CTA collaboration. We want to focus on the extrapolation to high energies of Fermi–like GRBs, both from the prompt emission using the GBM sample and from the late emission using the LAT sample, in particular using those bursts with measured redshift. For this contribution, we use the extremely fluent GRB 130427A as a test case and see how well we might observe this event at several epochs after the trigger with various CTA configurations.

# Simulation of GRB observations with ctools

In order to estimate the possible detection of GRBs by CTA, we intend to set up a library of GRBs at different times, extrapolating the LAT flux to the highest energies and properly taking into account the time evolution of their flux. However, since we do not yet model the effect of the EBL, our current simulations are limited up to 100 GeV, an energy where the effect of the cosmological attenuation is not relevant (see Section 5. for more details).

To estimate the detectability of a GRB we make use of the *ctools*, a software package specifically developed for the scientific analysis of CTA data . In these first tests, we simulate the high–energy emission of GRB 130427A as detected by LAT, with a spectral index $\gamma = 2.2$ (which is almost constant from 400 s up to 70 ks post trigger) and a power law decay with a temporal index $\tau = 1.35$ (valid for t $>$ 380 s). We cross checked our extrapolation to the CTA energies by comparing it with the UL placed on this GRB by VERITAS . This was calculated assuming an intrinsic GRB spectrum of (dN/dE) $\propto$ E$^{−\gamma}$ absorbed using the EBL model of .

We simulate two possible observations of this GRB by CTA: (a) the first one lasting 10 min at t $=$ 1 ks post trigger; (b) the second one lasting half an hour at t $=$ 10 ks post trigger. We assume both observations on axis with respect to the CTA array and a zenith angle $\theta = 20^{\,\rm o}$. Figure shows two count maps of the simulated GRB in the energy range 20 GeV–100 GeV. They were obtained using the *ctools* functions *ctobssim* and *ctbin*, and adopting the recently provided CTA instrument response functions (IRFs) . In this particular case, we made use of the *North_0.5h* and *North_5h* IRFs, respectively. A preliminary *ctlike* analysis is performed on the two observations, getting a significant detection in both cases. We plan to perform similar analyses for other Fermi GRBs with known redshift, using different observing profiles and spectra.

# Effect of the Extragalactic Background Light

The interaction of extragalactic VHE photons with EBL at UV–optical wavelengths produces an $e^+ e^-$ pair and thus an exponential attenuation of the gamma–ray flux. This absorption increases with the redshift of the source, the gamma–ray energy and the photon density of the EBL (for a review, see ). Moreover, it is particularly important for GRBs, since they are cosmological sources with a mean redshift $z \simeq 2$. Many EBL models have been published in the last decades, with a general trend towards a decrease of the corresponding optical depth due to the observation of VHE photons at larger redshifts , which were recently detected from blazars at $z \simeq 1$ . The EBL models are getting close to the firm lower limits derived from integrated galaxy number counts . As an indication, the high–energy spectrum of a source at z$\simeq1$ shows a cutoff at E$\approx$`<!-- -->`{=html}100 GeV. In our future work we will consider the EBL model by . For simplicity, in our simulations we will extend the spectrum only up to 1 TeV, the maximum energy after which the source is assumed to be totally absorbed.

However, the observed spectral indices in blazars do not seem to follow the amount of softening with increasing redshift predicted by the EBL absorption , and there are other indications of an overestimation of the EBL photon density (e.g., ). A possible explanation of these results could be the oscillation from photons to Axion–Like Particles (ALPs), which propagate unimpeded over cosmological distances before reconversion, reducing the optical depth along the VHE gamma–ray path . In this framework, GRBs, with their cosmological distances, may be useful to add stronger constraints on the EBL and give new hints on the existence of ALPs.

# CTA operating modes

CTA's complex and varied experimental layout will be run in different ways depending on the target features:

1.  highest sensitivity observations, with all telescopes pointing toward a single source;

2.  normal operations, with the array split into sub–arrays with different targets;

3.  sky survey, with the aim of covering large portions of the sky to detect new or transient sources.

Case 1. applies mainly to flaring or varying sources, when the observation cannot be postponed and the maximum sensitivity over the whole energy range is required. This is not the case of deep observations, since the comparison of the primary spectra with the expected sensitivity shows that the core energy window (around 1 TeV) will be covered in a time considerably smaller than the high energy tail. We therefore expect for deep field observations that the SST array will make very long exposures on selected candidates suggested by MSTs on the basis of hard spectrum or high flux. This is case 2., with the array split into the different telescope types, but mainly for MSTs, this can apply also to telescopes of the same type. Case 3. is the most relevant one for this study, since it allows the detection of serendipitous sources, variable or unexpected, and GRBs are both. This observation mode will be therefore analyzed in more details.

The current field of view (FoV) of operating IACT arrays (4–5$^{\circ}$) will be widened by MSTs and SSTs up to 8–10$^{\circ}$. Unfortunately, since the detection of primary photons is through the Cherenkov images, the sensitivity is not uniform inside the FoV and dramatically drops towards its edge. This means that, if we want to enlarge the total FoV by pointing the telescopes towards different directions, the FoV of each telescope must overlap in order to obtain a uniform sensitivity. At present, only the Galactic plane was scanned, starting from HEGRA up to the 1400 hours–long survey by H.E.S.S. . Outside the Galactic plane, only small promising regions were observed, limiting our knowledge of the VHE sky to a few per cent.

In order to efficiently scan large portions of the sky, the divergent mode was firstly proposed. In this mode, instead of having all telescopes pointing towards the same direction (the so called "parallel mode"), each telescope points to an angle slightly increasing from the center to the edges of the array. Another possibility, the so called "convergent mode", envisages that this angle is reduced from the array center to its edges (see the sketch in Figure ). Both modes were deeply studied for MSTs by means of an accurate simulation of the shower development in the atmosphere and of the telescopes response in , for different angle separations. The results show that the maximum sensitivity for sky survey is achieved in divergent mode with large offset angles ($\sim$ 6$^{\circ}$ from the central to the outermost telescopes), with a decrease of the observing time for a given sensitivity by a factor of $\sim$`<!-- -->`{=html}2.3 with respect to the parallel mode. However, the angular and energy resolutions are worsened by a factor up to 2. The convergent mode is a better choice at high energy, favoring the observation of sources with hard spectra. In this work, we propose a third possibility, that is a mixture between parallel and divergent modes.

In the past, exciting observations made by Whipple in the late '80s were performed using a single telescope recording the Cherenkov images by means of a pixelated camera that allowed the discrimination between gammas and hadrons. With just one image, the core location (and thus the primary energy) could be badly determined, limiting the detection inside the Cherenkov "pool" (with radius r$_p$ $\sim$ 120 m) where the lateral distribution of photons is approximately flat. After imaging, the next fundamental improvement to the IACT technique was the stereo approach firstly used by HEGRA: with at least two images, the shower axis could be determined geometrically with a better resolution on both energy and arrival direction. This approach was so widely adopted that even for CTA the main trigger will be given by the coincidence of at least two telescopes, and the mono events will be collected only for calibration and testing purposes.

Since for IACTs the single observing unit is a couple of nearby telescopes, we propose here a "coupled divergent mode" in which couples of telescopes are pointing to slightly different positions as in Figure b) for single ones. The tilt angle will be chosen on the basis of the sensitivity decrease over the FoV for each couple of telescopes. If, for example, the sensitivity is halved at 3$^{\circ}$, a tilt angle of 6$^{\circ}$ will assure a uniform sensitivity at least along the line connecting the center of the FoV of the two couples of telescopes. For the entire sky we expect that this value will be reduced, and we plan to perform an accurate simulation to optimize this separation angle. Even if the main contribution in GRB searches will come from LSTs, which due to their paucity can hardly benefit of whatever divergent mode, the contribution of MSTs could still be not negligible. The medium sized telescopes are not designed for fast slewing as LSTs (which can repoint within 100 s or less), but there are indications from EGRET and from Fermi observations that the VHE emission is delayed with respect to the prompt phase. Concerning the energy window, the energy range from 100 GeV to 1 TeV corresponds respectively to an horizon from z $\simeq$ 1 to z $\simeq$ 0.1, depending on the EBL absorption model. For follow–up observations, the GRB location is often given by fast satellite analysis with large uncertainties, mostly greater than the 50% sensitivity FoV, so some kind of divergent mode must be applied. Moreover, a sky survey with large angular acceptance could reveal a serendipitous GRB.

In the future, Monte Carlo simulations to obtain the expected performance of the MST array operated in "coupled divergent mode" are planned, and the results will be compared with those presented in . An estimate of the rate of serendipitous GRBs detectable during the sky survey will be also given.

# Acknowledgments

We gratefully acknowledge support from the agencies and organizations listed under Funding Agencies at this website: `http://www.cta-observatory.org/`.

[^1]: Full consortium author list at `http://cta-observatory.org`


abstract: We report the observation of macroscopic field-free orientation, i.e. more than 73 % of CO molecules pointing in the same direction. This is achieved through an all-optical scheme operating at high particle densities ($>10^{17}$cm$^{-3}$) that combines a one-color ($\omega$) and a two-color ($\omega + 2\omega$) non-resonant femtosecond laser pulses. We show that the achieved orientation solely relies on the hyperpolarizability interaction as opposed to an ionization-depletion mechanism thus opening a wide range of applications. The achieved strong orientation enables us to reveal the molecular-frame anisotropies of the photorecombination amplitudes and phases caused by a shape resonance. The resonance appears as a local maximum in the even-harmonic emission around 28 eV. In contrast, the odd-harmonic emission is suppressed in this spectral region through the combined effects of an asymmetric photorecombination phase and a sub-cycle Stark effect, generic for polar molecules, that we experimentally identify.
author: P. M. Kraus; D. Baykusheva; H. J. Wörner
date: 2024-10-02
title: Two-pulse field-free orientation reveals anisotropy of molecular shape resonance

Techniques for fixing molecules in space are invaluable tools for a broad range of experiments in ultrafast science . The availability of transiently aligned molecular samples has particularly advanced strong-field and attosecond spectroscopies, providing new insights into the electronic structure of molecules and its temporal evolution . High-harmonic spectroscopy (HHS) provides a new access to the rich structures of photoionization continua, such as Cooper minima and shape resonances . The investigation of the inherent structural and dynamical anisotropies of polar molecules has however been prevented by the difficulty of orienting molecules. Interesting phenomena tied to polar molecules include the predicted recombination-site dependence of structural minima and attosecond charge migration triggered by strong-field ionization. Here, we demonstrate a protocol for molecular orientation which achieves macroscopic field-free orientation and exploit this progress to probe the anisotropy of photorecombination dipole moments at a molecular shape resonance.

Successful approaches to laser-induced molecular orientation include the combination of an electrostatic field with a rapidly turned-off laser field , alignment in combination with quantum-state selection and a weak dc-field , adiabatic and impulsive two-color orientation . All of these techniques are subject to substantial limitations: The presence of electric fields may alter the electronic structure of the molecule, its photo-induced dynamics or the subsequent probing process. The low particle densities available after quantum-state selection make the application of such techniques to high-harmonic and attosecond spectroscopies challenging to impossible. The two-color scheme , recently applied to HHS , relies on an ionization-depletion mechanism and thus ties the achievable degree of orientation to the ionization fraction of the sample. This fact does not only limit orientation to modest degrees but it also makes charged-particle detection difficult.

In this letter, we describe a technique of molecular orientation that overcomes all of the above-mentioned limitations. By applying a one-color alignment pulse (170 fs duration, (5$\pm$`<!-- -->`{=html}1)$\times$`<!-- -->`{=html}10$^{13}$ W/cm$^2$) followed by a two-color orientation pulse (150 fs, (6$\pm$`<!-- -->`{=html}1)$\times$`<!-- -->`{=html}10$^{13}$ W/cm$^2$ ) to CO molecules in a supersonic expansion (Fig. 1(a)), as theoretically proposed in , we achieve macroscopic orientation, i.e. more than 73 % of the molecules pointing in the same direction, under completely field-free conditions. Details of the experimental setup have been given elsewhere . The degrees of orientation obtained in this study rival those obtained after quantum-state selection with a $\sim$`<!-- -->`{=html}10$^{6}$-fold increase in particle density. Moreover, we show through calculations that the present technique solely relies on the interaction of the laser field with the molecular hyperpolarizability thus circumventing the need to ionize major parts of the sample .

We exploit this technique to showcase the complementary encoding of electronic-structure information in the even and odd high-harmonic spectra of oriented molecules. We identify a shape resonance in CO around 28 eV that manifests itself as a pronounced local maximum in the intensity of the even orders while the odd orders are suppressed in the same region. Quantitative modelling shows that this remarkable observation is the consequence of a $\sim\pi$-phase shift of high-harmonic radiation emitted from the two ends of the molecule from the 15$^{\rm th}$ to the 22$^{\rm nd}$ order (H15-H22). This phase shift is caused by the combined effects of the recombination phase and a sub-cycle Stark effect, generic for polar molecules. The present study thus opens the field of side-dependent attosecond photorecombination delays to polar molecules.

Figures 1(b) and 1(c) illustrate the concept of our experiment through calculated values of orientation and alignment parameters for CO molecules ($B=1.93128~\textnormal{cm}^{-1}$, $T_{\textnormal{rot}}=8.63~\textnormal{ps}$, $I_p= 14.014~\textnormal{eV}$ ). The two-color orientation pulse alone creates only weak net orientation at the full rotational revival ($T_{\textnormal{rot}}$, see Fig. 1(b)), whereas it generates strong orientation in the isolated contributions of the even- and odd-*J* (rotational quantum number) levels at the rotational half-revival (around $T_{\textnormal{rot}}$/2). These contributions however interfere destructively yielding no net orientation. The application of a one-color pulse $\Delta t_1=T_{\textnormal{rot}}$/4 before the two-color pulse, leads to the alignment (anti-alignment) of the odd- (even-) $J$ contributions (Fig. 1(c)). The two-color pulse can then selectively orient the pre-aligned odd-$J$ contributions and create strong net orientation at the rotational half-revival. Fine-tuning of the exact delays even enables constructive interference to be achieved between odd- and even-$J$ contributions to orientation .

Figure 2(a) shows the high-harmonic spectrum of CO molecules oriented with the two-pulse scheme. The delay between the two-color orientation and alignment pulses was set to $\Delta t_1=2.25$ ps and the delay between the orientation and HHG pulses was $\Delta t_2=3.90$ ps. The emission of intense even-harmonic orders, with H18 and H19 displaying comparable intensities, indicates macroscopic orientation of the sample. The even-harmonic emission is dramatically enhanced as compared to orientation by a single two-color laser pulse shown in Fig. 2(b). This latter spectrum was recorded at a pump-probe delay of 8.85 ps and shows even-harmonic emission that is weaker by a factor of $\sim$ 9, as compared to the spectrum obtained with the two-pulse scheme (Fig. 2(a)). In both experiments, the time delays were optimized to obtain maximal orientation whereas the other experimental settings were kept identical. The evolution of the even-harmonic emission as a function of $\Delta t_2$ is shown in Fig. 2(c) for a representative harmonic order (H18, 27.9 eV).

We calculate the wave-packet dynamics by solving the time-dependent Schrödinger equation (TDSE) as previously described in e.g. using the hyperpolarizability interaction and additionally incorporate the ionization-depletion mechanism with details given in Ref. . Figure 2(d) shows the calculated value of $\braket{\cos\theta}^2$ which nicely reproduces the temporal evolution of the even-harmonic signal. Strikingly, orientation is found to exclusively result from the hyperpolarizability interaction (dashed red line) whereas the contribution of the ionization-depletion mechanism (dash-dotted green line) is negligible. The present two-pulse scheme for orientation is thus the first field-free scheme which orients molecules without ionizing a substantial fraction of the sample. Hence, it is a valuable tool for all probe techniques that suffer from ionization of the sample by pump pulses, such as photoelectron spectroscopies.

We now study the encoding of electronic-structure information in the even- and odd-harmonic spectra of polar molecules and quantify the degree of orientation. A local maximum is observed in the even-harmonic spectrum at H18 (27.9 eV) (Fig. 2(a)), which is even more pronounced in the ratio of the even harmonics divided by the mean value of their two adjacent odd harmonic orders (the even-to-odd ratio, red dots in Fig. 3(a)). This local maximum is independent of the probe-pulse intensity. Contrary to the even-harmonic spectrum, the intensities of the odd harmonic orders are continuously decreasing over the spectral range where the even harmonic emission maximizes. The analysis of the temporal modulation of the odd harmonic intensities further shows that the latter are strongly suppressed in the same spectral region (H17-cutoff, 26.3-48 eV) when CO is aligned parallel to the HHG probe pulse. This observation is illustrated in Figs. 3(d) and (e) which show results from a one-color-pump HHG-probe experiment under conditions otherwise identical to those described above. The calculated degree of axis alignment measured by $\braket{\cos^2\theta}$ (Fig. 3(c)) maximizes around 8.8 ps. At this delay, the intensity of the low harmonic orders (H9-H15, Fig. 3(d)) maximizes as well, whereas the intensity of the higher harmonics (H17-cutoff, Fig. 3(e)) minimizes .

To explain these observations, we simulate the high-harmonic spectra of oriented CO molecules. We calculate the single-molecule induced dipole moment as product of the recombining electron wave packet – obtained as the product of the ionization amplitude $\sqrt{I_\mathrm{ion,i}(\theta)}$ and a propagation factor $a_\mathrm{prop,i}(\Omega,\theta)$ – and the photorecombination dipole moment $d_\mathrm{rec,i}(\Omega,\theta)$. $\theta$ represents the polar angle between the molecular axis and the laser field and $\Omega$ stands for the angular frequency of the emitted radiation. The induced dipole moment is coherently averaged over the molecular axis distribution $A(\theta)$ obtained from a TDSE calculation and summed over the channels $i$ corresponding to ionization from the highest-occupied molecular orbital (HOMO) and HOMO-1 : $$\begin{split}
\ d(\Omega) \propto &\sum_i \int_{0}^{\pi} A(\theta) \sqrt{I_\mathrm{ion,i}(\theta)}a_\mathrm{prop,i}(\Omega,\theta)
\\ & \times d_\mathrm{rec,i}(\Omega,\theta) \sin\theta \, \mathrm{d}\theta.
\label{eq:HHGdip}
\end{split}$$

We refer here to orbitals rather than electronic states of the cation because the X $^2\Sigma^+$ and A $^2\Pi$ states of CO$^+$ are well described by single-hole configurations of the neutral molecule. We obtain the ionization rate for the HOMO of CO from recent theoretical work and calculate the ionization rate of HOMO-1 using the partial Fourier-transform method . Their relative rates are chosen to reproduce the results of Ref. and scaled to the intensity applied in our study. We obtain the complex amplitude of $a_\mathrm{prop,i}(\Omega,\theta)$ numerically using the strong-field approximation . Only the phase of $a_\mathrm{prop,i}(\Omega,\theta)$ is angle dependent as a consequence of a linear sub-cycle Stark effect of the ionized orbital which can be expressed as scalar product of the permanent dipole moment of the orbital and the return velocity of the electron . The Stark phase is thus independent of the intensity and wavelength of the driving laser field and only depends on the emitted photon energy. The photorecombination dipole moments are obtained through *ab initio* quantum scattering calculations using epolyscat with a polarized valence-triple-zeta (pVTZ) basis set.

From our experimental data we obtain an even-to-odd ratio of 0.44 determined as an average over the harmonic orders 12 to 24 (H12-H24). This allows us to roughly estimate $\eta=\frac{n_\textnormal{up}-n_\textnormal{down}}{n_\textnormal{up}+n_\textnormal{down}}=0.66$, i.e. $\zeta=\frac{n_\textnormal{up}}{n_\textnormal{tot}}=0.83$, because the spectrally-averaged even-to-odd ratio is proportional to $\eta^2$. This is consistent with both the experimentally observed enhancement of the even orders by a factor of 9 and the corresponding improvement in the degree of orientation $\eta$ by a factor of 3 compared to Ref. . Since the odd (even) harmonic orders correspond to the sum (difference) of the electric fields emitted from the two sides of the molecule , we simulate the odd (even) harmonic spectra using a symmetrized, $A_{\mathrm{sy}}(\theta)=(A(\theta)+A(\pi-\theta))/2$ (anti-symmetrized, $A_{\mathrm{as}}(\theta)=A(\theta)-A(\pi-\theta)$/2) version of the normalized axis distribution $A(\theta)$ obtained from the TDSE calculation shown in Fig. 3(b) ($\zeta=0.73$, $\braket{\cos\theta}=0.38$). The calculated and measured even-to-odd ratios are shown together in Fig. 3(a). The full theory (solid red line) is in very good agreement with the measured even-to-odd ratio (red dots). From these two independent approaches to estimating the degree of orientation, we conclude $0.73 \leq \zeta \leq 0.83$.

We now rationalize the maximum in the even-harmonic emission and the concomitant suppression of odd-harmonic emission. An excellent agreement between theory and experiment is achieved when both HOMO and HOMO-1 are taken into account (red line in Fig. 3(a)). Neglecting the contributions of the Stark effect shifts the maximum from H18 to H14 and substantially deteriorates the overall agreement (green dashed line), demonstrating the importance of the Stark effect. However, the local maximum at 28 eV is unchanged when HOMO-1 is excluded from the calculation, which allows us to concentrate on HOMO. Figure 4(a) displays the calculated differential photoionization cross section of the HOMO of CO for photoemission along the positive $z$-axis defined by the polarization of the ionizing radiation. This cross section displays a pronounced maximum around H18 (27.9 eV). The maximum occurs both when the oxygen (0$^\circ$ alignment angle) or the carbon (180$^\circ$) atom point in the positive $z$ direction, with the latter being more pronounced. We demonstrate the origin of this maximum by decomposing the cross section into the contributions from individual partial waves (Fig. 4(b)). The prominent maximum in the $\ell=3$ contribution (dash-dotted blue line) arises from a centrifugal barrier in the effective potential determined by the angular momentum $\ell$ of the continuum electron, which causes the latter to be trapped in a quasibound state. This phenomenon is commonly referred to as "shape resonance" and has been observed at 23.6 eV in the partial photoionization cross section of CO pertaining to the X $^2\Sigma^+$ state of CO$^+$ . The shape resonance observed in our experiment lies 4.3 eV higher compared to the shape resonance observed in photoionization. This shift is mainly caused by the Stark effect as shown in Fig. 3a. The observed width of the shape resonance (7.5-8 eV) is very similar to that observed in photoionization (8 eV).

We now demonstrate an application of HHS to probing the hitherto inaccessible anisotropy of molecular shape resonances. The shape resonance causes unequal enhancements of the differential photoionization cross section on the two sides of the molecule (Fig. 4 (b)). On the oxygen end, the partial-wave contributiosn with $\ell=1$ and $\ell=2$ interfere destructively, such that the energy-dependence of the $\ell=3$ contribution is recovered. The carbon side shows an enhanced cross section caused by a constructive interference of the $\ell=1-3$ contributions. This demonstrates that HHS of oriented polar molecules directly probes the energy- and angle-dependent contributions of isolated partial waves to the photoionization cross section. The shape resonance further causes a pronounced energy dependence in the phase of the photoionization matrix elements. This is illustrated in Fig. 4 (c) which shows the difference of the photorecombination dipole phases between C- and O-sides (dash-dotted line). The total phase difference between high-harmonic emission from the C- and O-sides (solid black line) is obtained by adding the Stark-phase contribution to the difference of the photorecombination phases and is found to be nearly constant around $\Delta\varphi=\pi\pm0.1~\textnormal{rad}$ in the range of H15-H22 (22-34 eV). Figure 4(d) illustrates the effect of this phase difference on the high-harmonic spectrum. The squared magnitudes of the coherent sum (dash-dotted blue line) and difference (solid red line) of the recombination dipoles from the C- and O-sides including the Stark phase are shown, reflecting the intensity envelopes of the odd and even harmonic orders. The $\sim\pi$ phase difference in the range H15-H22 causes a strong suppression of the odd harmonics in this region, whereas the even harmonics display a maximum at H18.

These results explain why the even-harmonic emission maximizes around the photon energy of the shape resonance (Fig. 3(a)), whereas the odd-harmonic emission from aligned molecules is suppressed in the same spectral range (Fig. 3(e)). High-harmonic spectroscopy is thus shown to be sensitive to the anisotropic dipole phase caused by a molecular shape resonance. Our analysis further reveals the striking importance of the sub-cycle Stark effect. If the latter is neglected, the dipole phase difference reaches $\pi$ around H12 (Fig. 4(c)), explaining why the maximum in the even-harmonic spectrum is shifted to lower photon energies compared to the experiment when the Stark effect is neglected (dash-dotted green line in Fig. 3(a)).

In conclusion, we have demonstrated an all-optical technique that creates macroscopic degrees of field-free molecular orientation, i.e. more than 73 % of the molecules pointing in one direction. This technique creates orientation solely through the hyperpolarizability interaction and overcomes the limitations of previous techniques such as low degrees of orientation, low particle densities or ionization of the sample. This progress enables us to characterize the anisotropy of complex photorecombination dipole moments at a molecular shape resonance. These anisotropies were inaccessible to previous synchrotron studies which only provided angle-averaged photoionization cross sections. In particular, the phases of photoionization dipole moments become accessible to HHS and contain valuable information on the electronic structure and dynamics. The pronounced asymmetry of the recombination dipole phase combined with a sub-cycle Stark effect, generic for polar molecules, causes the high-harmonic emission from the two sides of the molecule to be $\sim\pi$ radians out of phase over an extended spectral range. This leads to the appearance of a spectral maximum only in the even-harmonic spectrum while the odd harmonics are suppressed in the same spectral region. The antagonistic encoding of spectral features, such as photorecombination resonances or Cooper minima, is expected to be generic for even- and odd-harmonic spectra as a consequence of their complementary dependence on the emissions from opposite recollision sites. The present results make macroscopic orientation available to high-harmonic and attosecond spectroscopies and open new directions of research such as the study of electronic dynamics in bound states, continuum resonances or photorecombination delays in polar molecules.