abstract: We present a sample-based, autoregressive (AR) method for the generation and time evolution of atmospheric phase screens that is computationally efficient and uses a single parameter per Fourier mode to vary the power contained in the frozen flow and stochastic components. We address limitations of Fourier-based methods such as screen periodicity and low spatial frequency power content. Comparisons of adaptive optics (AO) simulator performance when fed AR phase screens and translating phase screens reveal significantly elevated residual closed-loop temporal power for small increases in added stochastic content at each time step, thus displaying the importance of properly modeling atmospheric "boiling". We present preliminary evidence that our model fits to AO telemetry are better reflections of real conditions than the pure frozen flow assumption.
address:  $^1$Dept. of Astronomy & Astrophysics, University of California Santa Cruz, Santa Cruz 95064, USA
  $^2$Lawrence Livermore National Laboratory, Livermore, CA 94450, USA
author: Srikar Srinath$^{1*}$, Lisa A. Poyneer$^2$, Alexander R. Rudy$^1$ and S. Mark Ammons$^2$
title: A computationally efficient autoregressive method for generating phase screens with frozen flow and turbulence in optical simulations

# Introduction

End-to-end simulation of the next generation of telescopes, their adaptive optics systems, instrumentation, and control systems is increasingly a necessity given their high cost and complexity. For existing systems with planned upgrades or changes, minimizing disruption by conducting rigorous computer or lab testbed simulations is essential. Simulations of the entire optical pipeline and control algorithms have been employed with great success in programs such as the Gemini Planet Imager (GPI) . Systems under construction (the Large Synoptic Survey Telescope (LSST) ) or in the early design stage (the Thirty Meter Telescope ) have devoted significant fractions of their time and monetary budgets to producing detailed error budgets. Systems like LSST, which are particularly sensitive to atmospheric variability , need atmosphere models that closely match real conditions while not extending their already significant computational requirements.

Underpinning all simulations is the atmosphere model through which wavefronts from artificial sources propagate and accumulate phase errors that a simulation in turn propagates through a telescope's optics to observe the effect on the system's point spread function (PSF). Systems with active or Adaptive Optics (AO) then make their best effort to correct for these phase errors. One of the more prevalent methods used to mimic wind in simulations is by invoking Taylor's frozen flow hypothesis: generating large phase screens of zero-mean, Gaussian white noise in the time domain, scaling them according to the desired statistics (*e.g.* Kolmogorov) in Fourier space , and translating them across the aperture at each time step. One phase screen is generated for each wind layer with dimensions based on exposure time, desired screen resolution, wind speed and direction in that layer, and considerations like making the array square and close to a power of 2 to apply a Fast Fourier Transform (FFT). While this method has proven its utility thus far, the advent of large-aperture telescopes or ones with wide fields of view means that simulations will and do rapidly run up against computation and storage limits, particularly for high-resolution or long-exposure runs. For example, the GPI Adaptive Optics Simulator (GPIAOS) is limited to a 4-second maximum exposure because of memory limits and the LSST simulation team has to generate phase screens that are tens of gigabytes per wind layer for their runs .

There are two major recent thrusts in phase screen generation research. 1) Solutions that preclude the need to generate large phase screens exist or are being developed, such as extending aperture-sized screens by a few columns at each time step . However, they do not address atmospheric "boiling" and changing the atmospheric turbulence model used is not trivial. 2) There are also efforts to increase the realism of simulations by generating and evolving turbulence at multiple scales or evolving atmosphere parameters, like phase coherence length . In this paper we present an autoregressive (AR) method that encompasses aspects of both research direction: it is a computationally efficient method that can be used to simulate frozen flow for long-exposure simulations or for wide-aperture systems; it accommodates a wide variety of turbulence models (Kolmogorov, von Karman, bespoke); it uses one parameter ($\alpha$) to vary stochastic content and another ($P$) to set the turbulence model parameters, both of which can be varied in time, as outlined in Section . The AR method starts with the translating screen generation method described in the previous paragraph , advances phase in the Fourier domain to simulate wind and adds in appropriately scaled noise at each time step to inject random variations to simulate "boiling." It provides a path around computational resource limitations and is simple to implement, especially when compared to the complexity of modal methods . The AR method is compatible with any sample-based phase screen generation method that operates in the Fourier domain. Both major shortcomings of FFT-based methods cited as motivation for current phase screen work , finite exposure times and static atmosphere parameters, are addressed by the AR method.

A method using a Markov process with elements similar to the one we propose is described in . Phase translation to simulate wind occurs in the Fourier domain and scaled, decorrelated phase is added to an instantiation of a phase screen. However, the decorrelation coefficient (analogous to our autoregression coefficient) increases with spatial frequency, something that we find does not correspond to measured telemetry as we outline in Section . In addition, the method is subject to periodicity, a shortcoming common to all Fourier-based methods, which results in phase screen sizes increasing with simulation length. Periodicity is an issue we explicitly address below and in Section . There is no change in phase screen size as simulation time increases using the autoregressive method described in this paper. The authors also do not make it explicitly clear whether they add decorrelated phase at each time step in a simulation, while we outline our algorithm in an easily replicable manner with detailed analysis of its memory and computation requirements.

Two additional criticisms of sample-based Fourier methods are the periodicity of generated screens, which wrap around the domain, and the fidelity of their power spectra to theory, in particular at low and high spatial frequencies. Algorithms to eliminate periodicity by extending the physical size of the screen exist and can be applied to the AR method, but, as we demonstrate in Section , the process of adding stochastic content with the desired power spectrum at each time step effectively results in the renewal of phase after a time interval that scales with the magnitude of the autoregressive scaling coefficient used.

Efforts to address the power spectrum problem have ranged from hybrid, covariance-based, Fourier-modal representations of phase to modifications of the sample-based Fourier methods . The sample-based methods have tended to concentrate on low spatial frequency power because spectral roll-off at the high-frequency end means that the low-frequency power mismatch is the more significant problem . A hybrid method , while producing screens whose power content agrees well with theory, still invokes frozen flow, suffers from periodicity and is order $O(n^2)$ in complexity (where $n$ is the number of points in the screen). Methods that add subharmonics to boost low-frequency power are efficient augmentations to the AR method as long as the power spectrum scaling coefficient (denoted $\beta$ in Section ) is appropriately adjusted to conserve total power. The Markov process screen generation method described above includes an analysis of how to add a computationally reasonable set of subharmonics that will yield a spatial power spectrum which conforms more closely to theory . However, we are in the process of analyzing telemetry gathered from instruments such as GPI and the ShaneAO Adaptive Optics system on the Shane 3-meter telescope at Lick Observatory and will exploit the capacity to modify power in each Fourier mode to match measured data.

Section describes the AR phase screen generation method in detail. Its reduced memory requirements, computation complexity that is at least comparable to (if not less complex) than commonly used existing phase screen generation methods and lack of periodicity are analyzed in Section . We test phase screen PSDs for fidelity to theory and illustrate the impact of adding small amounts of boiling to frozen flow phase on the GPI AO simulator, which incorporates large parts of the as-built system, in Section . Key metrics such as contrast ratio are reduced by factors of 2–20 in the central regions of a target bright star's PSF, *i.e.* in precisely the area the system is expected to find and take spectra of exoplanets. We analyze differences in power content of AR atmospheres and frozen flow screens. Finally, we provide an example of fitting collected telemetry with the AR model in Section . This section serves to provide justifications for our assumptions on model parameters and to tie this work into the broader picture of our ongoing work on wind predictive AO. Wind predictive AO relies on Kalman filters generated from AR model fits to telemetry in real-time. The filter generator will be trained on more realistic phase screens that incorporate multiple wind layers and "boiling" parameters extracted from a large corpus of existing telemetry from GPI and the Shane telescope.

# Description of method

The key differentiator between the AR method and others is the ability to tailor atmospheres and vary power in frozen flow and stochastic components at a modal level. Each Fourier mode has an associated complex autoregressive scaling parameter, $\alpha$ with $|\alpha|\leq 1$, which attenuates phase from the previous timestep. In addition, the phase of $\alpha$ encodes wind velocity and direction for a given layer . Multiplying by an exponential in the Fourier domain translates each Fourier mode resulting in a frozen flow effect which wraps around the domain. Setting $|\alpha|=1$ produces pure frozen flow.

The autoregressive method for a given wind layer starts at time $t=0$ by generating a phase array, $\phi_0$, in the commonly used manner outlined above which is slightly wider than the telescope primary pupil diameter to mitigate edge effects. White noise is scaled in Fourier space by a Kolmogorov power law, $P$, (*n.b.* $P$ can also follow von Karman statistics where outer scale is important or other turbulence models, even, as we foreshadow in Section , fits to telemetry) to give it the appropriate power spectrum as follows:

$$P \propto \frac{2 \pi}{S} N_p r_0^{-5/6} \left(f_x^2 + f_y^2\right)^{-11/12}$$

where:

-   $S =$ screen diameter in meters

-   $N_p =$ number of pixels across screen (which is $N_p\times N_p$ in extent)

-   $r_0 =$ Fried's parameter or phase coherence length in meters at a wavelength of 500 nm

-   $f_x,f_y$ = spatial frequency components (meter$^{-1}$).

For each subsequent iteration, we treat the evolution of phase, $\phi$ (in radians), as an autoregressive process of order 1 in Fourier space, denoted AR(1) – *i.e.* one where the phase at at any given time step is only dependent on the phase from the previous time step and added noise scaled by $P$ . For Fourier-transformed (FT) phase from the previous timestep, $FT(\phi_{t-1}) = \widetilde{\phi}_{t-1}$, each Fourier mode ($k \propto N_p f_x,\ l \propto N_p f_y$) is scaled by $\alpha$. AR coefficient $\alpha$ is a complex array, $N_p \times N_p$ in extent, and a function of spatial frequency ($\alpha = \alpha(f_x, f_y)$) so each frequency can be individually modified. The magnitude of $\alpha$ determines how much past phase is attenuated at every time step. The phase of $\alpha$ (designated $\theta$ to avoid confusion with atmospheric phase) contains wind velocity and direction information. It translates $\phi_{t-1}$ in the Fourier domain and is given by:

$$\theta = -2\pi T \left(f_x v_x + f_y v_y\right)
\label{eqn:alphaphase}$$

where:

-   $v_x, v_y$ are wind velocity components (meters s$^{-1}$)

-   $T$ is the sampling interval (in seconds)

Appropriately scaled, Fourier-transformed, unit-variance, spatially white noise ($\widetilde{\omega}$, also an $N_p\times N_p$ array, $\sigma^2_{\omega}=1$) is added back in at each timestep to simulate the effect of turbulence. The white noise scale factor ($\beta = \sqrt{1-|\alpha^2|} P$) is a function of $\alpha$ and $P$ so that total power is conserved across phase screen time series as $\alpha$ varies – something that we verify in Section . Any method used to boost low spatial frequency power would also require modifying $\beta$. The atmospheric phase in the Fourier domain at any timestep $t$ is:

$$%\phi_t = \alpha \phi_{t-1} + \sqrt{1-|\alpha|^2}\ \omega_t
\widetilde{\phi}_t = \alpha \widetilde{\phi}_{t-1} + \sqrt{1-|\alpha^2|} P \widetilde{\omega}_t
\label{eqn:areqn}$$

# Memory and computational costs

When using a FFT, spatial frequencies are indexed as Fourier modes, ($k, l$). Flowcharts comparing the traditional frozen flow method and this new autoregressive method are shown in Fig. .

For our implementation of this method we chose a computationally efficient DFT method, FFTW3 . Each complex Fourier transform is of order $5 N \log_2(N)$ floating point operations (flops), where $N= N_p^2$. FFTW3 has the advantage of delivering the same performance for values of N that are not powers of 2 as well, which is not the case for conventional implementations of the FFT algorithm (such as that bundled with the IDL programming language). As seen in Fig. , in the normal course of operation, each iteration involves generating an $N_p \times N_p$ array of white, Gaussian noise, followed by a scaling operation and a Fourier transform (FT)/inverse Fourier transform (IFT) pair for each wind layer. Other than the FT/IFT pair, the other operations are element-wise, 2-D array operations of order $O(N)$. Hence, the overall complexity of the AR algorithm is of order $O(N\log N)$. More realistic simulations will have to evolve $r_0$ and wind speeds in time, which means $P$ and $\alpha$ will have to be recalculated periodically in the course of a simulation. While this will increase computation time, the memory requirements and complexity will not change.

The AR method's computational complexity is of the same order as the traditional frozen flow method ($O(N\log N)$), with a significantly smaller N, as shown later in this section. When comparing per iteration, sub-pixel shifts of a large array to simulate wind in any direction for the conventional method can be done in the Fourier domain on a sub-array around the aperture. This entails performing a FT/shift/IFT operation on an array at least as large as the AR phase array, so the AR method performs no worse, computationally speaking, per timestep. Both methods could improve performance by generating Hermitian white noise in the frequency domain , which removes the need for an FFT per timestep. Covariance-based methods can do no better than $O(N^2)$, with $N$ inflated by the requirement of creating a large, translating screen to mimic wind using a frozen flow model.

Phase screen extrusion methods which extend phase screens by a few columns, $N_{col}$, in the direction of wind flow compute a covariance matrix at each timestep which is a matrix multiplication operation involving a matrix of size $N_p \times (N_p \times N_{col})$ and a vector with $N_p \times N_{col}$ elements. This is an $O(N)$ operation (where $N= N_p^2$ as noted above) – more efficient than the AR method.

If a simulation is tracking more than one atmosphere layer, phase screen generation and evolution of each layer is handled separately in the Fourier domain before a linear combination of the real part of their IFTs is used as the new wavefront in a given simulation timestep. Memory requirements for a multiple wind layer simulation are:

$$M = 2b\ (n+1) N
\label{eqn:memeqn}$$

where:

-   $n$ is the number of wind layers (+1 for the cumulative phase that is actually fed into simulation)

-   $N = N_p^2$, number of points and $N_p= aD/p$, the pixel width of the screen. $a$ is a scale factor on the order of unity used to scale up screen size to the nearest size convenient for FFTs, reduce periodicity or increase low-order spatial power. $D$ is entrance pupil diameter in meters and $p$ is the spatial sampling scale or resolution of the phase screen (meters pixel$^{-1}$)

-   $b$ is the number of bytes per phase array element (varies depending on the precision desired – typically 4 or 8) and the factor of 2 arises because phase in the Fourier domain is a complex number

For example, phase screens used with the GPIAO simulator are 48 subapertures across and each subaperture is 8 pixels wide, resulting in $384 \times 384$ pixel screens for each time step. The Gemini telescope's primary mirror is 7.7 meters in diameter with 43 subapertures across it. A complex phase screen Fourier array for a single wind layer (regardless of wind speed, direction and exposure time/simulation length) occupies $8 \times 384^2 = 1.18$ MB for a given time step in the IDL-based GPIAOS. In contrast, a translating frozen flow phase screen with a typical wind velocity measured on Cerro Pachon of $\sim 10$ m/s East-southeast requires a large value of $N$ (Equation ) to account for the wind vector and simulation length. Since the screen has to be square and $N_p$ can only be a limited set of values (*e.g.* powers of 2 form one subset of allowable $N_p$ values), a 1-second simulation requires a $1536 \times 1536$ pixel screen (18.9 MB) per wind layer while a 4-second run demands a $4096 \times 4096$ pixel screen (134 MB) per layer. In addition, the entire screen for a wind layer has to be carried in memory (though memory mapping can solve the problem of RAM capacity with the trade-off of slowing down simulation time with hard drive accesses) and manipulated at each timestep. Hence, the storage requirements for a 1-second simulation run are 16 times smaller per timestep when using an AR phase screen versus a translating phase screen. For a 4-second simulation, the analogous factor is 113.8$\times$ smaller. Memory requirements for the AR method are no different than those for phase screen extrusion methods: the screen size for each timestep is the determined by the desired aperture size and resolution.

For computation comparisons, $N$log$_2(N)$ for the $N = 384^2$, $1536^2$ and $4096^2$ screen size cases is $2.5\times 10^6$, $5\times 10^7$ and $4\times 10^8$ respectively, which indicate the AR method is $20\times$ and $160\times$ more efficient for 1-second and 4-second simulations.

## Periodicity

Sample-based Fourier methods suffer from the problem of periodicity , in that phase screens wrap around the domain. In the case of AR-generated phase screens, the addition of scaled noise at each timestep results in a substantially uncorrelated phase screen after a time interval dependent on $|\alpha|$. For example, after 500 timesteps, with $|\alpha|=0.99$, less than 1% of the phase from the initial timestep is retained ($0.99^{500} = 0.0066$). The corresponding interval for $|\alpha|=0.999$ is 5000 timesteps. A phase screen for a given wind layer with velocity $\sim 8$ m/s would take approximately 1 second to traverse the Gemini aperture, hence using $|\alpha|=0.999$ for that layer would require a screen spanning a spatial extent $5\times$ larger than the aperture at an AO system rate of 1 KHz to minimize periodicity. However, this screen size would not change with wind direction or simulation length, unlike for the traditional frozen flow screen generation methods. It is only dependent on wind speed, AO system rate and the chosen value of $|\alpha|$ for a given layer.

# Analysis and validation

The set of tests we performed to validate our AR atmospheres can be classified as **a)** verification of the shape and distribution of spatial and temporal power spectra to show that the AR phase screen generation method conforms to theory and **b)** comparisons in AO simulator performance between pure frozen flow atmospheres and AR atmospheres to quantify how changes in assumptions about the atmosphere model affect system performance metrics. In the case of GPI, a planet imager that seeks to occult a target star and enhance contrast ratio in the region immediately surrounding the star, one key metric is residual scattered light at low radii centered on the parent star, a region where there is a high likelihood of discovering planets. The LSST project has also observed material changes in PSF recovery from the addition of "boiling" via the AR method to their simulations which is causing them to revisit assumptions about their atmosphere model . Characterizing the changes in system response to injected stochastic content is a necessary step in the process of deciding whether replacing the existing model with another is worth the effort.

In both tests, atmosphere datacubes (time series of 2-D phase screens) were sized slightly larger than the Gemini South primary pupil ($N_p=384$ with a pixel scale of 0.0224 meters pixel$^{-1}$) and with $|\alpha|=0.99$ or $|\alpha|=0.999$. For the power spectrum checks, we generated higher-resolution (in time) AR atmosphere datacubes with 8192 timesteps (representing a 5.46-second exposure at the GPI frame rate of 1.5 KHz). Comparative simulations were fed atmosphere realizations for a 1 second exposure when analyzing point spread function (PSF) differences between AR and frozen flow screens. Longer 22-second exposures at 1 KHz were used when comparing to GPI telemetry, which was collected in blocks with the same parameters. Three wind layers with differing velocities and directions derived from conditions observed at Cerro Pachon were tracked .

## Spatial and temporal power spectral densities

A Blackman window was used to generate an unbiased, low-leakage periodogram with interval length set to 1024 samples for AR atmosphere phase screen time series of varying lengths in time and scaling parameter, $\alpha$. Spatial PSDs were calculated at each radial spatial wavenumber within the GPI aperture by exploiting Parseval's theorem and compared to theory . Temporal PSDs were plotted over a frequency range \[-512,512\] Hz from the generated periodograms .

Spatial PSDs for $|\alpha|={0.99,0.999}$ are shown in Fig . These atmosphere realizations used Kolmogorov statistics and demonstrate the expected $\kappa^{-11/3}$ slope for spatial frequency, $\kappa$ (the theoretical power spectrum is overplotted for comparison) . The PSD for $|\alpha|=0.999$ shows a greater variance around the theoretical slope because of the higher correlation between phase from one timestep to the next. The greater stochastic content in the $|\alpha|=0.99$ phase screens makes it hew closer to the power spectrum for infinite realizations of purely random screens.

Correctly scaling $\beta$ ensures overall power is conserved between datacubes with varying $\alpha$ – Fig. shows phase screen series with lower magnitudes of alpha have lower peaks and wider temporal power spectrum profiles, but the integral under the curve is constant as $|\alpha|$ varies.

## Simulation tests

AR and frozen flow atmosphere realizations were fed into the GPI AO simulator and the spatial and temporal power content of the residual phase in each case was compared to test the sensitivity of modeled instrument behavior to atmosphere models with greater turbulence added.

The GPI AO simulator is a detailed end-to-end system simulation with Fourier optics components. It uses a standard approach of translating large phase screens to generate a frozen flow atmosphere. The output of the AO system feeds an apodized-pupil Lyot coronagraph. The coronagraph suppresses diffraction and the spatial filter creates a dark hole where the residual scattered light is a function of both propagated wavefront sensor noise and residual atmosphere. Until the AR method was used for atmosphere generation, simulations were limited to 4-second exposure times, a constraint that has now been lifted. In addition, it is now possible to simulate aspects of GPI's calibration system, which compensates for slowly varying non-common path errors (like temperature, flexure and changing gravity vector) and operates on timescales of tens of seconds to minutes.

We observed that the AR-generated atmospheres resulted in PSF images with more scattered light (higher intensity) at lower radii when compared to pure frozen flow ($2\times$ for $|\alpha|=0.999$ and $\sim 20\times$ for $|\alpha|=0.99$). Fig. shows a PSF comparison of the default controller for each atmosphere type: pure frozen flow, AR with $|\alpha|=0.999$, and AR with $|\alpha|=0.99$. The plot is a radial average of the normalized intensity of the instantaneous science PSF recorded half way through a 1-second simulation run. Intensity is normalized such that the peak of the ideal, unblocked PSF is set equal to 1. Scattered light at a given location in the PSF plot corresponds to residual power for a specific Fourier mode . Hence, the excess power seen close to the center of the PSF (at low radii) translates into more power at low order Fourier modes, $[k,l]$. Below a radius of 10 pixels, power is suppressed by the apodized Lyot coronagraph. However, in the 10-100 pixel radius region (GPI's "dark hole" ), significantly different PSFs were recorded for small variations in atmospheric power distributions, which impacts the ability of the instrument to detect planets that lie close to the target star. To verify that this was a feature inherent in AR atmospheres that could be exploited when comparing to telemetry or a vestige of artefacts introduced by GPIAOS, we compared the temporal PSDs of the input atmosphere models.

Analysis of the temporal power spectra of AR atmospheres versus pure frozen flow atmospheres reveals the source of the elevated low-order intensity in Fig. . Fig. plots the open- and closed-loop temporal PSDs, *i.e.* the power content of the AR atmosphere datacubes (open-loop) and that of the residuals as seen by the system's wavefront sensor using the default controller (closed-loop). The two types of atmospheres have similar low-frequency power but the broad wings beyond 5 Hz in the open-loop AR PSD are not suppressed resulting in broad wings in the closed-loop PSD and, hence, more scattered light close to the center of the PSF in Fig. . The residuals for an AR atmosphere with $|\alpha|=0.99$ in the right-hand plot in Fig. show elevated power levels (the area under the curve is $\sim 20\times$ greater) over the frozen flow model atmosphere consistent with the differences seen in Fig. .

The temporal structure of the atmosphere model evidently has a large impact on system performance. The magnitude of the impact can be estimated using the error transfer function (ETF) of a simulator because a change in input PSD changes the error output of a closed loop system . The ETF for GPIAOS , which applies corrections two frames after measurement, is given by:

$$ET\hspace{-0.5mm} F = \left| \frac{1}{1 + z^{-2} C(z)} \right|^2
\label{eqn:etf}$$

where:

-   $C(z) = g/(1 - c z^{-1})$, $0 < g < 1$ is the default modal gain and $c\lesssim 1$ is integrator leak gain.

The temporal PSD of the measurements of the AO system in closed loop is estimated by:

$$P_{CL} = ET\hspace{-0.5mm} F(P_{\phi} + P_N)
\label{eqn:psd_cl}$$

where:

-   $P_{\phi}$ is the temporal PSD of the phase (measured when the system applies no correction).

-   $P_N$ is the temporal PSD of the WFS noise (measured when the system is run closed-loop with no phase aberrations passed through it *i.e.* with no phase screen).

We can neglect the impact of the WFS response on the atmosphere as it is significant only at high temporal frequencies. Further details on this type of modeling are outlined in Ch. 6 of .

More than an exercise in curiosity, in at least the GPI case, greater scattered light near the core of the PSF has the very real effect of decreasing contrast and reducing sensitivity to and detection of exoplanets. Hence, simulating the operation of the system with more realistic atmosphere models that have the same statistics as recorded telemetry will provide more credible limits to the system's capabilities. Using such models will allow tests of control schemes that improve contrast in conditions that more closely resemble the real-world observing environment than pure frozen flow atmosphere models. In the next section we describe our ongoing effort to exploit the flexibility and customizability of the AR method to build a better atmosphere model by extracting meaningful parameters like wind speed and direction, wind layers with the most power content, and statistics like $r_0$ and $L_0$.

# Motivation

The magnitude of the effect of introduced "boiling" on PSF quality, plotted in Fig. , illustrates the need to better understand atmospheric characteristics, find ways to measure them and correct for them. In the previous sections we have outlined an AR model with a number of degrees of freedom that account for and temporally evolve aspects of atmospheric turbulence such as statistics ($r_0$, $L_0$), "boiling", and wind layers with differing wind velocities and directions while still remaining computationally tractable and suitable for large-scale simulations. Tuning those degrees of freedom to improve upon prior models and better reflect reality by comparing to gathered telemetry is the next phase.

We are in the process of analyzing telemetry gathered by AO systems on GPI and the Shane telescope at Lick Observatory , by performing AR model fits to temporal PSDs and extracting parameters from spatial PSDs. This effort, in turn, feeds into implementing wind predictive AO on a test bed and on-sky to eliminate temporal errors caused by wind-blown phase. The wind predictive method employs Kalman filters and Linear Quadratic Gaussian (LQG) control to notch out power in wind peaks found. The AR method will be used to generate synthetic atmospheres on which the Kalman filter generator will be trained. We wish to explore the sensitivity of Kalman filters to power in the random element of the atmosphere and crafting a realistic atmosphere model is the first step in the process.

In simulations, for each realization of the atmosphere the GPI AO simulator was run with the default unoptimized controller, an optimized gain filter and a Kalman filter . In all simulation comparisons, the Kalman filter had the greatest residual suppression (especially at higher orders), followed by the optimized filter. The Kalman filter achieved the least improvement for $|\alpha|=0.99$, which is expected behavior when a filter optimized for flow is applied to an atmosphere with higher power in boiling, but was still comparable to or better than an optimized filter.

One example of peak fitter and layer finder output is shown in Fig. . Closed loop residual phase as measured by GPI's wavefront sensor over 22.1 seconds with the system pointed at a bright star and an AO system rate of 1 kHz was stored at the date and time indicated on the plot. Using the system's error and noise transfer functions, pseudo-open loop phase is estimated, which is then converted to Fourier modes and temporal PSDs calculated. A PSD peak fitter and layer finder algorithm identifies and fits wind peaks and then finds wind layers. The width of each peak sets the magnitude of $\alpha$ for that peak; wider peaks have lower values of $|\alpha|$, as seen in Fig. . The temporal frequency of the peak is used to set the phase of $\alpha$ per equation . Fig. plots the temporal PSD of a Fourier mode ($k=14, l=36$) displaying AR model fits to power peaks at 0 Hz (DC, $|\alpha|=0.995$) and 10.2 Hz ($|\alpha|=0.993$). Values of $|\alpha|$ for the other peaks fit range from 0.991 to 0.996, a characteristic shared by other telemetry sets analyzed which motivates our restriction of $|\alpha|$ values analyzed in AR modeling to the interval $[0.990, 0.999]$. Two wind layers found have wind velocities of 1.2 m/s and 4.9 m/s (with no altitude information).

Only wind peaks which contain significant power (defined to be greater than 20% of the peak with the highest power content, the D.C. peak in this case) are identified and used for wind prediction, *i.e.* ones that are worthwhile for the system to correct. Hence, peaks at -2, -5 and beyond +20 Hz are ignored.

In contrast to the method outlined in , we have observed no frequency dependence of the decorrelation parameter $\alpha$. Our fits to telemetry indicate that from one time step to the next its magnitude is confined to the interval $[0.990, 0.999]$ which is significantly different from the implied range of $[1/\sqrt{3},1/\sqrt{2}]$ in the prior work.

The process of analyzing telemetry is work in progress, but there is already evidence to suggest that the AR model is a better representation of the power distribution of the atmosphere (between frozen flow and "boiling" components) than pure Kolmogorov frozen flow. Long simulations also need information about the temporal evolution of atmospheric characteristics like power in wind layers, $r_0$ and $L_0$ (both of which are extracted from the spatial PSD of atmospheric telemetry). Telemetry collection through individual nights in varied conditions from different sites and its analysis will better inform how to set and evolve atmosphere model parameters in long exposure simulations.

# Conclusions and future work

We have implemented, validated, tested and analyzed an autoregressive method for the generation of phase screens with turbulence. Where comparisons can be made, the AR atmosphere technique is 10-160 times more efficient in its use of computational resources and memory than sample-based phase translation methods. By raising the limit on simulation length, it significantly extends the capabilities of existing simulators and allows the testing of longer time-scale corrections. The method's flexibility allows for the easy incorporation of any desired atmospheric turbulence model and and variation of parameters (like coherence length and outer scale). Finally, its simplicity when compared to other worthy attempts at optimization makes it ready for immediate use and adaptation in simulation (source code in the form of IDL and python routines is available on `github`).

Spatial power spectra are as expected (and can be tweaked based on the turbulence model used – we chose Kolmogorov for this exercise). Temporal power spectra show AR atmospheres have very different power distributions compared to frozen flow realizations which results in significant differences in PSF profiles in simulations. Work is ongoing to relate AR atmospheres to real-world telemetry gathered by multiple AO systems so that the statistics of generated atmospheres (*e.g.* power content, turbulence strength and evolution of parameters like $r_0$) are as close as possible as those extracted from observations. Tailored atmosphere models that resemble real conditions will then be used to train Kalman filter generators for optimizing Fourier Wind Identification schemes before they are tested on-sky.

# Acknowledgment

The work of SS and ARR is funded by the UC Lab Fees Research Program grant 12-LF-236852.

This work is performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. The document number is LLNL-JRNL-667773.

SS wishes to thank Erik Johansson at the NSO for his insight during highly productive discussions about the AR method and Jim Chiang at SLAC for the considerable assistance he provided in creating the python version of the AR screen generation code.

We thank the anonymous referees for their constructive and insightful criticism and commentary.

The Gemini Observatory is operated by the Association of Universities for Research in Astronomy, Inc., under a cooperative agreement with the NSF on behalf of the Gemini partnership: the National Science Foundation (United States), the National Research Council (Canada), CONICYT (Chile), the Australian Research Council (Australia), Ministério da Ciência, Tecnologia e Inovação (Brazil) and Ministerio de Ciencia, Tecnología y Innovación Productiva (Argentina)


abstract: ; How romantic partners interact with each other during a conflict influences how they feel at the end of the interaction and is predictive of whether the partners stay together in the long term. Hence understanding the emotions of each partner is important. Yet current approaches that are used include self-reports which are burdensome and hence limit the frequency of this data collection. Automatic emotion prediction could address this challenge. Insights from psychology research indicate that partners' behaviors influence each other's emotions in conflict interaction and hence, the behavior of both partners could be considered to better predict each partner's emotion. However, it is yet to be investigated how doing so compares to only using each partner's own behavior in terms of emotion prediction performance. In this work, we used BERT to extract linguistic features (i.e., what partners said) and openSMILE to extract paralinguistic features (i.e., how they said it) from a data set of 368 German-speaking Swiss couples (N = 736 individuals) which were videotaped during an 8-minutes conflict interaction in the laboratory. Based on those features, we trained machine learning models to predict if partners feel positive or negative after the conflict interaction. Our results show that including the behavior of the other partner improves the prediction performance. Furthermore, for men, considering how their female partners spoke is most important and for women considering what their male partner said is most important in getting better prediction performance. This work is a step towards automatically recognizing each partners' emotion based on the behavior of both, which would enable a better understanding of couples in research, therapy, and the real world.

bibliography: refs.bib
title: "You made me feel this way": Investigating Partners' Influence in Predicting Emotions in Couples' Conflict Interactions using Speech Data

# Introduction

Understanding the emotions partners feel during and after conflict interactions is important because of its long-term effects on couples' relationship quality and stability . Happy couples, for example, experience more positive and less negative emotions during conflict interactions compared to unhappy couples . The current study focuses on one fundamental aspect of the conflict mechanism — how the emotional experience *within* each partner is influenced by the behavioral exchange *between* partners.

A crucial aspect of conflict interaction in couples is how the behavioral exchange makes each person feel during and after the interaction . But although both partners experience the same interaction, they can feel very differently about it. For example, if we assume that partner A shows contempt and criticizes partner B, we can assume that partner A might feel angry or superior whereas partner B might feel hurt or humiliated. Thus, the experience can be very different for the partner who communicates something compared to the partner who perceives it . This differentiation allows us to reflect on another mechanism, namely that the emotions a person experiences are the results of two kinds of influences. Obviously, a person's emotional experience is constantly influenced by a partner's behavior as a kind of co-regulating force, talk turn by talk turn. In addition, however, each person has the ability to regulate one's own emotional response (e.g., cognitive appraisal, emotion regulation) , which then affects one's own subsequent behavioral response. Thus, what partners experience emotionally during and at the end of a conflict interaction is a reflection of the co-regulation and self-regulation processes .

To better understand emotions in couples and their impact on relationships, often self-report assessments are used in which each partner is asked to provide a rating of their own emotions right after an interaction, or partners are asked to watch a video recording of the interaction and provide continuous ratings using a joystick, for example, . Self-reports are burdensome to complete and may not be collected frequently. This means that the relationship between behavior and emotions cannot be studied often. Thus, an automatic emotion recognition system would allow scale couples research.

Various works have used linguistic features (i.e., *what* has been said) and paralinguistic features (i.e., *how* it was said) to predict the emotions of each partner in couples interactions more broadly and in conflict interactions in particular . Most of these works have used observer ratings (perceived emotions) as labels rather than self-reports (one's actual emotions). Hence, the prediction task becomes that of recognizing external individuals' perception of each partner's emotion rather than each partner's emotion per their own assessment. Though similar, the latter is more challenging than the former for a number of reasons. First, the rating might be biased may not reflect their actual emotions over the period the rating is for (e.g. the past 5 minutes). Whereas for observer ratings, coders are generally trained over several weeks, it is done by more than one person and various approaches are employed to resolve ratings that are not in agreement and ensure the validity of the labels. Second, the self-reported emotion may not be reflected in that partner's behavior in comparison to observer ratings which are purely based on behavioral observation.

Despite these challenges, insights from psychology research could be leveraged to make the prediction task easier. Specifically, given that partners' behaviors influence each other's emotions in conflict interaction, the behavior of both partners could be considered to better predict each partner's end-of-conversation emotion. However, it is yet to be investigated how doing so compares to using each partner's own behavior only in terms of emotion prediction performance. In this work, we used a dataset collected from 368 couples who were recorded during an 8-minute conflict interaction, extracted linguistic and paralinguistic features, and used machine learning approaches to predict how each partner felt directly after the conflict interaction (self-reported emotion). We answer the following research questions (RQs)

**RQ1:** *How well can the end-of-conversation emotion of each partner be predicted by their own behavior — a combination of linguistic and paralinguistic data? (self-regulation)*

**RQ1:** *How does the prediction performance change when including the other partner's behavior — (a) linguistic only, (b) paralinguistic only, and (c) combination of linguistic and paralinguistic data? (co-regulation)*

Our primary contribution is an evaluation of how well a partner's own linguistic and paralinguistic features predict one's own end-of-conversation emotion. Our second contribution is an investigation of how the prediction performance changes when including one's partner's features (linguistic, paralinguistic, and both). Our third contribution is the use of a unique dataset — spontaneous, real-life, speech data collected from German-speaking, Swiss couples (n=368 couples, N=736 participants), which is the largest ever such dataset used in the literature for automatic recognition of partners' end-of-conversation emotion. The insights from our work would advance the use of methods to automatically recognize the emotions of each partner which could enable research and applications to better understand couples' relationships in therapy and the real world.

The rest of the paper is organized as follows: In Section 2, we describe our data collection, preprocessing and feature extraction, in Section 3, we describe our experiments and evaluation, in Section 4 we present and discuss our results, in Section 5, we present limitations and future work, and we conclude in Section 6.

# Methodology

We extracted linguistic features using BERT, and paralinguistic features using openSMILE and performed multimodal and dyadic fusion.

## Data Collection and Preprocessing

This work used data from a larger dyadic interaction laboratory project conducted at the premises of a university with 368 heterosexual German-speaking Swiss couples (N=736 participants; age 19-82) . The inclusion criterion was to have been in the current relationship for at least 1 year. Couples had to choose one problematic topic for the conflict interaction from a list of common problems, and participants were then videotaped as they discussed the selected issue for 8 minutes. The data used in this work had one interaction from each couple and consequently, 368 8-minute interactions.

After each conversation, each partner provided a self-report of their emotions on four bipolar dimensions — namely "good mood versus bad mood," "relaxed versus angry," "happy versus sad" and "calm versus stressed" — with the scale: 1 — very much, 2 — much, 3 — a little, 4 — a little, 5 — much, 6 — very much. In this work, we sort to focus on predicting emotional valence (positive or negative) based on Russell's circumplex model of emotions . Hence, we used an average of the "good mood versus bad mood" and "happy versus sad" scales. We did not use the other two scales because their polarity also could represent the arousal dimension of emotion (low vs high arousal). We then binarized the averaged values such that values greater than or equal to 3.5 were negative (0) and the rest were positive (1).

The speech data were manually annotated with the start and end of each speaker's turn, along with pauses and noise. This was necessary in order to later be able to extract linguistic and paralinguistic features for each partner separately. In addition, speech content of both partners was manually transcribed for each partner separately and stored in 15-second chunks. Given that Swiss German is mostly spoken with different dialects across Switzerland, the spoken words were written as the corresponding German word equivalent.

Some couples requested their data to be removed and some data were missing due to technical problems in data collection. Of the original 368 couples that took part in the study, we could use 338 samples for females (46 negative labels) and 341 samples for males (32 negative labels). The distribution highlights a significant class imbalance that is characteristic of real-world datasets and partners' behavior as seen in other similar works (e.g. ).

## Linguistic Features

We extracted linguistic features from the transcripts of the whole 8-minute interaction using a pre-trained model — Sentence-BERT (SBERT) . Sentence-BERT is a modification of the BERT architecture with siamese and triplet networks to compute sentence embeddings such that semantically similar sentences are close in vector space. Sentence-BERT has been shown to outperform the mean and CLS token outputs of regular BERT models for semantic similarity and sentiment classification tasks. Given that the text is in German, we used the German BERT model as SBERT's Transformer model and the mean pooling setting. The German BERT model was pre-trained using the German Wikipedia dump, the OpenLegalData dump, and German news articles. The extraction resulted in a 768-dimensional feature vector.

## Paralinguistic Features

We extracted acoustic features from the voice recordings. First, we used the speaker annotations to get the acoustic signal for each gender separately. Next, we used openSMILE to extract the 88 eGeMAPS acoustic features which have been shown to be a minimalist set of features for affective recognition tasks . The features are extracted in 25 ms sequences and then various functions (e.g. mean, median, range, etc) are computed over the sequences resulting in 88 features for the whole 8-minute audio. The original audio was encoded with 2 channels. As a result, we extracted the features for each channel resulting in a 176-dimensional feature vector.

## Multimodal and Dyadic Feature Fusion

Given that emotions are reflected in what and how people say things, we performed multimodal fusion (early fusion) by concatenating the linguistic and paralinguistic features resulting in a 944-dimensional feature. We did this separately for each partner. This feature vector was used as the baseline approach to answer research question (1).

Additionally, we fused features from both partners to answer research question (2). Specifically, for partner A, we concatenated their multimodal feature vector with the features of partner B and used it to predict partner A's emotion label. This process was done for partner B as well. In order to investigate which behavioral data of the interacting partner was most important in the prediction of the emotions, we included the features in the following order (1) linguistic only, (2) paralinguistic only, and (3) multimodal fusion of both.

Consequently, we had four feature sets: (1) Multimodal fusion (baseline — own features), (2) Multimodal + Dyadic Fusion (with partner's linguistic only), (2) Multimodal + Dyadic Fusion (with partner's paralinguistic features only) (4) Multimodal + Dyadic Fusion (with partner's combined linguistic and paralinguistic only). These were passed to machine learning models to answer the two research questions.

# Experiments and Evaluation

We run experiments using scikit-learn the following machine learning models: support vector machine (SVM) algorithm with linear and radial basis function kernel, and random forests. We trained models to perform binary classification of each partner's self-reported positive and negative emotion using different feature sets. We used the four feature sets described in the previous section. To train and evaluate the models, we used nested K-fold cross-validation. The nested procedure consisted of utilising an "inner" run of 5-fold cross-validation for hyperparameter tuning, followed by an "outer" run of 10-fold cross-validation which utilizes the best values for each hyperparameter found by the "inner" run. We prevented data from the same couple from being in both the train and test folds, thereby evaluating the model's performance on data from unseen couples. As the data was imbalanced, we used the metric balanced accuracy which is the unweighted average of the recall of each class and confusion matrices for evaluation. We used the "balanced" hyperparameter for all the models to mitigate the class imbalance while training. We compare to a random baseline of 50% balanced accuracy.

```latex
\begin{tabular}{|l|c|c|}
\hline
\textbf{Approach} &
  \multicolumn{2}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Balanced \\ Accuracy (\%)\end{tabular}}} \\ \hline
                             & \textbf{Male} & \textbf{Female} \\ \hline
Multimodal fusion (baseline) & 49.8          & 59              \\ \hline
\begin{tabular}[c]{@{}l@{}}Multimodal + Dyadic Fusion (with partner's \\ combined linguistic and paralinguistic only)\end{tabular} &
  52.3 &
  63.2 \\ \hline
\begin{tabular}[c]{@{}l@{}}Multimodal + Dyadic Fusion (with partner's \\ linguistic features only)\end{tabular} &
  53.5 &
  \textbf{64.8} \\ \hline
\begin{tabular}[c]{@{}l@{}}Multimodal + Dyadic Fusion (with partner's  \\ paralinguistic only)\end{tabular} &
  \textbf{56.1} &
  59.9 \\ \hline
\end{tabular}
```

# Results and Discussion

Our results are shown in Table . The baseline approach with multimodal fusion was not better than chance in predicting men's emotions at the end of the conflict interaction (49.8%). This is unexpected as this indicates that men's own behaviors during the interaction are not related to how they feel at the end of the interaction. This might be due to self-regulation processes or not showing much emotions during the interaction. This is clearly different from women as their emotions can be predicted by their own behavior (59%). Thus, it seems that women seem to express their emotions more clearly in their behavior. These results of poorer prediction performance for men compared to women is consistent with the results of

Including features from the interacting partner improved the results for both men (52.3%) and women (63.2%). These results are consistent with psychology research the behavior of partners' have an effect on each other's emotions in conflict interaction . This is a crucial finding because (i) previous research shows that the behavior of one person influences the behavior of the other person (ii) as well as that the emotional changes of one person affect the emotions of the other . However, this is the first study showing that behavioral features assessed during the conflict interaction can be used to predict one partner's emotion at the end of the conversation. In addition, the improvement in women's emotion prediction at the end of interaction is greater when including their partner's linguistic data (64.8%) whereas there is hardly any difference when including partner's paralinguistic features. This is a surprising finding as women generally pay more attention to paralinguistic cues . Notably, the results are different for men. The prediction for men's emotions slightly increases when including their partner's linguistic features but the prediction improves substantially when including women's paralinguistic features. Although we do not know which specific paralinguistic features are the main drivers for predicting the emotions, this finding is in line with prior findings — when women "nag", men experience strong negative physiologic reactions and tend to withdraw . Future research is needed to investigate which aspects of one partner's behavior exactly cause their own emotion prediction performance to decline. In addition, these results have implications for the kind of behavioral information to consider to best predict each partner's end-of-conversation emotions.

We show the confusion matrices for the best models for the men and women in Figures and respectively. They reveal the models' challenges at recognizing positive emotions, resulting in them being misclassified as negative emotions.

## Limitations and Future Work

Further work is needed to investigate if these results are the same for couples in a different cultural context and also explore the effect of the interacting partners' behavior at a more granular level such as on a talk-turn basis. More fine-grained emotion ratings are needed to investigate that.

We used BERT as a feature extractor in this work. Generating domain-specific sentence embeddings via fine-tuning the BERT model and exploring deep transfer learning models for the paralinguistic features may improve the results. BERT models have been shown to encode gender and racial bias because of the models they are trained on. Further investigations are needed in the future of potential biases in prediction. .

Finally, we used manual annotations and transcripts. To accomplish true automatic emotion prediction, speaker annotations need to be done automatically and our approach needs to use and work for automated transcriptions. Current speech recognition systems do not work for this unique dataset given that couples speak Swiss German, which is (1) a spoken dialect and not written, and (2) varies across different parts of the German-speaking regions of Switzerland. Further work is needed to develop automatic speech recognition systems for Swiss German.

# Conclusion

In this work, we investigated one's own and the partner's behavior in predicting end-of-conversation emotions in the context of conflict interactions in German-speaking Swiss couples. We extracted linguistic features using BERT and paralinguistic features using openSMILE. We fused both features in a multimodal approach for each partner. We also fused the features of both partners to predict the emotions of each partner. Our results show that including the behavior of the other partner improves the prediction performance. Furthermore, for men, considering *how* their female partners spoke is most important, and for women considering *what* their male partners said is most important in getting better prediction performance. These insights have implications for the behavioral information to (not) include to better predict each partner's end-of-conversation emotions which will enable a better understanding of couples relations in research, therapy, and the real world.


address:  $^{1}$ Ruhr Astroparticle and Plasma Physics Center (RAPP Center), Insitut für theoretische Physik IV, Ruhr-Universität Bochum, D-44780 Bochum, Germany
  $^{2}$ Centre for Space Science, North-West University, Potchefstroom, 2520 South Africa
  $^{3}$ Croatian MAGIC Consortium, Rudjer Boskovic Institute, University of Rijeka, University of Split and University of Zagreb, 10000, Zagreb, Croatia; email@example.com (D.D.P.); firstname.lastname@example.com (T.T.)
  $^{4}$ Landessternwarte, Universität Heidelberg, Königstuhl, D-69117 Heidelberg, Germany; firstname.lastname@example.com (F.J.); email@example.com (M.M.); firstname.lastname@example.com (S.W.)
  $^{5}$ Finnish MAGIC Consortium, Tuorla Observatory, University of Turku and Astronomy Division, University of Oulu, 90014 Oulun, Finland; email@example.com
  $^{6}$ W. W. Hansen Experimental Physics Laboratory, Kavli Institute for Particle Astrophysics and Cosmology, Department of Physics and SLAC National Accelerator Laboratory, Stanford University, ; email@example.com
  $^{7}$ DESY, D-15738 Zeuthen, Germany; firstname.lastname@example.com
  $^{8}$ Laboratoire d'Annecy-le-Vieux de Physique des Particules, Universite Savoie Mont-Blanc, CNRS/IN2P3, 74941 Annecy-le-Vieux, France; email@example.com
  $^{9}$ Department of Astrophysics, University of Lodz, 90236 Lodz, Poland; email@example.com
  $^{10}$`<!-- -->`{=html}Institute of Nuclear Physics, Polish Academy of Sciences, PL-31342 Krakow, Poland; email@example.com

# Introduction

The correlations between blazar emissions in different energy bands are best probed with long-term monitoring, providing unbiased sampling. Especially for ground-based observatories this is hard to achieve for even a small number of sources. The *Fermi* satellite has transformed the monitoring of blazars in the high-energy (HE) $\gamma$-ray band ($E>100\,$MeV) through its continuous surveillance of the whole sky every three hours (although somewhat less uniform after its hardware failure in March 2018) as detailed in . In the optical and radio bands many monitoring programs are run thanks to the large number of available telescopes. However, in other energy bands the monitoring capabilities are limited. In the X-ray band the Neil Gehrels *Swift* observatory runs a limited monitoring effort and can follow up on flares. MAXI on board the International Space Station provides all-sky capabilities within 1 orbit with limited sensitivity. In the very-high-energy (VHE) $\gamma$-ray band ($E>100\,$GeV) the monitoring effort is limited by sensitivity, e.g., for FACT and HAWC , or by time constraints due to competition with other objects. The latter strongly influences the monitoring efforts of the three large imaging atmospheric Cherenkov telescope (IACT) facilities H.E.S.S., MAGIC and VERITAS. Nonetheless, they have been running limited monitoring projects on a number of blazars.

Here, we report on the ongoing monitoring efforts by H.E.S.S. and MAGIC of the flat spectrum radio quasar (FSRQ) PKS 1510-089. It is located at a redshift $z_{\rm red}=0.361$ and possesses a bright broad-line region (BLR), e.g., . Hence, VHE photons produced within the boundaries of the BLR should be absorbed. As the emission region of $\gamma$-rays was thought to be close to the central black hole, VHE emission from FSRQs was considered unlikely by many. However, several detections of FSRQs  challenge this picture and suggest that jets are able to produce $\gamma$-rays also further downstream in the jet.

To verify that these are not simply one-time-only flaring events, but that FSRQs produce VHE emission on all time scales, monitoring programs have been initiated with H.E.S.S. and MAGIC on PKS 1510-089. While these are not unbiased, they have already provided important information. During a strong multiwavelength flaring event in 2015, variability on night-by-night scales at VHE $\gamma$-rays was observed for the first time from this source . Furthermore, MAGIC observations integrated during low-states in the HE band revealed a significant VHE signal with an average, integrated flux $\bar{F}(E>150\,\mbox{GeV})=(4.3\pm0.6)\times 10^{-12}\,$cm$^{-2}$s$^{-1}$ . Hence, PKS 1510-089 is not only variable in VHE $\gamma$-rays but also a persistent source. This has a direct and very important consequence: the absorption of VHE photons through the BLR cannot be too severe, and the emission region must be at the edge or even outside of the BLR at all times. This, in turn, implies that the usual model for FSRQ $\gamma$-ray emission, namely inverse-Compton scattering of BLR photons, might not be correct.

This paper gives the status of the H.E.S.S. monitoring efforts on PKS 1510-089, and its early results. One of the important outcomes is the detection of an unprecedented VHE flare in 2016, which was also followed-up with MAGIC. Details of this flare are reported here. Additional multiwavelength data are gathered for comparison from *Fermi*-LAT in the HE $\gamma$-ray band, from *Swift*-XRT in the X-ray band and from ATOM in the R-band.

# Monitoring with H.E.S.S.

After the detection in 2009 , H.E.S.S. has continued observing PKS 1510-089 with low cadence. Since 2015 this effort has been significantly increased with several hours of observations each month during the visibility period (which typically lasts from February to July each year) resulting in observations almost every night without moon interference. The resulting nightwise lightcurve including all observations is shown in Figure , and a focus on the 2015 and 2016 season is shown in Figure . Note that nightwise bins do not guarantee a significant flux per night due to the limited sensitivity and the dimness of the source in the low state. In fact, about 50% of the nights shown in Figure  are compatible with zero. The bright VHE flare in 2016 clearly stands out with peak fluxes up to 10 times higher than the previous record holder in 2015. In order to reveal details of the other times, the inset shows the zoom in on the fluxes without the 2016 flare. The flare is further discussed in Section . The average, integrated flux for the whole time frame is $\bar{F}(E>150\,\mbox{GeV})=(5.1\pm0.3)\times 10^{-12}\,$cm$^{-2}$s$^{-1}$, which is compatible within errors with the MAGIC low-flux level, but includes the bright states, as well. The average of the 2015–2016 time frame is compatible with the average of the whole time frame. In both cases, a constant flux is ruled out with very high significance. This is underlined by the fractional variability

$$\begin{aligned}
 F_{\rm var} = \frac{\sqrt{S^2-\sigma_{\rm err}^2}}{\bar{F}} \label{eq:Fvar},
\end{aligned}$$ where $S^2$ is the variance, $\sigma_{\rm err}^2$ is the mean square error, and $\bar{F}$ is the average flux of the source in the considered data set. For the whole data set $F_{\rm var}^{\rm VHE} = 3.2\pm 0.1$, and for the 2015–2016 time frame $F_{\rm var}^{\rm VHE} = 3.3\pm 0.1$. These large values are driven by the 2016 flare. Removing the two nights of that event give $F_{\rm var}^{\rm VHE} = 0.8\pm 0.2$, which still implies significant variability. Defining the variability time scale between two subsequent flux points as $$\begin{aligned}
 t_{\rm var} = \bar{F} \frac{t_{i+1}-t_{i}}{|F_{i+1}-F_{i}|} \label{eq:tvar}
\end{aligned}$$ the minimum variability time scale is $t_{\rm var}^{\rm VHE} = (0.8\pm0.06)\,$h, which was exhibited during the major flare in 2016. The error on the variability time scale has been derived through error propagation.

The bottom panel in Figure  shows the nightwise HE $\gamma$-ray fluxes obtained with *Fermi*-LAT. The average, integrated flux in this band is $\bar{F}(E>100\,\mbox{MeV})=(3.39\pm0.03)\times 10^{-7}\,$cm$^{-2}$s$^{-1}$ for the whole time frame, and $\bar{F}(E>100\,\mbox{MeV})=(2.58\pm0.05)\times 10^{-7}\,$cm$^{-2}$s$^{-1}$ for 2015-2016. The source has been very active in this band for large parts of the considered time frame with $F_{\rm var}^{\rm HE}~=~1.170\pm 0.005$ for the whole time frame, and $F_{\rm var}^{\rm HE} = 1.11\pm 0.01$ for 2015–2016. The minimum variability time scale is $t_{\rm var}^{\rm HE} = 0.69\pm0.06\,$h for the whole time frame exhibited during a flare in 2011, and $t_{\rm var}^{\rm HE} = 1.3\pm0.2\,$h for the 2015–2016 time frame exhibited in August 2015.

Unfortunately, many of the HE flares were not followed up with with H.E.S.S. due to observational constraints. Nonetheless, it is interesting to investigate whether there is any correlation between these two bands. Plotting the simultaneously recorded fluxes of the two bands against each other can reveal direct correlations. The scatterplot is shown for H.E.S.S. and *Fermi*-LAT fluxes in Figure a. The discrete cross-correlation function (DCCF) can uncover correlations with time-delays in non-simultaneous and unevenly spaced data : $$\begin{aligned}
 DCCF(\tau) = \frac{1}{N}\sum\limits_{i,j} \frac{(F_i^a - \bar{F}^a)\,(F_j^b - \bar{F}^b)}{S^a\,S^b} \label{eq:dccf}
\end{aligned}$$ where $F^a$ and $F^b$ are the fluxes of two lightcurves with mean $\bar{F}^a$ and $\bar{F}^b$ and variance $S^a$ and $S^b$, respectively. The sum goes over all $N$ pairs $i,j$ in the time interval $\tau$. The DCCF between the VHE and HE $\gamma$-ray fluxes is shown in Figure b for the full time frame and 2015-2016, respectively.

The scatterplot does not contain any strong evidence for a direct correlation between VHE (here integrated above 150 GeV) and HE fluxes in the data. While on several occasions a high HE flux is accompanied with a significant VHE flux, this is not a general rule, as also VHE-flux levels compatible with zero are recorded for similar HE fluxes. On the other hand, similar VHE fluxes can occur at different HE-flux levels. The 2016 VHE flare again stands out for the relatively low simultaneous HE fluxes. The non-correlation of VHE and HE fluxes is also underlined by the flat DCCF. It should be noted that the different integration times (a few hours for H.E.S.S. and $24\,$h for *Fermi*-LAT) might influence the conclusions here given the relatively fast variability found in this source.

For the 2015-2016 time frame, data from *Swift*-XRT and ATOM have been analyzed, giving the X-ray and R-band lightcurves for these years. The X-ray average, integrated flux is $\bar{F}(2\,\mbox{keV}~<~E~<~10\,\mbox{keV})~=~(9.5\pm0.1)\times 10^{-12}\,$erg cm$^{-2}$s$^{-1}$. The flux is incompatible with a constant flux with high significance, and $F_{\rm var}^{X} = 0.19\pm0.01$. The fastest variability is $t_{\rm var}^{X}= 8\pm4\,$h.[^1] Given the low cadence in these observations as visible in the third panel of Figure , it is difficult to distinguish flares from a ground state. In the optical R-band, the average, integrated flux is $\bar{F}(\mbox{R})=(8.605\pm0.005)\times 10^{-12}\,$erg cm$^{-2}$s$^{-1}$. The flux is highly variable with $F_{\rm var}^{R} = 0.679\pm0.002$, and $t_{\rm var}^{R}= 10.3\pm0.5\,$h. The lightcurve, shown in the bottom panel of Figure , reveals a highly active state in 2015 and a mostly quiet state in 2016. In April and May 2015 ATOM recorded correlated activity in the optical band with the HE $\gamma$-ray band. The very bright optical flare in July 2015, which was the brightest flux state ever recorded with ATOM in PKS 1510-089, only had a mild counterpart in the HE band. Compared to the other optical flares in 2015, the July outburst was more than 2 times brighter.

Scatterplots have also been produced for VHE versus X-ray and VHE versus R-band fluxes, which are shown in Figure a,b, respectively. As the number of data points are low, no DCCFs have been calculated. No conclusions can be drawn from the VHE versus X-ray scatterplot at this point. Unfortunately, no X-ray coverage was obtained during the 2016 VHE flare. The VHE versus R-band scatterplot suggests that high optical fluxes (i.e., above $2\times 10^{-11}\,$erg/cm$^2$/s) imply significant VHE fluxes (i.e., fluxes that deviate by more than 1$\sigma$ from zero). The low number of data points makes this a weak conclusion. However, high VHE fluxes do not imply high optical fluxes (using the same threshold), as is demonstrated by the 2016 VHE flare.

# The 2016 VHE Flare

The summary of the monitoring results in Sec.  has already hinted at the unprecedented nature of the flare in 2016. The nightly lightcurve around this event is shown in Figure . The flare lasted less than 3 days in the VHE band with a peak in the late hours of MJD 57538 (30 May 2016 —hereafter "maximum night"). In the HE band a flux rise seems to have happened. However, this is barely significant, as it is hovering around the long-term average and more than a factor 10 below previous flares. On the other hand, the spectral index clearly reduces compared to the average $\sim$`<!-- -->`{=html}2.4, peaking at $\sim 1.6$. Hence, while the integrated flux in the HE band barely changed, the spectrum itself significantly hardened. The optical flux rises by a factor of 2 from the beginning of the event to its peak. However, this again is a much smaller flux than that exhibited in previous outbursts. Unfortunately, there is no strictly simultaneous coverage of this flare in any other band.

A detailed lightcurve of the maximum night is shown in Figure . The VHE flux exhibits a peak with a flux $\sim$`<!-- -->`{=html}80% of the Crab above an energy of $200\,$GeV and a subsequent decay. From the peak to the minimum the flux fell by almost an order of magnitude. As the low flux in the HE band coupled with the small effective area of *Fermi*-LAT inhibits short-time binning, individual photons recorded with *Fermi*-LAT with energies $E>1\,$GeV are shown in the second panel. *Fermi*-LAT recorded photons with energies up to $E\sim 25\,$GeV during the H.E.S.S. observation window, but only 2 photons with energies $E>1\,$GeV in the MAGIC observation window. This is indicative of a softening of the spectrum at that time. The optical R-band flux recorded with ATOM exhibits a double-peaked structure, which is different than the VHE $\gamma$-ray lightcurve. The optical flux only changes by $\sim$`<!-- -->`{=html}30%.

The $\gamma$-ray spectra of the maximum night are shown in Figure  along with the HE and VHE $\gamma$-ray low-state spectra . The VHE $\gamma$-ray spectra have been corrected for the absorption by the extragalactic background light (EBL) using the model of . The resulting deabsorbed spectra of the flare are compatible with power-laws with indices $\Gamma_{H.E.S.S.} = 2.9\pm 0.2_{\rm stat}$ and $\Gamma_{MAGIC}= 3.37\pm 0.09_{\rm stat}$, respectively. The HE $\gamma$-ray spectra in the two VHE observation windows are compatible within errors being $\Gamma_{LAT}=1.4\pm 0.2_{\rm stat}$ during the H.E.S.S. time frame and $\Gamma_{LAT} = 1.7\pm 0.2_{\rm stat}$ during the MAGIC time frame, respectively. The spectral breaks between the HE and the VHE $\gamma$-ray band are, therefore, $\Delta\Gamma = 1.5\pm 0.3_{\rm stat}$ during the H.E.S.S. time frame and $\Delta\Gamma = 1.7\pm 0.2_{\rm stat}$ during the MAGIC time frame. The comparison with the low-state data clearly shows the significant shift of the peak energy from $E\sim 100\,$MeV to $E\sim 30\,$GeV during this flare.

The spectral break can have several causes. The underlying particle distribution could exhibit a break due to the interplay of acceleration and cooling. The break could also be a sign of the Klein-Nishina reduction of the inverse-Compton cross-section at high energies. That the break remains roughly constant between the respective observation time windows while the spectra get softer, could be an indication of the Klein-Nishina reduction. A third possibility is that the spectral softening results from the absorption of VHE photons by external soft photon fields, such as those from the BLR. The resulting optical depth for different VHE emission region distances from the black hole is shown in Figure . Obviously, the distance of the emission region from the black hole has a strong influence on the optical depth $\tau_{\gamma\gamma}$.

One can conservatively estimate an upper limit on the degree of absorption by assuming that the spectrum detected by *Fermi*-LAT represents also the intrinsic spectrum in the VHE domain. The degree of absorption $\tau$ can then be derived by $$\begin{aligned}
 \tau = \ln{\frac{F_{extra}}{F_{obs}}} \label{eq:tau},
\end{aligned}$$ where $F_{extra}$ is the extrapolated flux, and $F_{obs}$ is the observed flux.

Without going into details , the calculation for the H.E.S.S. data set give a maximum value of $\tau$ for the highest energies of $\tau=5.4\pm0.9_{\rm stat}$, while the MAGIC data gives a maximum value of $\tau=3.9\pm1.4_{\rm stat}$. These estimates agree within errors. Assuming the absorption is due to the BLR, the absorption values can be translated into a minimum distance of the emission region from the black hole. The emission region could be located at roughly $2\times R_{{\rm Ly}_{\alpha}} \sim 1.6\times 10^{17}\,$cm$\,\sim 0.05\,$pc from the black hole (c.f. Figure ). Assuming that the distance of the Ly$_{\alpha}$ line represents the radius of the BLR , the flaring region would at the very least be located on the outer edge of the BLR. This underlines the statement given above during the discussion of the monitoring data: the jet must be able to produce VHE $\gamma$-rays on distances on the order of a significant fraction of a parsec from the black hole.

# Summary & Conclusions

FSRQs are by now established VHE $\gamma$-ray emitters. However, whether they are able to produce the VHE emission at all times or only during short bright flares has been an open question. This has led to the establishment of monitoring programs by the IACT experiments H.E.S.S. and MAGIC on the FSRQ PKS 1510-089, which is one of the closest of this type of blazars (redshift $z_{\rm red}=0.361$). These programs are supplemented with multiwavelength data in the HE $\gamma$-ray, X-ray and optical regime. The first important result obtained by MAGIC is that PKS 1510-089 can be detected at VHE $\gamma$-rays during low states in the HE band . Similarly, variability has also been established through MAGIC observations .

The H.E.S.S. monitoring described in detail here adds important features to these earlier results. Strong variability is detected in the VHE $\gamma$-ray domain, while the observations also hint to a persistent flux at other times. Interestingly, comparison of the VHE $\gamma$-ray lightcurve with other energy bands does not reveal any obvious correlation. This exemplifies the need for deep monitoring programs across the entire multiwavelength spectrum. Otherwise important effects—such as better sampling of correlation functions, variability time scales, etc.—might be missed for the interpretation of certain events.

The latter statement is further emphasized by the detection of an unprecedented VHE $\gamma$-ray flare in 2016 with H.E.S.S. that was followed up with MAGIC, as well. It was more than 10 times brighter than any flux seen at VHE $\gamma$-rays before (with a peak flux of $80$% of the Crab) and lasted only 2 days. It was accompanied by a significant hardening of the HE $\gamma$-ray spectrum as observed with *Fermi*-LAT, while the HE fluxes remained rather low compared to other flares. The optical R-band observations with ATOM revealed a mild counterpart that was also much dimmer than previous flares, but exhibited a different flux evolution compared to the VHE band. Unfortunately, no other simultaneous data is available that could further constrain the spectrum.

All these observations reveal that the jet of PKS 1510-089 is able to accelerate particles to high energies to produce VHE $\gamma$-rays at all times. It also implies that these emission regions are probably located beyond the BLR, as otherwise the VHE emission should be strongly absorbed. This has been shown here specifically for the 2016 VHE flare. A simple estimate of the maximum absorption allowed for by the data results in a lower limit on the black hole distance, which indicates a location of the emission region on the edge of or beyond the BLR.

In conclusion, deep and, preferably, unbiased monitoring programs on FSRQs and blazars in general are important to reveal the general behavior of the sources, as well as to uncover new and unexpected features.

[^1]: The large error implies that this value is not highly significant. Trials might reduce the significance further. Hence, this time scale should be regarded as a lower limit.