abstract: Since May 1999 the BaBar detector has been taking data at the PEP-II asymmetric electron-positron collider at the Stanford Linear Accelerator Center, California. This experiment requires a very large data sample and the PEP-II accelerator uses intense beams to deliver the high collision rates needed. This poses a severe challenge to the BaBar trigger system, which must reject the large rate of background signals resulting from the high beam currents whilst accepting the collisions of interest with very high efficiency. One of the systems that performs this task is the Level 1 Calorimeter Trigger, which identifies energy deposits left by particles in the BaBar calorimeter. It is a digital, custom, fixed latency system which makes heavy use of high-speed FPGA devices to allow flexibility in the choice of data filtering algorithms. Results from several intermediate processing stages are read out, allowing the selection algorithm to be fully analysed and optimized offline. In addition, the trigger is monitored in real time by sampling these data and cross-checking each stage of the trigger calculation against a software model. The design, implementation, construction and performance of the Level 1 Calorimeter Trigger during the first year of BaBar operation are presented.
author: P.  D.  Dauncey,[^1] J.  C.  Andress,[^2] T.  J.  Adye,[^3] N.  I.  Chevalier, B.  J.  Claxton, N.  Dyce, B.  Foster, S.  Galagedera, A.  Kurup,[^4] A.  Mass, J.  D.  McFall, P.  McGrath, S.  J.  Nash, D.  R.  Price, U.  Schäfer, I.  Scott and D.  C.  H.  Wallom
title: Design and Performance of the Level 1 Calorimeter Trigger for the BaBar Detector [^5]

# PEP-II and BaBar

BaBar experiment , has been built to study $CP$-violation in the decays of $B$ mesons. These are produced in the decays of $\Upsilon(4S)$ mesons created in $e^+e^-$ collisions at the PEP-II collider at the Stanford Linear Accelerator Center in California, USA. $CP$-violating effects are expected to be subtle, and so these measurements require very large statistical samples of $B$ mesons. This means that the PEP-II machine must deliver unprecedentedly high luminosity, and indeed it has already set a new world record luminosity of greater than $2\times 10^{33}$ cm$^{-2}\,$s$^{-1}$ during the first year of operation, enabling BaBar to record a data sample of approximately 20 million $B$ meson decays.

The high luminosity of PEP-II has been obtained by using very high beam currents, of order 1 A. The PEP-II machine design achieves this by filling the machine with 1658 bunches in each beam, with a bunch collision rate of 238 MHz. A serious consequence of this high current is that the machine-induced backgrounds have been very significant. This has provided a major challenge for the BaBar trigger and data acquisition system, as this unwanted background must be rejected without losing the $B$ meson decays of interest. Since high statistics are vital for this experiment, the trigger design emphasized the need for high efficiency.

# The Level 1 Trigger Requirements

The BaBar trigger has two levels, a hardware Level 1 and a software Level 3 trigger. This article describes one of the two main sections of the Level 1 trigger, the ElectroMagnetic calorimeter Trigger (EMT). The other main section, the Drift Chamber Trigger (DCT), has been described previously . A third component, the Instrumented Flux return Trigger (IFT) is used to collect events with muon particles which are used for calibration purposes. This part of the trigger is not primarily designed to be efficient for $B$ mesons.

These two main subsystems work independently and in parallel, processing data from the calorimeter and drift chamber detectors respectively, and seek characteristics of particles coming from $B$ meson decays to identify signals. These signals, termed trigger "primitives", are then passed to a third system, the Global Level 1 Trigger (GLT). Here the individual primitives are combined to form a picture of the whole particle collision, and the GLT then decides whether or not to accept this collision. This trigger decision is passed to the fast control system where it is distributed to the whole detector. Fig. 1 shows the systems with which the Level 1 trigger interacts.

The technical requirements for the Level 1 trigger system as a whole are set by the data acquisition  and other electronics systems of BaBar. To ensure maximum redundancy, the EMT and DCT subsystems were each designed to satisfy these requirements independently.

All the front-end electronics systems of the BaBar detector buffer their data for 12 $\mu$s, which defines the latency allowed for the Level 1 trigger decision. The collision rate of 238 MHz is effectively continuous as far as the time resolution of the drift chamber and calorimeter detectors are concerned, and so the Level 1 trigger has the task of determining the time of the collision of interest to within 1 $\mu$s. Each system in BaBar then reads out data in a window of at least 1 $\mu$s, in order to ensure that the data from the triggered collision are fully collected.

The data acquisition system is designed to accept a maximum Level 1 output rate of 2 kHz. This is much higher than the rate of physics collisions, which totals around 120 Hz. The Level 1 trigger is designed to be as unbiased as this maximum output rate allows, and to accept all physics events without preselection. The challenge is to provide such a trigger whilst still rejecting sufficient background to keep the total rate below 2 kHz.

A number of other requirements must be satisfied in addition to high efficiency. The efficiency must be measurable accurately, the trigger must be stable and robust to varying background levels, and it must be flexible enough to be adaptable to unexpected operating conditions. The first of these is ensured by the redundancy of the EMT and DCT triggers. The remainder were satisfied by designing the trigger to be able to operate at up to ten times the expected level of background. The EMT design described below has been shown to fulfill all the requirements.

# The EMT Data Interfaces and Algorithm

The EMT receives its data from the electromagnetic calorimeter. There are 6580 channels in the electromagnetic calorimeter, each giving a 16-bit energy value at 3.7 MHz. These data are extracted continuously (without triggering) from the front-end electronics via 280 fibre-optic links . In order to reduce the EMT input data rate, the data are summed over typically 24 channels before being sent to the EMT, since a physical energy deposit is usually spread over multiple calorimeter channels. This reduces the input rate to 2.1 GBytes/s, which are transmitted as serial 59.5 MHz differential ECL signals. To compensate for differing cable lengths and component time delays, the data are resynchronised to the 59.5 MHz clock when they are received by the EMT.

The technique used by the EMT to identify particle energy deposits relies on the fact that the probability of multiple particles arriving close together in time in the same region of the calorimeter is low. This is true even in the high background conditions of PEP-II. This means energy deposits may be summed over azimuthal ($\phi$) slices of the calorimeter, yielding 40 so-called "$\phi$ strips", without significantly increasing the probability of pile-up. This considerably simplifies the trigger logic, and the EMT processing now becomes a one-dimensional, rather than two-dimensional, problem. Each $\phi$ strip corresponds to 165 crystals; these are added in overlapping pairs to form 40 "$\phi$ sums" in order to fully contain energy deposits which extend into a neighbouring $\phi$ region.

An overview of the algorithm applied to the $\phi$ sums is shown in Fig. 2.

The data from each sum are used in two ways: to find the time at which the energy was deposited, and to compare the total energy against several thresholds.

The time is determined using a digital 8-tap finite impulse response (FIR) filter which operates on the summed energy signal from each $\phi$ sum at a frequency of 3.7 MHz. Before digitisation, the calorimeter signal is shaped by a three stage differentiator-integrator-integrator (CR-RC-RC) circuit with time constants of 0.8 $\mu$s, 0.25 $\mu$s, 0.25 $\mu$s respectively. The EMT FIR weights are chosen such that this shaped signal drives the filter output from positive to negative at a fixed time interval after the particle arrives. This is illustrated in Fig. 3. It is this "zero-crossing" which is detected and used as the basic time estimate. To increase the accuracy of this determination, the EMT also performs a linear interpolation between consecutive FIR output values to approximate the output at 7.4 MHz.

In parallel, the energy values of the $\phi$ sums are compared against three threshold values. These configurable thresholds are typically set to values around 120 MeV, 300 MeV and 800 MeV. The first is designed to be efficient for minimum ionising particles, which on average deposit 180 MeV in the calorimeter. Typical GLT triggers would require four non-neighbouring $\phi$ sums above this lowest threshold, or three $\phi$ sums with a minimum $\phi$ angle separation between at least one pair of them of $120^\circ$. The other thresholds are set as low as possible, consistent with reasonable background trigger rates. These data are typically used in the GLT for triggers requiring two $\phi$ sums, such as two $\phi$ sums above the 300 MeV threshold with a minimum $\phi$ angle separation of $120^\circ$, or two non-neighbouring $\phi$ sums above the 800 MeV threshold with no further angle requirement.

Each of the three thresholds from each $\phi$ sum has a corresponding output bit. The bits from pairs of $\phi$ sums are put through an OR so as to reduce the number of bits for the whole range of $\phi$ from 40 to 20; this is necessary as the GLT does not have the input bandwidth for a 40-bit wide array. The degradation on the trigger performance because of this OR has been checked to be negligible. These bits together form the EMT primitives. The number and position of these bits are used in the GLT to determine if a valid trigger condition existed, such as the ones outlined above. The time determination from the FIR filter is used to gate the threshold bits so that they are only set on around the time of a valid energy deposit. The bits are transmitted to the GLT as differential PECL at a rate of 7.4 MHz, giving a total data rate for these data of 450 Mbits/s.

# Implementation of the EMT

The algorithm outlined above was implemented so that data from each $\phi$ sum are processed by a single "Algorithm" Xilinx 4020E FPGA clocked at 59.5 MHz. All 40 such components operate in parallel. They are distributed over ten Trigger Processor Boards (TPB), each containing four such FPGA's, all with identical firmware. Each FPGA, and TPB, runs independently of the others, so the data cannot be considered at a "global" event level until they reach the GLT. An overview of the whole EMT system is shown in Fig. 4.

The TPB's are housed in a 9U VME crate with a standard J1 backplane and three custom backplanes covering J2 and J3. The EMT was built to operate with the same protocol as every front-end system in BaBar, in order to minimise the need for system-specific readout software. This protocol is implemented via a TX/RX fibre-optic connection  from a BaBar ReadOut Module (ROM) , which is common to every system in the detector. A split J2 and J3 backplane divides the crate into two independent sections, one occupied by the ROM and a related fast control and timing module, the other containing the EMT cards.

The whole EMT is timed and controlled via the fibre-optic connection from the ROM. The fundamental 59.5 MHz clock signal and all configuration and control commands are sent along this fibre and distributed across the custom backplane to the TPB's via an optical-to-electrical converter TRansition Board (TRB) in the centre of the crate. Because of the high clock speed, all the custom backplane signals were implemented as differential 100 $\Omega$ ECL point-to-point connections. In addition, care was taken to route the differential lines on the custom backplane as closely as possible and equalise the signal return delay times between the TRB and each TPB. The command protocol is decoded on the TPB by a "Fast Control" Xilinx 4013E FPGA, which also performs all control functions on the board. All output data from the TPB's are sent via the backplane to the TRB where they are converted to optical signals. Hence, the fibre-optic connection back to the ROM is used to transmit the event data and to read back configuration data, as standard in BaBar.

The major components on the TPB are shown in Fig. 5 and a photograph of a production board is shown in Fig. 6. Besides the algorithm calculation itself, the TPB performs several other functions that are described in the following paragraphs.

The data associated with the triggered event are read from the EMT whenever there is a Level 1 trigger and are buffered in the TPB for the 12 $\mu$s latency. They are then stored in one of four event buffers until a readout request is sent from the ROM. Each event buffer can store up to $\pm 2 \mu$s of data around the event time, this length being a configurable parameter. The latency and event buffers are configured and controlled by a "Formatter" Xilinx 4013E FPGA. These data are used offline for detailed checks on the trigger performance and for tuning the algorithm configuration parameters.

The TPB's have large memory arrays which can be loaded with arbitrary data patterns via the ROM. These come in two sets; the "front-end" and "back-end" playback buffers. These can be configured to inject the stored data to the front of the algorithm processor, instead of the input data from the calorimeter, or to inject the stored data to the EMT output, instead of the normal output data to the GLT. Around 140 $\mu$s of data can be stored. Any chosen data values can be clocked through the entire EMT chain. This functionality proved invaluable while testing the boards after production.

In addition to the standard BaBar control path, a read-only VME interface was built into the TPB. This allows several on-board "spy" memories to be read at a low rate while the trigger is active. These spy memories are filled with data from several locations in the trigger algorithm data path, namely the raw input data, the results of a number of intermediate processing stages and the final output to the GLT. The memories allow around 140 $\mu$s of data to be stored at a time. This allows detailed, bit-level validation that the EMT is functioning correctly by comparing the spy data with a software model. The combination of the spy and the playback memories significantly reduced the time taken in the prototype and production testing cycles.

# Performance of the EMT

The EMT was fully installed for the start of BaBar data-taking in May 1999 and has operated successfully thereafter. Fig. 7 shows a typical event from the perspective of the Level 1 trigger. The detector is drawn end-on, with the calorimeter information being shown in the outermost layer. The solid blocks give the offline reconstructed calorimeter information, the hashed blocks the EMT data. Four significant energy deposits were identified, which together caused a trigger. In addition, four tracks were found in the DCT and these are shown as the lines in Fig. 7; they are clearly correlated with the calorimeter deposits. The other DCT information, shown in the inner part of Fig. 7, indicates the track hits detected (the small circles) overlayed on the drift chamber cell geometry. This event in fact satisfied several redundant triggers, which is typical of the physics processes of interest. Such events allow cross-checks of the EMT and DCT trigger systems against each other.

No major components have needed to be replaced. The only minor hardware concern has been the pin housing of the calorimeter-to-EMT cables which was damaged through removal of misplaced pins during the cable assembly. The faulty housings have been replaced.

The EMT satisfied the technical requirements for latency and time jitter without any tuning of the FIR filter weights beyond the values found from a simulation study during the design phase. Fig. 8 shows the distribution of times determined in the EMT for individual energy deposits, compared to a very much more accurate offline estimate using the drift chamber data. This demonstrates the time of each deposit is usually determined to much better than 1 $\mu$s. The overall requirement for the Level 1 trigger of 1 $\mu$s applies to the whole event. This event time is determined in the GLT from all the primitives, potentially including those from the DCT, and is effectively a weighted average. The time resolution of the event is therefore significantly better than that of each individual energy deposit.

Typical Level 1 trigger rates are below 1 kHz, well within the 2 kHz limit. Fig. 9 shows the overall rate for DCT and EMT triggers combined as a function of the PEP-II beam currents. These rates were achieved despite a factor of twenty times higher backgrounds than expected levels, showing the importance of designing for high background rates from the outset. The GLT uses a combination of EMT-only triggers, DCT-only triggers and EMT-DCT combined triggers. In addition, an event often has multiple trigger conditions satisfied simultaneously. It is therefore not possible to give an unambiguous value for a trigger rate due to the EMT; the rate of events which only trigger because of the EMT is around 14% of the total, while removing the EMT completely from the Level 1 trigger would make the rate fall by around 50%.

The EMT efficiency for finding individual energy deposits above threshold is above 99%, except in regions with noisy channels which had to be excluded from the trigger (see below). Fig. 10 shows this efficiency as a function of the deposited energy around the lowest threshold used of 120 MeV. This threshold is designed to give a good efficiency for minimum ionising particles, which deposit an average of 180 MeV. It is seen that the trigger is at full efficiency by this energy.

During the initial data-taking period, several faulty channels were found which sporadically became noisy. Whenever this occurred, it was necessary to modify a configurable mask to exclude these noisy regions from the trigger. Some were due to isolated calorimeter channels which were then disabled individually within the calorimeter electronics, allowing the others in the 24 channel sum to continue to be used. However, some noisy regions were due to faults downstream of the sum, in which case the input from all 24 had to be disabled. The average number of input channels masked out over the first year was 1.5%. Simulation studies indicate that these dead regions caused a negligible loss of efficiency for $B$ meson decays.

The high efficiency per energy deposit translates directly into high efficiencies for physics processes. The EMT efficiency for Bhabha events, with both outgoing particles falling within the calorimeter acceptance, has been measured to be at least 97%, with some of the apparent inefficiency being due to muon pairs contaminating the sample used to measure the efficiency. The EMT efficiency for $B$ meson final states is more than 99%, where the high figure is achieved through multiple redundancy; typically the GLT requires four of the $\phi$ sums to be above the 120 MeV threshold, but hadronic events have around ten such deposits on average.

# Conclusions

The EMT has performed well during the first year of BaBar data-taking. The hardware has had no significant problems and the high-speed FPGA design has functioned correctly throughout this period. The trigger has proven to be robust against backgrounds that were significantly higher than expected, has been flexible enough to deal with changing conditions, and has maintained a very high efficiency for all physics collisions of interest.

# Acknowledgment

The authors would like to thank Su Dong for his help in the preparation of this paper.

[^1]: Paul Dauncey and David Price are with the Department of Physics, Imperial College, University of London, Prince Consort Road, London SW7 2BW, UK. For inquiries about this paper, contact Paul Dauncey; telephone: +44-20-7594-7803, email: email@example.com.

[^2]: John Andress, Nicole Chevalier, Neil Dyce, Brian Foster, Alexander Mass, Jason McFall, Steven Nash, Uli Schäfer and David Wallom are with the Department of Physics, University of Bristol, Tyndall Avenue, Bristol, BS8 1TL, UK.

[^3]: Timothy Adye, Brian Claxton and Senerath Galagedera are with the Rutherford Appleton Laboratory, Chilton, Didcot, OX11 0QX, UK.

[^4]: Ajit Kurup, Paul McGrath and Iain Scott are with the Physics Department, Royal Holloway and Bedford New College, University of London, Egham, TW20 0EX, UK.

[^5]: Manuscript first received November 2, 2000. This work was funded by the Particle Physics and Astronomy Research Council, UK.


abstract: Scattering scanning near-field optical microscopy enables optical imaging and characterization of plasmonic devices with nanometer-scale resolution well below the diffraction limit. This technique enables developers to probe and understand the waveguide-coupled plasmonic antenna in as-fabricated heat-assisted magnetic recording heads. In order validate and predict results and to extract information from experimental measurements that is physically comparable to simulations, a model was developed to translate the simulated electric field into expected near-field measurements using physical parameters specific to scattering scanning near-field optical microscopy physics. The methods used in this paper prove that scattering scanning near-field optical microscopy can be used to determine critical sub-diffraction-limited dimensions of optical field confinement, which is a crucial metrology requirement for the future of nano-optics, semiconductor photonic devices, and biological sensing where the near-field character of light is fundamental to device operation.
author: Lauren M. Otto; Stanley P. Burgos; Matteo Staffaroni; Shen Ren; Özgün Süzer; Barry C. Stipe; Paul D. Ashby; Aeron T. Hammack
bibliography: sSNOM.bib
date: 2024-10-03
title: Predicting Scattering Scanning Near-field Optical Microscopy of
  Mass-produced Plasmonic Devices

# Introduction

Today's optic, photonic, and plasmonic devices incorporate light into nanoscale devices that enable diverse new functionality for chemical or biological sensing and imaging , precision light sources , silicon photonics-based communications and signal processing , and data storage . These devices are often critically dependent on the behavior of light well below the diffraction limit, and therefore require robust near-field characterization techniques, linked with numerical predictions and validations, to design and fabricate functioning devices. This requirement becomes even more essential when such devices are to be manufactured in massive quantities that require process quality control. In fact, future hard disk drives (HDDs) will rely on near-field optical heaters for data storage, and the successful development and production of novel HDD magneto-optical write heads is largely contingent on the ability to reliably predict and validate the near-field behavior of as fabricated devices by direct experimental characterization. The technical improvements to scattering scanning near-field optical microscopy that we have developed for HDD write head optical antenna characterization are potentially extensible and beneficial to any field of study involving the direct optical measurement of photonic modes arising in proximity to nanoscale features.

# Heat Assisted Magnetic Recording Background

Despite alternative technology developments, most of the world's increasing data storage demands are still satisfied by HDDs. To improve efficiency, the HDD industry has encountered and worked to overcome fundamental limitations preventing the further areal density growth on a disk of magnetic material . Currently used magnetic media cannot support smaller bit sizes (known as the superparamagnetic limit, smaller bits are less stable), but magnetically harder media has proven too difficult to write (alter magnetization) using traditional magnetic write head strengths/capabilities. In order for data to be recorded into this media, a local spot on the disk with dimensions of the desired bit size (10s of nanometers) must be heated close to its Curie temperature resulting in a significant drop of its magnetic coercivity, thus allowing the magnetic polarization to be switched under the applied field. The chosen method for locally heating the magnetic media in under-development devices is a plasmonic antenna, which focuses far-field optical light into high-intensity and rapidly decaying near-field optical energy, which is directed onto the recording medium (Figure a). The high-intensity near-field energy is capable of heating the magnetic media (only a few nanometers away) to the desired temperature while the rapid decay of the evanescent field ensures that the thermal gradient in the media is large so the resulting bit size is comparable to the size of the antenna and its near-field "spot." During drive operation the media is a fast-rotating disk, and as the disk rotates, the magnetic grains in the heated spot experience an applied magnetic field from the write pole, thereby switching their magnetization. Upon further disk rotation, the recorded bit is removed from the near-field hot spot, cools, and the magnetization is therefore "frozen" into the media. This technology is known as heat-assisted magnetic recording (HAMR), and it is capable of achieving the desired smaller bit sizes and higher areal densities .

Designing, developing, and fabricating the HAMR technology has numerous challenges associated with the newly integrated laser, photonic, and plasmonic elements in addition to the magnetic elements already present. While electromagnetic simulations are immensely powerful and able to guide design and predict results, experimental characterization techniques are necessary to verify modeling results when compared to empirical observations of as-fabricated devices, which can be subjected to many variations, and to determine failure mechanisms. In HAMR, the introduction of a plasmonic antenna generating a deep subwavelength near-field spot calls for a characterization technique capable of investigating the properties and consistency of this crucial feature of the HAMR head's performance. Former work has demonstrated scattering scanning near-field optical microscopy (sSNOM) to be a top candidate for this task .

# Scattering Scanning Near-field Optical Microscopy Background

SNOM was originally developed as a probe system with an aperture in a diaphragm , which later was traded for a tapered optical fiber coated with metal such as gold, silver, or aluminum . Propagating far-field light inside the fiber is converted to near-field radiation by the angled metal coating and then illuminates the sample. This probe also serves to convert the sample-interacted near-field back into propagating far-field light, which would be observed back through the optical fiber. However, the resolution limit of apertured SNOM is determined by the dimensions of the aperture and the excitation wavelength so that as the aperture shrinks, the amount of electromagnetic energy coupled through the aperture and transmitted to the far-field falls off precipitously according to the Bethe limit ($\propto a^6/ \lambda ^{2}$) . (Bethe's analytical near-the-hole solutions were later corrected by Bouwkamp , and a modern discussion is provided by Novotny *et al.* ) As a result, for small aperture sizes, the near-field signal accessible using an aperture probe falls below the background and noise thresholds for the technique, thus limiting resolution of apertured SNOM to $\sim$`<!-- -->`{=html}50 nm for visible or near-infrared light. Because the HAMR plasmonic antennas are generally smaller than this size, a technique offering better resolution is necessary.

In sSNOM, the resolution of the SNOM technique is increased by incorporating an atomic force microscopy (AFM) scan probe tip that scatters, rather than transmits, the near-field light into the far-field, thus causing the sharpness of the AFM probe to determine the resolution rather than the size of the aperture and the wavelength . For sub-ten-nanometer tip radii, the scattering cross-section of light is still more than sufficient to beat the signal-to-noise and signal-to-background limits when using a lock-in amplifier to detect the light scattered from the tip . In the case of metallic or metal-coated tips, the tip both generates the probing near-field (through the "lightning rod" or "nanofocusing" effect) and performs the function of a dipole scatterer by converting the sample-interacted near-field radiation back into far-field light that can be collected by an objective . However, for HAMR plasmonic antennas (as well as for other general plasmonic structures), the near-field is generated by the sample (excited by an external source), and so tips with minimal signal disruption while still maintaining high scattering cross-sections are desired . In this work, uncoated silicon tips with a typical radius of $\sim$`<!-- -->`{=html}5-15 nm and optically accessible tip apexes were used.

Given the large amount of background scattered light that is generally present in sSNOM measurements, the signal specific to the tip apex can be extracted through the use of a lock-in amplifier tuned to the fundamental and higher harmonics of the AFM tip cantilever's resonant oscillation frequency . Prior work has demonstrated that the lower order harmonics include both the propagating and evanescent components of the scattered light, while the higher order harmonics have a stronger dependence on the near-field strength and hence contain higher contrast near-field information . The sSNOM system used to measure the HAMR plasmonic antennas (Figure b) in this work uses this lock-in driven technique and is capable of capturing up to three harmonics simultaneously during AFM measurements (Figure c).

# Predicting Behavior of Mass-produced Plasmonic Devices

In order to use sSNOM to guide the design and development of HAMR devices, we developed a model to map the electric field data from electromagnetic simulations into the expected harmonic mappings generated by the sSNOM system. The first stage began with a few general approximations. It was assumed that the position of the AFM tip varied sinusoidally in time with the resonant frequency of the cantilever ($z \propto sin(\omega _{0} t)$), that the evanescent electric field followed an exponential decay with increasing distance from sample surface $z$ ($E \propto e^{-z}$), and that the scattering was proportional to intensity ($\sigma \propto E^2$). As expected, the model yields maximum scattering intensity when the tip is closest to the surface (Figure a, here $\omega _{0}$ = 300 kHz), and a fast Fourier transform (FFT) of the scattered field yields harmonics that decay exponentially in intensity with increasing order. Using the sSNOM system and oscilloscope capabilities of the lock-in amplifier software, a scattering signal similar in shape to that of the simple evanescent approximation was observed when the tip was hovered (oscillating) over the center feature of the plasmonic antenna containing the most intense near-field signal (Figure c). Despite the noise present in this data, a numerical FFT still reveals several harmonics (Figure d). It is worth noting that this FFT is performed through numerical analysis of the captured time-domain oscilloscope trace, not by the lock-in amplifier hardware. The lock-in harmonic acquisition during imaging was performed with sufficiently long time constants in order to improve the signal-to-noise ratio in the higher harmonics significantly above the threshold apparent in this post-processed FFT trace.

Because the sSNOM system and HAMR devices are considerably more complex than the simple evanescent model, further improvements to the model were made to better match experimental conditions. To substitute for the simple evanescent electric field assumption ($E \propto e^{-z}$), data from electromagnetic simulations were introduced to account for the actual field expected for the as-designed dimensions of the HAMR plasmonic antenna. The tip-sample interaction was then accounted for in two stages. The first stage weights each 1 nm $\times$ 1 nm $z$-column (where the column is aligned normal to the surface of the sample) of simulated electric field data by a scattering parameter $\sigma$: $$E_{sSNOM} = \sigma E_{model}
    \label{sSNOMfield}$$ which yields the scattering parameter adjusted electric field. The scattering parameter $\sigma$ is derived from a spherical probe and image dipole interaction and is defined by the following set of equations: $$\begin{aligned}
    \label{sSNOMscattering}
    \sigma &= \alpha _{eff} \frac{\pi \sqrt{\frac{8\pi}{3}}}{\lambda ^{2}} \\
    %\label{sSNOMeffpolariz}
    \alpha _{eff} &= \frac{\alpha (1 + \beta)}{1 - \frac{\alpha \beta}{16 \pi (a + z)^{2}}} \\
    %\label{sSNOMpolariz}
    \alpha &= 4\pi a^3 \frac{\epsilon _t - 1}{\epsilon _t +2} \\
    %\label{sSNOMbeta}
    \beta &= \frac{\epsilon _s -1}{\epsilon _s +1} \\
\end{aligned}$$ where $\alpha _{eff}$ is the effective polarizability of the tip-sample combination, $\alpha$ is the polarizability of the silicon tip (approximated here as a sphere), $\lambda$ is the free space wavelength of the incident light, and $\epsilon _{t}$ and $\epsilon _{s}$ are the complex dielectric functions of the tip and sample, respectively. $\epsilon _{s}$ was allowed to vary based on the material (either gold, oxide, or magnetic material) located directly below the $z$-column being computed.

After applying the scattering transformation to the electric field, the second stage accounted for the shape of the tip. Planar slices of the resulting weighted electric field parallel to the air-bearing surface (ABS), which is the surface facing the rotating magnetic media containing the plasmonic antenna (in the $xy$ plane), were serially convolved with two matrices with each representing different components of the shape of the AFM tip. One matrix was designed as a cone to represent the shape of the tip's shaft, and the other matrix was designed as a top-hat to represent the shape of the end of the tip. After these convolutions, for a 1 nm $\times$ 1 nm $z$-column positioned directly above the plasmonic antenna, it can be seen that the modeled time-dependent scattered field (Figure e) more closely matches the shape of the measured scattered field of the sSNOM signal (Figure c) than does the scattered field resulting from the simple evanescent model (Figure a). Similarly, a FFT of this modeled sSNOM signal (Figure f) mimics the behavior of the measured signal from the lock-in amplifier (Figure d) yielding smooth harmonic spikes, and when performed over the entire $xy$ plane, yields the desired modeled near-field harmonic mappings.

# Effects of the AFM Tip's Shape

Development of the sSNOM model required further investigation of the AFM tip's local shape interacting with the high-intensity near-field. Based on manufacturer specifications, our experience, and previous transmission electron microscopy (TEM) images, it was assumed that most tips used in these experiments contained a 5-15 nm radius, where the sharper tips (5 nm radius, Figure a) gave a sSNOM signal with better resolution than average (10 nm radius, Figure b) or blunter ($>$`<!-- -->`{=html}15 nm radius) tips. Furthermore, TEMs of tips imaged post-scanning revealed several nanometers of material built up on the tip, including in the region separating the silicon tip from the sample. Therefore, the corresponding change in tip-sample separation was accounted for in the sSNOM model in addition to the tip's radius. The TEM images also revealed the angled nature of the used tip's shaft as well as the flat top-hat nature of its end.

Both the shape of the tip's shaft as well as its end were incorporated into the sSNOM model through the two convolution matrices described above. The shaft was approximated as a cone with a half-angle of 9$^\circ$, and the tip end was approximated as a top-hat (step function in cylindrical coordinates) with variable radius. Radii of 5, 10, and 15 nm represented the range of usual tips while radii of 20 and 40 nm represented more extreme cases of blunt tips (Figure c). Based on the developed sSNOM model and previous experience with these tips, a radius of 15 nm was used in the model for correlation with the experimental data. Further improvement of the model is possible through the use of a three-dimensional volumetric convolution, rather than two-dimensional planar convolution used in this work, of the conical tip shaft matrix (including the tip's angled rather than vertical approach) with the scattered field, which is expected to more accurately model the physical configuration and may increase the far-field contribution to the $1 \omega _{0}$.

# Effects of the AFM Tip's Distance of Closest Approach

Further investigation of the mechanical AFM behavior of the sSNOM system was performed in order to determine the tip's distance of closest approach (DCA) to the sample. In the sSNOM model, the DCA value was set as the minimum $z$ height (and cutoff) for the FFTs of the scattered signal. It was found that the chosen DCA had a significant effect on the sSNOM model's results (Figure ). A smaller DCA (tip approaches much closer to the ABS) incorporates much more of the dominant near-field signal (as opposed to the weaker far-field signal) into the FFT, which becomes evident in the harmonic maps for the fundamental frequency ($1 \omega _{0}$, Figure a) when compared with the higher $6 \omega _{0}$ frequency (Figure b). Maximum intensity plots are also included for all investigated harmonics and heights as well as additional harmonic maps in the Supplemental Information (Figures -).

Estimation of tip DCA was also performed with a mechanical model of tapping mode AFM, which solves the equation of motion for the cantilever numerically as it encounters tip-sample forces modeled to include van der Waals attraction, capillary adhesion due to a water layer, and Pauli hard sphere repulsion . When inputting the tip radius, free amplitude, and amplitude setpoint (see Supplementary Information), and using the measured phase signal as a check for accuracy, the model shows that the cantilever was oscillating in the repulsive regime indicating that the tip approaches the surface very closely and experiences repulsive contact with the sample at the bottom of most oscillations.

# Results

Accounting for the thickness of material collected by the tips during scans ($\sim$`<!-- -->`{=html}5 nm in Figure a,b) as well as other complex effects due to the water meniscus between the tip and the sample surface , and comparing the experimentally measured data with the sSNOM model's results, a DCA of 8 nm was chosen together with a tip radius of 15 nm to represent the best fit of the sSNOM model to the experimental measurements and expectations. A "good fit" between the harmonic maps generated by the sSNOM model and the experimental data was established when a bifurcated near-field signal was observed in the higher order harmonics ($5 \omega _{0}$, $6 \omega _{0}$) but not in the lower order harmonics ($1-3 \omega _{0}$). The visible near-field bifurcation requirement for the higher harmonics ruled out the possibility of the experimental tip having a radius of 20 nm or larger since no bifurcation was evident, even in the $6 \omega _{0}$ (Figure ), when the chosen distance of closest approach was 1 nm (the closest possible in the present model). In addition, good qualitative agreement between the experimental and modeled near-field maps using smaller tip radii ($<$`<!-- -->`{=html}15 nm) required DCA values exceeding expectations and so were ruled out by the AFM mechanical model (Figures -). Since the sSNOM model's results show significant dependence on the DCA, incorporating simulated electric field data with topographic sensitivity (the presented simulations assumed a planar surface) would more closely match the experimental conditions revealed by the recessed nature of the waveguide relative to the antenna seen in the AFM image and could improve the model.

Slices of the simulated electric field intensity at different heights of interest (Figure a, with further details included in the Supplemental Information are compared to the corresponding harmonics from the sSNOM model results using a tip radius of 15 nm and a DCA of 8 nm (Figure b) and the experimentally measured harmonic maps (Figure c). From this comparison, it can be seen that higher harmonics, such as $5 \omega _{0}$ and $6 \omega _{0}$, are more representative of the near-field in the region that would be occupied by the magnetic media during drive operation than their lower order harmonic counterparts. The nature of the higher order harmonics in which this relationship was found signified the prescribed rapid change in near-field intensity closer to the plasmonic antenna. The strange behavior exhibited by the measured $4 \omega _{0}$ was likely caused by a non-sinusoidal feature at the bottom of the tip's oscillation due the tip's possible contact with and/or repulsion by the sample surface during its motion. The non-sinusoidal motion of the tip was not accounted for in the present sSNOM model, but it could be introduced upon further mechanical characterization of the sSNOM-tip system and its interaction with the sample. An introduction of a sharp change in the peak scattering would surely become evident in one or more of the higher order harmonics in the presence of non-sinusoidal tip oscillation. In the case of this experiment, this feature was found in the $4 \omega _{0}$, and it is also expected to have been present in the $8 \omega _{0}$, etc. This observation as well as the AFM mechanical model results and the flat nature of the post-scanning tips suggests that the scan parameters cause significant tip interaction with the sample likely leading to the tip's top-hat shape. Scan parameters for typical non-contact tip operation involve small oscillation amplitudes that fail to sweep sufficient evanescent field variation and are not capable of performing the desired full characterization of the plasmonic antenna's near-field. For this reason, large oscillation amplitudes were used that resulted in greater tip-sample interactions.

In conclusion, sSNOM is a useful technique capable of characterizing HAMR heads for next-generation hard disk drives. The development of a sSNOM model to translate from simulated electric field data was necessary in order to derive meaning from the measured near-field harmonics. Furthermore, the same model approach can be applied to other plasmonic structures across many disciplines and applications.

# Methods

Steady state electromagnetic modeling of the plasmonic system was performed using the FDTD method in Lumerical.

sSNOM measurements were performed with the AIST-NT CombiScope 1000-SPM and tips from Nanosensors (ATEC-NC). The auto non-contact/tapping default mode was used with the 160 $\mu$m cantilevers ($\sim$`<!-- -->`{=html}300 kHz resonance), 80 nm oscillation amplitude, and 87 percent set point. An 830 nm laser (diode: Sanyo DL8142-201) was polarized (Thorlabs LPVIS100-MP) and directed with free space optics into the microscope. The microscope's bottom objective (Olympus ULWD MSPlan 50) mount was piezo tuned in three dimensions first for coarse alignment and then for optimal coupling of the laser spot into the waveguide once the tip was hovered over the plasmonic antenna. A 50$\times$ objective (Mitutoyo NIR M Plan Apo NIR 50$\times$, 378-825-5) was precisely aligned with piezo scanners (in objective's $xy$ plane, manual in $z$) and used to collect the scattered light, which was then observed by an APD (Thorlabs APD120A). The resulting signal passed through a lock-in amplifier (Zurich Instruments HF2LI), and data was collected through the AIST-NT software.

# Acknowledgements

L.M.O. acknowledges the National Institutes of Health Biotechnology Training Grant (T 32 GM008347) and the National Science Foundation Graduate Research Fellowship Program (00039202). Work performed at the Molecular Foundry (TEM images and AFM mechanical model) was supported by the Office of Science, Office of Basic Energy Sciences, of the U.S. Department of Energy under contract no. DE-AC02-05CH11231.


author: Aki Vehtari[^1]; Daniel P. Simpson[^2]; Yuling Yao[^3]; Andrew Gelman[^4]
date: 12 Oct 2018
title: **Limitations of "Limitations of Bayesian leave-one-out cross-validation for model selection"[^5]**

# What was not but could be if

The most important aspect of communicating statistical method to a new audience is to carefully and accurately sketch out the types of problems where it is applicable. As people who think leave-one-out cross validation (LOO-CV or LOO for short) is a good method for model comparison and model criticism, we were pleased to discover that Gronau and Wagenmakers (2018, henceforth GW) chose to write a paper aimed at explaining the nuances of LOO methods to a psychology audience. Unfortunately, we do not think the criticisms and discussions provided in their paper are so relevant to LOO as we understand it. The variant of LOO that GW discuss is at odds with a long literature on how to use LOO well; they focus on pathologizing a known and essentially unimportant property of the method; and they fail to discuss the most common issues that arise when using LOO for a real statistical analysis. In this discussion we try to discuss a number of concerns that everyone needs to think about before using LOO, reinterpret GW's examples, and try to explain the benefits of allowing for epistemological uncertainty when performing model selection.

# We need to abandon the idea that there is a device that will produce a single-number decision rule

The most pernicious idea in statistics is the idea that we can produce a single-number summary of any data set and this will be enough to make a decision. This view is perpetuated by GW's paper, which says that the only way that LOO can provide evidence for choosing a single model is for the pseudo-Bayes Factor to grow without bound (or, equivalently, that the model weight approaches 1) as sample size increases. This is not a good way to use LOO and fundamentally misjudges both its potential and its limitations as a tool for model selection and model criticism.

For a Bayesian model with $n$ data points $y_i \sim p(y | \theta)$ and parameters $\theta \sim p(\theta)$, LOO provides an estimate of the *expected log posterior predictive distribution*, $$\mathbb{E}_{\tilde{y}}\left(\log\left(\int\! p(\tilde{y}| \theta) p(\theta | y_\text{\rm all})\,d\theta\right)\right) \approx \frac{1}{n} \sum_{i=1}^n\log\left(\int\! p({y}_i| \theta) p(\theta | y_{-i})\,d\theta\right),$$ where the expectation is taken with respect to new data, $y_\text{\rm all}$ is all n observed data points, and $y_{-i}$ is all data points *except* the $i$th one.

There are two things to note here. Firstly, the computed LOO score is an empirical approximation to the expectation that we actually want to compute. This means that we must *never* consider it as a single number without also considering its inherent randomness. Considering both the LOO estimate and its empirical variance has been recommended since at least Breiman et al. (1984, chapter 11). The model with higher estimated performance should not be automatically selected; rather, uncertainty in the estimates should be considered (see also Vehtari, Gelman, and Gabry, 2017).

Since GW cite Vehtari, Gelman, and Gabry (2017) and Yao et al. (2018), we wish to emphasize that those papers of ours explicitly recommend computing the uncertainties due to not knowing the future data, and do not recommend LOO weights as used in the experiments by GW. Hence the claimed "limitations of Bayesian leave-one-out cross-validation" from GW do *not* apply to the version of Bayesian leave-one-out cross-validation that we actually recommend.

At the very least, we recommend that GW replace their pseudo-Bayes factors with the pseudo-BMA+ weights of Yao et al. (2018), which assume a normal distribution of the empirical mean of the log predictive densities and makes the appropriate modification to the model weights. Although this does not make a difference in their very specialized examples, we worry that people would see GW's work and assume that the method they describe is a "best practice" use of LOO rather than a version that happens to work in these extremely specialized situations. In Section we suggest that stacking (Yao et al., 2018) is a much better way to define model weights as it uses the LOO principle directly rather than simply defining ad hoc model weights.

The second thing to note is that the difference of LOO scores of two models can be decomposed into a sum of differences of log predictive densities for each left-out data point. This means that a two-way model comparison with LOO has more in common with a paired difference test than something like a Bayes factor. Vehtari, Gelman, and Gabry (2017) argue that plots of these individual differences can be useful, especially as an exploratory tool, and can bring attention to cases where the preferred model is affected by some unmodeled structure in the data. Considering this decomposition does not lead to a formal statistical test, but it can be useful when the aim of the analysis is to understand the data at hand rather than reach a "discovery" threshold.

This second property is extremely useful for applied work as it can be used to identify problems with model fit. As such, we recommend plotting the differences between the LOO log predictive densities even if LOO is not being used to perform model selection. Similarly, the $\hat{k}$ diagnostic produced by the `loo` package in R can be used as a generalized measure of leverage: if an observation has a $\hat{k}$ value larger than $0.7$ there is a very large difference between the predictive distribution that is conditioned on that observation and one that is not. These observations should be investigated as potentially problematic. Gabry et al. (2018) has a detailed discussion of how plots generated in from LOO cross validation are a useful part of the applied Bayesian workflow.

Hence, we strongly feel that the question raised by GW is not "Should we use LOO at all?", but rather, "Should we make model selection decisions based on LOO?".

# Closed thinking in an $M$-open world: Some tensions between reality and statistical convenience

Cross-validation and LOO have many limitations. Gronau and Wagenmakers (2018) focus on a variation of one known limitation: "in the idealized case where there exists a data set of infinite size that is perfectly consistent with the simple model $M_S$, LOO will nevertheless fail to strongly endorse $M_S$."

The strongest assumption in GW's paper is that the true model is one of the models under consideration, which Bernardo and Smith (1994) call the $M$-closed case. Cross-validation is usually recommended for the $M$-open case, where the true model is not included in the set of models compared. In real life, we are just about always in the $M$-open scenario; like nearly all models, $M$-closed is a simplification of reality. While it can sometimes be a useful simplification, we should never lose sight of the fact that methods that rely strongly on the $M$-closed assumption may be irretrievably biased.

Assuming that a model selection problem is $M$-closed allows the scientist to ignore epistemic uncertainty about their understanding of the process, and focus entirely on different models of aleatory uncertainty (the inherent randomness in the system). The extent to which this is a serviceable assumption will depend strongly on context of. With their exclusively focus on $M$-closed model selection, GW have severely limited the scope of their paper compared to the real applicability of LOO methods.

When we estimate the predictive performance of a model for future data that are not yet available, we need to integrate over the true future data distribution $p_T(y)$, which is typically unknown. Vehtari and Ojanen (2012) provide an extensive survey of alternative scenarios including also different cases of conditional modeling $p_T(y|x)$, with stochastic, fixed or deterministic $x$. In the $M$-closed case, we know that one of the models is $p_T(y)$, but we don't know which one. In that case, it makes sense to model $p_T(y)$ as a posterior weighted average of all models under consideration, that is, replace $p_T(y)$ with a Bayesian model average (BMA) as proposed by San Martini and Spezzaferri (1984). This approach has the same model-selection-consistent behavior as the Bayes factor. Thus if the assumption is that one of the models is true, the BMA reference model approach could be used, although asymptotic model selection consistency does not guarantee good finite sample performance, and BMA may have worse performance than LOO for small-to-moderate sample sizes even if one of the models is true.

On the other hand, in an $M$-open scenario, the BMA weights will still asymptotically converge to a single model (the one closest in Kullback-Leibler divergence to the data generating model), however this will no longer be the truth. In this case it is harder to make a case for using marginal likelihood-based weights.

Instead of using any explicit model or model averaging, LOO makes the assumption that future data can be drawn exchangeably from the same distribution as the observed data. This conditional independence structure is *vital* to the success of LOO methods as it allows us to re-use observed-data as pseudo-Monte Carlo draws from the future data distribution (Bernardo and Smith, 1994).

Cross-validation methods can have high variance when used on data sets that don't fully represent the data distribution. In such cases, some modeling assumptions about the future data distribution would be useful, in the same way that we use poststratification to adjust for extrapolations in sampling.

# If you use LOO better, you get more enlightening cases of failure

The main claims in GW is that when LOO is used for model selection (in a way that isn't in line with best practice) it is unable to select the true model and the LOO model weights are prior-dependent. Their claims are true to some extent, but we don't feel they do a good job at representing the behaviour of LOO or are representative of how we'd recommend these tools are used.

The inconsistency of LOO model weights reported by GW for the first two examples is entirely an artifact of their poor application of the LOO principle. To wit: if you want to construct LOO model weights, they should be based on the weighted predictive distribution. Yao et al. (2018) recommend using stacking weights, which choose the model weights that give the best expected log predictive density $$\max_{\substack{w: w_k\geq 0, \\ \sum_{k=1}^K w_k=1}} \frac{1}{n}\sum_{i=1}^n \log\left(\sum_{k=1}^K w_k p(y_i| y_{-i},M_k)\right),$$ where $M_k$ are the models under consideration. Yao et al. (2018) demonstrate with several $M$-open examples better performance of stacking compared to pseudo-BMA and BMA.

Looking at the examples in GW, it turns out that stacking can also work well in the $M$-closed case.

In the beta-Bernoulli examples, the stacking weights can be derived analytically. The weights are $1$ for $H_0$ and $0$ for $H_1$. To see this, we note that model $H_0$ makes a perfect prediction, so $p(y_i | y_{-1}, H_0)=1$ and model $H_1$ also has a constant predictive performance $p(y_i | y_{-i}, H_1)= (a+n-1)/(a+n-1+b) =   c < 1$. The stacking objective function is $n \log ( w_1 + w_2 c )$ which is maximized at $w_1= 1$ and $w_2=0$. This is independent of both the prior distribution and sample size $n$. Moreover, when model $H_0$ does better than model $H_1$ in every data point (example 1) or every pair of data point consistently, the stacking weight corresponding to $H_0$ has to be $1$.

The lack of dependence on $n$ may look suspicious. But intuitively when each data point (example 1) or each pair of data point (example 2) uniformly supports $H_0$ more than $H_1$, it does not require $n \rightarrow\infty$ to conclude that $H_0$ dominates $H_1$. This conclusion instead follows from the cross-validation assumption that the current data is representative of the future data.

In examples 2 and 3 and idealized data, stacking with symmetric-leave-two-out would also converge faster due to the contrived nature of the data generation process. This is not to claim that stacking weights will generally give asymptotically consistent model selection in $M$-closed problems, just that it does for these two cases due to the simple data chosen by GW.

The examples that GW chose are so specialized that the empirical mean of the log-predictive LOO densities has zero variance. This is a consequence of the data being both balanced and perfectly consistent with one of the hypotheses as well as the priors in examples 2 and 3 being centred on the mean of the data. These three properties imply that $p(y_i | y_{-i})$ is constant for all $i$ and hence Pseudo-BMA+ reduces to Pseudo-BMA used by GW.

The zero variance of the log predictive densities and the lack of dependence on $n$ also demonstrates a clear problem with the cross-validation assumption that we can use subsets of existing data to approximate the distribution of future data. The problem is that, for any finite $n$, the observed data sets in these examples are very strange (either all $1$s or all $\{0,1\}$ pairs) with zero variance. The LOO assumption will then assume that *all* future data is also like this with zero variance, which is problematic. Because even though we get consistency of the stacking weights in this case, if $H_0$ was $\theta =1-\epsilon$ for some small $\epsilon>0$, there would be a very high chance of observing a data set that was all $1$s even though the future data would include some values that are $0$. This means that LOO will be biased when the true model lies very close to an edge case.

The experiments in GW could be modified to better illustrate the limitation of LOO by considering cases where the complex model is correct. Consider the following alternative experiments where the more complex model is true:

-   In example 1, let the true $\theta_T= 1-\epsilon$, but for the simpler model keep $\theta_0=1$.

-   In example 2, let the true $\theta_T= \frac{1}{2}+\epsilon$, but for the simpler model keep $\theta_0=\frac{1}{2}$.

-   In example 3, let the true $\theta_T= \epsilon$, but for the simpler model keep $\theta_0=0$.

If we choose $\epsilon$ very small (but within the limits of the floating point accuracy for the experiments), we should see the same weights as in the original experiments as long as we observe the same data, and only when we occasionally observe one extra 0 in example 1, one extra 1 in example 2, or extra positive value in 3 we would see differences.

One way to illustrate the limitations of LOO (and cross-validation and information criteria in general), would be to compare the behaviour of the LOO stacking weights as $\epsilon$ moves from very small to much larger, plotting how large $\epsilon$ needs to be before we see that the more complex model is strongly favored.

We know that we can get the standard deviation of the parameter posterior smaller than $\epsilon$ with $\mathcal{O}(\epsilon^{-2})$ data points. However because, we need enough data to be confident about the distribution of future data, we expect to require far more data points than that for the stacking weights to confidently select the correct model in the above $\epsilon$-experiment. See the demonstration in Vehtari (2018a) and related results by Wang and Gelman (2014).

This is the price we pay for not trusting any model and thus not getting benefits of reduced variance through the proper modeling of the future data distribution! This variability makes cross-validation bad for model selection when the differences between the models are small, and it just gets worse in case of a large number of models with similar true performance (see, e.g., Piironen and Vehtari, 2017). Even with just two models to compare, cross-validation has also a limitation that the simple variance estimate tend to be optimistic (Bengio and Grandvalet, 2004).

# It really helps to look at the estimates

GW demonstrate that LOO is not consistent for model selection by showing three examples where the pseudo-BMA weight for the true model does not approach one as more data arrives. We feel there is a more nuanced way to look at this: namely the models that GW compare are nested, which means that the more complex model contains the simpler model. This means that the posterior predictive distribution from the more complex model will eventually be the same as the posterior predictive distribution constructed from the simpler model if the data is consistent with the simpler model. One way to interpret this phenomenon is that the two hypotheses are no longer distinct after we see enough data. Viewed this way, the prior dependence that GW demonstrated was just the prior choice slightly affecting the speed at which the two hypotheses are converging to each other.

A quick look at the parameter estimates in any of the three GW models shows that in each case the posterior for $\theta$ is rapidly converging on the specific value of $\theta$ (respectively $1$, $1/2$, and $0$) that defines the null model. For concreteness, let's consider example 2, where the difference between the models is that the null model sets $\theta=1/2$ while the alternative model uses $\theta \sim \mathcal{B}(a,b)$. We do not see a large difference between reporting "$H_0$ best models the data" or "$H_1$ best models the data and $\theta \approx 1/2$".

Our recommendation is that if the LOO comparison taking into account the uncertainties says that there is no clear winner, then neither of the models should be selected and instead model averaging or expansion should be used. If two models are being compared and they give similar predictions, then it shouldn't matter which one is used. Unlike GW, we typically prefer to use the more complex one.

There is some tension between our preference for predictive models and GW's implied interest in discriminative ones. GW's preference for parsimony appears to come in part from their desire to make statements like "x is not associated with y." Whereas, when we use our models to make statements about future data, we would prefer the more complex model in the nested model in order to be certain that uncertainties are not underestimated.

We don't recommend using the more complex model thoughtlessly. We require strict model checking and calibration of the complex model, and then proceed using projective variable selection (Goutis and Robert, 1998; Dupuis and Robert, 2003; Piironen and Vehtari, 2017; Piironen et al., 2018) to decide if some parts of the model can be dropped safely. Overfitting can be viewed as the use of too flat a prior distribution—but we and others use flat, or relatively flat, priors all the time, so overfitting is a real and persistent concern. Beyond concerns of stability in estimation and prediction, simpler models can also be motivated by a desire to reduce measurement costs in the future or to make it easier to explain the model for application experts, but this decision task then includes implicit or explicit costs for the measurements or complexity of the explanation. This final step is not considered by GW, but we consider it to be a *vital* part of any workflow that attempts model selection using LOO.

To clarify, we recommend using the encompassing model if (a) the model is being used to make predictions, (b) the encompassing model has passed model checking, and (c) the inference has passed diagnostic checks. From a Bayesian perspective, a larger and smaller model can be considered as two different priors on a single larger set of distributions, and there is no reason why the larger, less restrictive model, should be associated with flat priors for the additional parameters, even though this has often been the tradition in model comparison.

Bayesian theory says that we should integrate over the uncertainties. The encompassing model includes the submodels, and if the encompassing model has passed model checking, then the correct thing is to include all the models and integrate over the uncertainties. For the encompassing model to pass model checks, good priors will typically be required that appropriately penalize the complexity of the model (Simpson et al., 2018). If the models have similar cross-validation performance, the encompassing model is likely to have thicker tails of the predictive distribution, meaning it more cautious about rare events. We think this is good.

Here are some reasons why it is common in practice to favor more parsimonious models:

-   The maximum likelihood inference is common and it doesn't work well with more complex models. Favoring the simpler models is a kind of regularization.

-   Bad model misspecification. Bayesian inference can perform poorly if the model has major problems, and with complex models there are more possibilities for misspecifing the model and the misspecification even in one part can have strange effects in other parts.

-   Bad priors. Actually priors and models are inseparable, so this is kind of same as the previous one. It is more difficult to choose good priors for more complex models, because it's difficult for humans to think about high dimensional parameters and how they affect the predictions. Favoring the simpler models can avoid the need to think harder about priors.

We agree with Neal (1996), who wrote:

>  Sometimes a simple model will outperform a more complex model \[…\] Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well.

# What else can go wrong?

We can't let this opportunity pass to discuss the shortcoming of LOO methods without talking about the assumptions of LOO and how they can be violated. We've already mentioned that the assumption that the existing data is a good proxy for future observations can lead to problems. This is particularly true when data sets contain rare events. But there is one other major assumption that LOO methods (and cross validation methods generally) make: the assumption of conditional exchangeability between observations (or groups of observations).

If the distribution of the observations, once conditioned on all of the model parameters, is not exchangeable LOO methods will fail. This is because the fundamental idea of LOO is that $\frac{1}{n} \sum_{i=1}^n\log\left( \int\! p({y}_i| \theta) p(\theta | y_{-i})\,d\theta\right)$ is an unbiased approximation to the true expectation over unseen data $\mbox{E}_{\tilde{y}}\left( \log\left( \int\! p(\tilde{y}| \theta) p(\theta | _\text{\rm all})\,d\theta\right)\right).$ This only holds for models where observations are conditionally exchangeable.

This means that we need to be particularly careful when using cross-validation methods to assess models with temporal, spatial, or multilevel structure in the observations. Typically, we can alleviate this problem using a cross validation scheme that has been carefully designed to ensure that the blocks are (approximately) exchangeable (see, e.g., Roberts et al., 2017 and Bürkner et al, 2018). Some evidence of correlation may be visible by plotting the difference in log predictive densities as a function of time or space (much in the same way we can check linear regression residuals for serial autocorrelation), as well as checking the individual models for uncorrelated residuals. Vivar and Ferreira (2009) present a visual check for cross-validation predictions.

To summarize our claim, we say that cross validation methods will fail in all their tasks if the current data is not a good representation of all possible future data, or if the observations are not exchangeable conditional on the model parameters. While stacking weights will consistently identify the correct predictive distribution, it will not necessarily choose the simplest model when multiple models asymptotically give the same predictive distribution. In the case of nested models, parsimony can be restored using projective prediction methods, however for non-nested models we don't know of a general solution.

# Can you do open science with $M$-closed tools?

One of the great joys of writing a discussion is that we can pose a very difficult question that we have no real intention of answering. The question that is well worth pondering is the extent to which our chosen statistical tools influence how scientific decisions are made. And it's relevant in this context because of a key difference between model selection tools based on LOO and tools based on marginal likelihoods is what happens when none of the models could reasonably generate the data.

In this context, marginal likelihood-based model selection tools will, as the amount of data increases, choose the model that best represents the data, even if it doesn't represent the data particularly well. LOO-based methods, on the other hand, are quite comfortable expressing that they can not determine a single model that should be selected. To put it more bluntly, marginal likelihood will always confidently select the wrong model, while LOO is able to express that no one model is correct.

We leave it for each individual statistician to work out how the shortcomings of marginal likelihood-based model selection balance with the shortcomings of cross-validation methods. There is no simple answer.

# References

Bengio, Y., and Grandvalet, Y. (2004). No unbiased estimator of the variance of $k$-fold cross-validation. *Journal of Machine Learning Research* **5**, 1089–1105.

Bernardo, J. M., and Smith, A. F. M. (1994). *Bayesian Theory*. New York: Wiley.

Breiman, L., Friedman, J., Olshen, R., and Stone, C. (1984). *Classification and Regression Trees*. London: Chapman and Hall.

Bürkner, P., Vehtari, A., and Gabry, J. (2018). Approximate leave-future-out cross-validation for time series models. <http://mc-stan.org/loo/articles/loo2-lfo.html>

Dupuis, J. A., and Robert, C. P. (2003). Variable selection in qualitative models via an entropic explanatory power. *Journal of Statistical Planning and Inference* **111**, 77–-94.

Gabry, J., Simpson, D. P., Vehtari, A., Betancourt, M., and Gelman, A. (2018). Visualization in Bayesian workflow (with discussion). *Journal of the Royal Statistical Society A*.

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). *Bayesian Data Analysis*, third edition. London: CRC Press.

Gelman, A., Vehtari, A., and Hwang, J. (2014). Understanding predictive information criteria for Bayesian models. *Statistics and Computing* **24**, 997–1016.

Goutis, C., and Robert, C. P. (1998). Model choice in generalised linear models: A Bayesian approach via Kullback–Leibler projections. *Biometrika* **85**, 29-–37.

Gronau, Q., and Wagenmakers, E. J. (2018). Limitations of Bayesian leave-one-out cross-validation for model selection. *Computational Brain & Behavior*.

Neal, R. M. (1996). *Bayesian Learning for Neural Networks*. Lecture Notes in Statistics No. 118. New York: Springer-Verlag.

Piironen, J., and Vehtari, A. (2017). Comparison of Bayesian predictive methods for model selection. *Statistics and Computing* **27**, 711–735.

Piironen, J, Paasiniemi, M., and Vehtari, A. (2018). Projective inference in high-dimensional problems: Prediction and feature selection. <https://arxiv.org/abs/1810.02406>

Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., and Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. *Ecography* **40**, 913–929.

San Martini, A., and Spezzaferri, F. (1984). A predictive model selection criterion. *Journal of the Royal Statistical Society B* **46**, 296–303.

Simpson, D., Rue, H., Riebler, A., Martins, T. G., and Sørbye, S. H. (2017). Penalising model component complexity: A principled, practical approach to constructing priors. *Statistical Science* **32**, 1–28.

Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. *Statistics and Computing* **27**, 1413–1432.

Vehtari, A. (2018a). Bayesian data analysis—beta blocker cross-validation demo. <https://avehtari.github.io/modelselection/betablockers.html>

Vehtari, A. (2018b). Model assessment, selection and inference after selection. <https://avehtari.github.io/modelselection/>

Vehtari, A., and Gabry, J. (2018). Using the loo package. <https://cran.r-project.org/web/packages/loo/vignettes/loo2-example.html>

Vehtari, A., and Ojanen, J. (2012). A survey of Bayesian predictive methods for model assessment, selection and comparison. *Statistics Surveys* **6**, 142–228.

Vivar, J. C., and Ferreira, M. A. R. (2009). Spatiotemporal models for Gaussian areal data. *Journal of Computational and Graphical Statistics* **18**, 658–674.

Wang, W., and Gelman, A. (2014). Difficulty of selecting among multilevel models using predictive accuracy. *Statistics and Its Interface* **7**.

Yao., Y., Vehtari, A., Simpson, D. P., and Gelman, A. (2018). Using stacking to average Bayesian predictive distributions (with discussion). *Bayesian Analysis* **13**, 917–1003.

[^1]: Department of Computer Science, Aalto University, Finland

[^2]: Department of Statisitcs, University of Toronto

[^3]: Department of Statistics, Columbia University.

[^4]: Department of Statistics and Department of Political Science, Columbia University.

[^5]: Discussion of Gronau and Wagenmakers (2018) for *Computational Brain & Behavior*. We thank Paul Bürkner for helpful comments, and the Academy of Finland, the Canadian Natural Sciences and Engineering Research Council, the U.S. Office of Naval Research, National Science Foundation, and Defense Advanced Research Projects Administration for partial support of this work.