abstract: Antibody therapeutics and vaccines are among our last resort to end the raging COVID-19 pandemic. They, however, are prone to over 1,800 mutations uncovered by a [Mutation Tracker](https://users.math.msu.edu/users/weig/SARS-CoV-2_Mutation_Tracker.html). It is urgent to understand how vaccines and antibodies in the development would be impacted by mutations. In this work, we first study the mechanism, frequency, and ratio of mutations on the spike (S) protein, which is the common target of most COVID-19 vaccines and antibody therapies. Additionally, we build a library of antibody structures and analyze their 2D and 3D characteristics. Moreover, we predict the mutation-induced binding free energy (BFE) changes for the complexes of S protein and antibodies or ACE2. By integrating genetics, biophysics, deep learning, and algebraic topology, we deduce that some of the mutations such as M153I, S254F, and S255F may weaken the binding of S protein and antibodies, and potentially disrupt the efficacy and reliability of antibody therapies and vaccines in the development. We provide a strategy to prioritize the selection of mutations for designing vaccines or antibody cocktails.
author: Jiahui Chen$^1$, Kaifu Gao$^{1,\dagger}$, Rui Wang$^{1,\dagger}$, and Guo-Wei Wei$^{1,2,3}$[^1]
$^1$ Department of Mathematics,
Michigan State University, MI 48824, USA.
$^2$ Department of Electrical and Computer Engineering,
Michigan State University, MI 48824, USA.
$^3$ Department of Biochemistry and Molecular Biology,
Michigan State University, MI 48824, USA.
$^\dagger$First three authors contributed equally.
date: 2024-10-03
title: Prediction and mitigation of mutation threats to COVID-19 vaccines and antibody therapies

Key words: vaccine, antibody, mutation, protein-protein interaction, binding affinity change, algebraic topology, deep learning.

# Introduction

The expeditious spread of coronavirus disease 2019 (COVID-19) pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has led to 34,667,658 confirmed cases and 1,030,040 fatalities as of September 30, 2020. In the 21st century, three major outbreaks of deadly pneumonia are caused by $\beta$-coronaviruses: SARS-CoV (2002), Middle East respiratory syndrome coronavirus (MERS-CoV) (2012), and SARS-CoV-2 (2019) . Similar to SARS-CoV and MERS-CoV, SARS-CoV-2 causes respiratory infections, and the transmission of viruses occurs among family members or in healthcare settings at the early stages of the outbreak. However, SARS-CoV-2 has an unprecedentedly high infection rate compared to SARS-CoV and MERS-CoV . Considering the high infection rate, high prevalence rate, long incubation period , asymptomatic transmission , and potential seasonal pattern of COVID-19, the development of specific antiviral drugs, antibody therapies, and effective vaccines is of paramount importance. Traditional drug discovery takes more than ten years, on average, to bring a new drug on the market . However, developing potent SARS-CoV-2 specified antibodies and vaccines is a relatively more efficient and less time-consuming strategy to combat COVID-19 for the ongoing pandemic . Antibody therapies and vaccines depend on the host immune system. Recently studies have been working on the host-pathogen interaction, host immune responses, and the pathogen immune evasion strategies , which provide insight into understanding the mechanism of antibody therapies and vaccine development.

The immune system is a host defense system that protects the host from pathogenic microbes, eliminates toxic or allergenic substances, and responds to an invading pathogen . It has innate immune system and adaptive immune system as two major subsystems. The innate system provides an immediate but non-specific response, whereas the adaptive immune system provides a highly specific and effective immune response. Once the pathogen breaches the first physical barriers, such as epithelial cell layers, secreted mucus layer, mucous membranes, the innate system will be triggered to identify pathogens by pattern recognition receptors (PRRs), which is expressed on dendritic cells, macrophages, or neutrophils . Specifically, PPRs identify pathogen-associated molecular patterns (PAMPs) located on pathogens and then activate complex signaling pathways that introduce inflammatory responses mediated by various cytokines and chemokines, which promote the eradication of the pathogen . Notably, the transmission of SARS-CoV-2 even occurs in asymptomatic infected individuals, which may delay the early response of the innate immune response . Another important line of host defense is the adaptive immune system. B lymphocytes (B cells) and T lymphocytes (T cells) are special types of leukocytes that are the acknowledged cellular pillars of the adaptive immune system . Two major subtypes of T cells are involved in the cell-mediated immune response: killer T cells (CD8+ T cells) and helper T cells (CD4+ cells). The killer T cells eradicate cells invaded by pathogens with the help of major histocompatibility complex (MHC) class I. MHC class I molecules are expressed on the surface of all nucleated cells . The nucleated cells will firstly degrade foreign proteins via antigen processing when viruses infect them. Then, the peptide fragments will be presented by MHC Class I, which will activate killer T cells to eliminate these infected cells by releasing cytotoxins . Similarly, helper T cells cooperate with MHC Class II, a type of MHC molecules that are constitutively expressed on antigen-presenting cells, such as macrophages, dendritic cells, monocytes, and B cells . Helper T cells express T cell receptors (TCR) to recognize antigen bound to MHC class II molecules. However, helper T cells do not have cytotoxic activity. Therefore, they can not kill infected cells directly. Instead, the activated helper T cells will release cytokines to enhance the microbicidal function of macrophages and the activity of killer T cells . Notably, an unbalanced response can result in a "cytokine storm," which is the main cause of the fatality of COVID-19 patients . Correspondingly, a B cell involves in humoral immune response and identifies pathogens by binding to foreign antigens with its B cell receptors (BCRs) located on its surface. The antigens that are recognized by antibodies will be degraded to petites in B cells and displayed by MHC class II molecules. As mentioned above, helper T cells can recognize the signal provided by MHC class II and upregulate the expression of CD40 ligand, which provides extra stimulation signals to activate antibody-producing B cells , rendering millions of copies of antibodies (Ab) that recognize the specific antigen. Additionally, when the antigen first enters the body, the T cells and B cells will be activated, and some of them will be differentiated to long-lived memory cells, such as memory T cells and memory B cells. These long-lived memory cells will play a role in quickly and specifically recognizing and eliminating a specific antigen that encountered the host and initiated a corresponding immune response in the future . The vaccination mechanism is to stimulate the primary immune response of the human body, which will activate T cells and B cells to generate the antibodies and long-lived memory cells that prevent infectious diseases, which is one of the most effective and economical means for combating with COVID-19 at this stage.

As mentioned above, secreted by B cells of the adaptive immune system, antibodies can recognize and bind to specific antigens. Conventional antibodies (immunoglobulins) are Y-shaped molecules that have two light chains and two heavy chains . Each light chain is connected to the heavy chain via a disulfide bond, and heavy chains are connected through two disulfide bonds in the mid-region known as the hinge region. Each light and heavy chain contain two distinct regions: constant regions (stem of the Y) and variable regions ("arms" of the Y) . An antibody binds the antigenic determinant (also called epitope) through the variable regions in the tips of heavy and light chains. There is an enormous amount of diversity in the variable regions. Therefore, different antibodies can recognize many different types of antigenic epitopes. To be specific, there are three complementarity determining regions (CDRs) that are arranged non-consecutively in the tips of each variable region. CDRs generate most of the diversities between antibodies, which determine the specificity of individuals of antibodies. In addition to conventional antibodies, camelids also produce heavy-chain-only antibodies (HCAbs). HCAbs, also referred to as nanobodies, or VHHs, contain a single variable domain (VHH) that makes up the equivalent antigen-binding fragment (Fab) of conventional immunoglobulin G (IgG) antibodies . This single variable domain typically can acquire affinity and specificity for antigens comparable to conventional antibodies. Nanobodies can easily be constructed into multivalent formats and have higher thermal stability and chemostability than most antibodies do . Another advantage of nanobodies is that they are less susceptible to steric hindrances than large conventional antibodies .

Considering the broad specificity of antibodies, seeking potential antibody therapies has become one of the most feasible strategies to fight against SARS-CoV-2. In general, antibody therapy is a form of immunotherapy that uses monoclonal antibodies (mAb) to target pathogenic proteins. The binding of antibody and pathogenic antigen can facilitate either immune response, direct neutralization, radioactive treatment, the release of toxic agents, or cytokine steam inhibition (aka immune checkpoint therapy). The SARS-CoV-2 entry of a human cell facilitated by the process of a series of interactions between its spike (S) protein and the host receptor angiotensin-converting enzyme 2 (ACE2), primed by host transmembrane protease, serine 2 (TMPRSS2) . As such, most COVID-19 antibody therapeutic developments focus on the SARS-CoV-2 spike protein antibodies that were initially generated from patient immune response and T-cell pathway inhibitors that block T-cell responses. A large number of antibody therapeutic drugs are in clinical trials. 28 S protein antibody structures are available in the Protein Data Bank (PDB), offering a great resource for mechanistic analysis and biophysical studies.

Currently, most antibody therapy developments focus on the use of antibodies isolated from patient convalescent plasma to directly neutralize SARS-CoV-2 , although there are efforts to alleviate cytokine storm. A more effective and economical means to fight against SARS-CoV-2 is vaccine , which is the most anticipated approach for preventing the COVID-19 pandemic. A vaccine is designed to stimulate effective host immune responses and provide active acquired immunity by exploiting the body's immune system, including the production of antibodies, which is made of an antigenic agent that resembles a disease-causing microorganism, or surface protein, or genetic material that is needed to generate the surface protein. For SARS-CoV-2, the first choice of surface proteins is the spike protein. There are four types of COVID-19 vaccines, as shown in . 1) Virus vaccines use the virus itself, in a weakened or inactivated form. 2) Viral-vector vaccines are designed to genetically engineer a weakened virus, such as measles or adenovirus, to produce coronavirus S proteins in the body. Both replicating and non-replicating viral-vector vaccines are being studied now. 3) Nucleic-acid vaccines use DNA or mRNA to produce SASR-CoV-2 S proteins inside host cells to stimulate the immune response. 4) Protein-based vaccines are designed to directly inject coronavirus proteins, such as S protein or membrane (M) protein, or their fragments, into the body. Both protein subunits and viral-like particles (VLPs) are under development for COVID-19 . Among these technologies, nucleic-acid vaccines are safe and relatively easy to develop . However, they have not been approved for any human usage before.

However, the general population's safety concerns are the major factors that hinder the rapid approval of vaccines and antibody therapies. A major potential challenge is an antibody-dependent enhancement, in which the binding of a virus to suboptimal antibodies enhances its entry into host cells. All vaccine and antibody therapeutic developments are currently based on the reference viral genome reported on January 5, 2020 . SARS-CoV-2 belongs to the coronaviridae family and the Nidovirales order, which has been shown to have a genetic proofreading mechanism regulated by non-structure protein 14 (NSP14) in synergy with NSP12, i.e., RNA-dependent RNA polymerase (RdRp) . Therefore, SARS-CoV-2 has a higher fidelity in its transcription and replication process than other single-stranded RNA viruses, such as the flu virus and HIV. Even though the S protein of SARS-CoV-2 has been undergoing many mutations, as reported in . As of September 30, a total of 1811 mutations on the S protein has been detected on 63556 complete SARS-CoV-2 genome sequences. Therefore, it is of paramount importance to establish a reliable computational paradigm to predict and mitigate the impact of SARS-CoV-2 mutations on vaccines and antibody therapies. Moreover, the efficacy of a given COVID-19 vaccine depends on many factors, including SARS-CoV-2 biological properties associated with the vaccine, mutation impacts, vaccination schedule (dose and frequency), idiosyncratic response, assorted factors such as ethnicity, age, gender, or genetic predisposition. The effect of COVID-19 vaccination also depends on the fraction of the population who accept vaccines. It is essentially unknown at this moment how these factors will unfold for COVID-19 vaccines.

It is no doubt that any preparation that leads to an improvement in the COVID-19 vaccination effect will be of tremendous significance to human health and the world economy. Therefore, in this work, we integrate genetic analysis and computational biophysics, including artificial intelligence (AI), as well as additional enhancement from advanced mathematics to predict and mitigate mutation threats to COVID-19 vaccines and antibody therapies. We perform single nucleotide polymorphism (SNP) calling to identify SARS-CoV-2 mutations. For mutations on the S protein, we analyze their mechanism , frequency, ratio, and secondary structural traits. We construct a library of all existing antibody structures from the Protein Data Bank (PDB) and analyze their two-dimensional (2D) and three-dimensional (3D) characteristics. We further predict the mutation-induced binding affinity changes of antibody and S protein complexes using a topology-based network tree (TopNetTree) , which is a state-of-the-art model that integrates deep learning and algebraic topology . After identifying mutations that are potentially disruptive to antibody and S protein interactions, we further infer their threats to vaccines based on antibody binding site analysis, mutation-induced disruptive free energy, and mutation occurrence frequency. We combine frequency and free energy change to prioritize mutation threats and guild the development of future vaccines and antibody therapies.

# Mutations on the Spike protein

As a fundamental biological process, mutagenesis changes the organism's genetic information and servers as a primary source for many kinds of cancer and heritable diseases, which is a driving force for evolution . Generally speaking, virus mutations are introduced by natural selection, replication mechanism, cellular environment, polymerase fidelity, gene editing, random genetic drift, gene editing, recent epidemiology features, host immune responses, etc . Notably, understanding how mutations have changed the SARS-CoV-2 structure, function, infectivity, activity, and virulence is of great importance for coming up with life-saving strategies in virus control, containment, prevention, and medication, especially in the antibodies and vaccines development. Genome sequencing, SNP calling, and phenotyping provide an efficient means to parse mutations from a large number of viral samples (see the Supporting material (S1)). In this work, we retrieved over 60,000 complete SARS-CoV-2 genome sequences from the GISAID database and created a real-time interactive [SARS-CoV-2 Mutation Tracker](https://users.math.msu.edu/users/weig/SARS-CoV-2_Mutation_Tracker.html)(<https://users.math.msu.edu/users/weig/SARS-CoV-2_Mutation_Tracker.html>) to report over 18,000 single mutations along with its mutation frequency on SARS-CoV-2 as of September 30. is a screenshot of our online Mutation Tracker. It describes the distribution of mutations on the complete coding region of SARS-CoV-2. The $y$-axis shows the natural log frequency for each mutation at a specific position. A reader can download the detailed mutation SNP information from our Mutation Tracker website.

As mentioned before, the S protein has become the first choice for antibody and vaccine development. Among 63,556 complete genome sequences, 1811 unique single mutations are detected on the S protein, and the $h$-index of S protein is 52 The number of unique mutations (N$_{\text{U}}$) is determined by counting the same type of mutations in different genome isolates only once, whereas the number of non-unique mutations (N$_{\text{NU}}$, i.e., frequency) is calculated by counting the same type of mutations in different genome isolates repeatedly. lists the distribution of 12 SNP types among unique and non-unique mutations on the S protein of SARS-CoV-2 worldwide. It can be seen that C$>$T and A$>$G are the two dominated SNP types, which may be due to the innate host immune response via APOBEC and ADAR gene editing .

| SNP Type | Mutation Type | N$_{\text{U}}$ | N$_{\text{NU}}$ | R$_{\text{U}}$ | R$_{\text{NU}}$ | SNP Type | Mutation Type | N$_{\text{U}}$ | N$_{\text{NU}}$ | R$_{\text{U}}$ | R$_{\text{NU}}$ |  |  |  |  |  |
|:--:|:---|:--:|:--:|:--:|:--:|:--:|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| A$>$T | Transversion | 100 | 351 | 5.52% | 0.56% | C$>$T | Transition | 332 | 8499 | 18.33% | 13.44% |  |  |  |  |  |
| A$>$C | Transversion | 76 | 154 | 4.20% | 0.24% | C$>$A | Transversion | 93 | 401 | 5.14% | 0.63% |  |  |  |  |  |
| A$>$G | Transition | 257 | 41033 | 14.19% | 64.89% | C$>$G | Transversion | 25 | 50 | 1.38% | 0.08% |  |  |  |  |  |
| T$>$A | Transversion | 43 | 10 | 2.37% | 0.16% | G$>$T | Transversion | 238 | 3506 | 13.14% | 5.54% |  |  |  |  |  |
| T$>$C | Transition | 315 | 1145 | 17.39% | 2.29% | G$>$C | Transversion | 81 | 386 | 4.47% | 0.61% |  |  |  |  |  |
| T$>$G | Transversion | 65 | 135 | 3.59% | 0.21% | G$>$A | Transition | 186 | 7162 | 10.27% | 11.33% |  |  |  |  |  |

The distribution of 12 SNP types among 1811 unique mutations and 63232 non-unique mutations on the S gene of SARS-CoV-2 worldwide. N$_{\text{U}}$ is the number of unique mutations and N$_{\text{NU}}$ is the number of non-unique mutations. R$_{\text{U}}$ and R$_{\text{U}}$ represent the ratios of 12 SNP types among unique and non-unique mutations.

Moreover, 133 non-degenerated mutations occurred on the S protein receptor-binding domain (RBD), which are relevant to the binding of SARS-CoV-2 S protein and most antibodies as well as ACE2. Additionally, 59 mutations occurred on the S protein domain (residue id: 14 to 226) are relevant to the binding of another antibody (4A8) and SARS-CoV-2 S protein.

Furthermore, since antibody CDRs are random coils, the complementary antigen-binding domains must involve random coils as well. lists the statistics of non-degenerate mutations on the secondary structures of SARS-CoV-2 S protein. Here, the secondary structures are mostly extracted from the crystal structure of 7C2L, and the missing residues are predicted by RaptorX-Property . We can see that for both unique and non-unique cases, the average mutation rates on the random coils of the S protein have the highest values. Particularly, the 23403A$>$G-(D614G) mutation on the random coils has the highest frequency of 39967. If we do not consider the 23403A$>$G-(D614G) mutations, then the unique and non-unique average rates on the random coils of S protein still have the highest values (0.98 and 10.83), indicating that mutations are more likely to occur on the random coils. Consequently, the natural selection of mutations may tend to disrupt antibodies.

| Secondary structure | Length | N$_{\text{U}}$ | N$_{\text{NU}}$ | AR$_{\text{U}}$ | AR$_{\text{NU}}$ |  |  |  |  |  |  |  |  |  |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| Helix | 249 | 162 | 1254 | 0.65 | 5.04 |  |  |  |  |  |  |  |  |  |
| Sheet | 276 | 227 | 2010 | 0.82 | 7.28 |  |  |  |  |  |  |  |  |  |
| Random coils | 748 | 734 | 48054 | **0.98** | **64.24** |  |  |  |  |  |  |  |  |  |
| Whole Spike | 1273 | 1123 | 51318 | 0.88 | 40.31 |  |  |  |  |  |  |  |  |  |

The statistics of non-degenerate mutations on the secondary structure of SARS-CoV-2 S protein. The unique and non-unique mutations are considered in the calculation. N$_{\text{U}}$, N$_{\text{NU}}$, AR$_{\text{U}}$, AR$_{\text{NU}}$ represent the number of unique mutations, the number of non-unique mutations, the average rate of unique mutations, and the average rate of non-unique mutations on the secondary structure of S protein, respectively. Here, the secondary structure is mostly extracted from the crystal structure of 7C2L, the missing residues are predicted by RaptorX-Property.

# SARS-CoV-2 antibodies

We construct a SARS-CoV-2 antibody library of 28 3D antibody structures deposited in the PDB. Among them, the binding sites of 27 antibodies are on the RBD of the S protein. While another antibody, 4A8 , has a distinguished binding domain. Additionally, MR17-K99Y is a mutant of antibody MR17 .

## 3D antibody structure alignment on the S protein

We align 26 antibody structures, excluding MR17-K99Y, with SARS-CoV-2 S protein in Figure . ACE2 is included as a reference. Clearly, except for antibody 4A8, all other 26 structures bind to the S protein RBD. It is interesting to note that 4A8 locates on a different domain. The PDB IDs of these complexes can be found in Figure .

Figure reveals, except for S309, CR3022, EY6A, and 4A8, all the other 23 antibodies have their binding sites spatially clashing with that of ACE2. Notably, the paratope of H014 does not overlap with that of ACE2 directly, but in terms of 3D structures, their binding sites still overlap. This suggests that the bindings of 23 antibodies are in direct competition with that of ACE2. Theoretically, this direct competition reduces the viral infection rate. For such an antibody with strong binding ability, it will directly neutralize SARS-CoV-2 without the need of antibody-dependent cell cytotoxicity (ADCC), antibody-dependent cellular phagocytosis (ADCP), or other immune mechanisms.

The paratopes of S309, CR3022, and EY6A on the RBD are away from that of ACE2, leading to the absence of binding competition . One study shows that the ADCC and ADCP mechanisms contribute to the viral control conducted by S309 in infected individuals . For CR3022, one research indicates that it neutralizes the virus in a synergistic fashion . For EY6A, the hypothesis is that the binding of EY6A could inhibit the glycosylation of ACE2 . A more radical example is 4A8 , it binds to the N-terminal domain (NTD) of the S protein (Figure (h)), which is quite far from the RBD, it is speculated that 4A8 may neutralize SARS-CoV-2 by restraining the conformational changes of the S protein, which is very important for the SARS-CoV-2 cell entry . Any antibody or drug that can inhibit serine protease TMPRSS2 priming of the S protein priming can effectively stop the viral cell entry .

## 2D residue contacts between antibodies and the S protein

Figure provides a visual illustration of antibody and ACE2 competitions. It remains to know in the residue detail what has happened to these competitions. To better understand the antibody and S protein interactions, we study the residue contacts between antibodies and the S protein. We include the ACE2 as a reference but excluding antibodies 4A8 and MR17-K99Y.

In Figure , the paratopes of 26 antibodies and ACE2 were aligned on the S protein RBD 2D sequence, and their contact regions are highlighted. From the figure, one can see that, except for H014, S309, CR3022, and EY6A, all the other 22 antibodies have their antigenic epitopes overlapping with the ACE2 RBD, especially on the residues from 486 to 505 of the SARS-CoV-2 RBD. Therefore, these 22 antibodies competitively bind against ACE2 as revealed in Figure .

## Antibody sequence alignment and similarity analysis

The next question is whether there is any connection or similarity between the antibody paratopes in our library, particularly for those antibodies that share the same binding sites. To better understand this perspective, we carry out multiple sequence alignment (MSA) to further study the similarity and difference among existing antibodies. Many antibodies are very similar to each other and can be described in a few groups. The first group includes BD-629, CC12.3, COVA2-04, CV30, CC12.1, B38, BD-236, BD-604, EY6A, and REGN10933, as well as CB6. Their identity scores to CB6 are 87.39 %, 88.51 %, 87.76 %, 87.02 %, 93.36 %, 92.27 %, 91.53 %, 91.51 %, 89.80 %, and 87.73 %, respectively. The second group is composed of H11-H4, and Nb, as well as H11-D4, and their identity scores to H11-D4 are 96.27 % and 93.28 %, respectively. These are nanobodies. The third group has REGN10987, and COVA2-39, as well as P2B-2F6, and their identity scores to P2B-2F6 are 80.05 %, 88.44 %, respectively. The alignment plots are in the Supporting material (S2).

Therefore, multiple sequence alignment suggests that the paratopes of the antibodies BD-629, CB6, COVA2-04, CV30, CC12.1, CC12.3, C105, BD-604, BD-236, and B38 are almost identical. Similarly, the paratopes of the antibodies H11-H4, H11-D4, Nb are highly consistent. So are the antibodies REGN10987, COVA2-39, and P2B-2F6. The above similarity indicates that the adaptive immune systems of individuals have a common way to generate antibodies. On the other hand, the existence of three distinct groups, as well as antibody 4A8 suggests the diversity in the immune response. Note that we have also included ACE2 in our MSA as a reference but none of the existing antibodies is similar to ACE2, because they were created from entirely different mechanisms.

# Mutation impacts on SARS-CoV-2 antibodies

To investigate the influences of existing S protein mutations on the binding free energy (BFE) of S protein and antibodies, we consider 133 mutations occurred on the S protein RBD which are relevant to the binding of SARS-CoV-2 S protein and antibodies as well as ACE2. Additionally, 59 mutations occurred on the NTD of the S protein (residue id: 14 to 226) which are relevant to the binding of SARS-COV-2 S protein and antibody 4A8 (PDB: 7C2L). We predict the free energy changes following existing mutations using our TopNetTree model . The RBD mutations are computed which are in the distance of 10 Å to antibodies. Our predictions are built from the X-ray crystal structure of SARS-CoV-2 S protein and ACE2 (PDB 6M0J) , and various antibodies (PDBs 6WPS, 6XC2, 6XC3, 6XC4, 6XC7, 6XCM, 6XDG, 6XE1, 6XEY, 6YLA, 6YZ5, 6Z2M, 6ZBP, 6ZCZ, 6ZER, 7BWJ, 7BYR, 7BZ5, 7C01, 7C2L, 7C8V, 7C8W, 7CAH, 7CAN, 7CH4, 7CH5, 7CHB, 7CHE, 7CHF, 7CHH, 7JMO, and 7JMP). The BFE change following mutation ($\Delta\Delta G$) is defined as the subtraction of the BFE of the mutant type from the BFE of the wild type, $\Delta\Delta G=\Delta G_{\rm W} - \Delta G_{\rm M}$ where $\Delta G_{\rm W}$ is the BFE of the wild type and $\Delta G_{\rm M}$ is the BFE of mutant type. Therefore, a negative BFE change means that the mutation decreases affinities, making the protein-protein interaction less stable.

## Single antibody-S protein complex analysis

We first present the BFE changes $\Delta\Delta G$ of SARS-CoV-2 S protein binding domain with antibody 4A8 in Figure , which is the only complex that is not on the RBD in our collections of S protein and antibody complexes. Most mutations have small changes in their binding free energies, while some of them have large changes. Notably, 25 out of 59 mutations on the binding domain have positive BFE changes, which means that the mutations increase affinities and would make protein-protein interactions more stable. However, the majority (58%) of mutations have negative BFE changes, including high-frequency mutations, M153I, S254F, and S255F. It is also noted that many mutations on the binding domain, such as G142D and K147N, have significant negative free energy changes. The mutations on the binding domain with negative binding affinities reveal that the binding of antibody 4A8 and S protein will be potentially disrupted.

Next, we study the BFE changes $\Delta\Delta G$ induced by 39 mutations on the SARS-CoV-2 S protein RBD for the antibody Fab 2-4 (PDB: 6XEY) in Figure . Most mutations induce small changes in the thee binding free energies, while mutations, G485R and S494L, have large negative BFE changes. Overall, 27 out of 39 mutations on the RBD lead to negative BFE changes, which means 69% of mutations will potentially weaken the binding between antibody Fab 2-4 and S protein. Particularly, mutation S477N on the RBD induces a negative BFE change with a high frequency of 3269. While some mutations leading to positive BFE changes, more mutations induce negative BFE changes with large magnitude. Antibody Fab 2-4 shares a similar binding domain with ACE2 and thus is a potential candidate for the direct neutralization of SARS-CoV-2. However, BFE change predictions indicate that the mutations on S protein weaken the Fab 2-4 binding with S protein and make it less competitive with ACE2.

In Figure , we illustrate antibody B38 (PDB: 7C8W), which shares the binding domain with ACE2 as well. One can notice that only four mutations, R403S, F490S, L455F, and S494L, have the magnitude of BFE changes larger than 1 kcal/mol and all are negative BFE changes. The rest mutations have a small magnitude of changes. Mutation V483A has a frequency of 31 and small positive BFE changes. Interestingly, mutation S494L induces large BFE changes for antibodies B39 and Fab 2-4. Antibody B39 will reduce its competitiveness with ACE2 if mutations R403S, F490S, L455F, and S494L become dominant.

Finally, we consider the BFE change predictions for antibody S309 and S protein complex, whose receptor binding motif (RBM) does not overlap with the RBM of ACE2. The BFE changes induced by 30 mutations are predicted. Among them, 11 changes are positive. Similar to the aforementioned antibodies, most of the mutations lead to small changes in their binding affinity magnitude but three mutations, T345S, V395I, and K444N, induce large negative changes. The binding of antibody S309 might be disrupted, considering that a majority of mutations induce negative BFE changes with large magnitude. While antibodies play a variety of functions in the human immune system such as neutralization of infection, phagocytosis, antibody-dependent cellular cytotoxicity, etc., their binding with antigens is crucial for these functions. Our analysis of BFE changes following mutations on S protein suggests that some antibodies will be less affected by mutations, which is important for developing vaccine and antibody therapies. The BFE change analysis of other antibodies is described in the Supporting material (S3).

## Mutation impact library

In this section, we build a library of mutation-induced BFE changes for all mutations and all antibodies. In principle, we could create a library of all possible mutations for all antibodies, as we did for ACE2 . Here, we limit our effort to all existing mutations. Antibody 4A8 on the NTD has been discussed above. We consider antibodies on the RBD.

Based on our earlier analysis, three types of SARS-CoV-2 S protein secondary structural residues have different mutation rates. Among them, the random coils are major components of the RDB and the NTD, as shown in Fig. . Therefore, mutations on the RBD are split into three categories based on their locations in secondary structures helix, sheet, and coil. In Figure , we present the BFE changes for ACE2 and antibodies induced by mutations on helix residues of the S protein RBD. The frequency for each mutation is also presented. Most mutations on helix residues lead to positive BFE changes (green squares), whereas some mutations induce negative BFE changes (pink squares). The N439K mutation having the largest frequency, 106, shows mild BFE changes on ACE2 and antibodies. Mutations K417N and Y505C induce positive BFE changes on most of ACE2 and antibodies. Especially, antibodies C105 and BD-604 have larger BFE changes than ACE2, which indicates that they are stronger competitive than ACE2. Antibody CB6 may be potentially a good therapeutic candidate as its BFE changes are positive following all mutations, but this needs to be confirmed by other mutations on the coil and sheet residues.

In Figure , we present the BFE changes for ACE2 and antibodies along with frequencies on mutations of sheet residues of the S protein RBD. The mutation R403S has a large variance of the BFE changes such that both positive and negative changes occurred on antibodies and ACE2. Clearly, antibodies BD23, BD38, CB6, MR17, and MR17-K99Y lead to negative BFE changes on mutations of RBD sheet residues, which reduce their competitive binding ability with ACE2 after mutations. As for mutations with high frequencies, the mutation R403K has negative changes on most antibodies, which poses a danger of disrupting the binding of antibodies and S protein.

Figure  presents the BFE changes for ACE2 and antibodies along with the log of frequencies on each mutation of coil residues on the S protein RBD. Overall, most mutations on coil residues lead to negative BFE changes. Interestingly, CV30 has the most positive BFE changes following mutations, which can be a good candidate for potential therapy. For the high-frequency mutation S447N, the BFE changes are mild on ACE2 and antibodies. However, mutation L455F induces negative BFE changes for all antibodies except for COVA2-39, which is considered as a potentially dangerous mutation for antibody therapies.

## Statistical analysis of mutation impacts on COVID-19 antibodies

In statistics, most mutations (94 of 133) occur on residues whose secondary structures are coil, while 20 out of 133 mutations are on the helix, and 19 out of 133 mutations are on the sheet. Here, 12 mutations on the random coils and 2 mutations on helix are not calculated due to the far distance to antibodies. Moreover, residues on coil have more negative BFE changes (548 negative BFE changes vs. 534 positive BFE changes), while residues on the helix or sheet have more positive BFE changes (74 and 95 negative BFE changes vs. 94 and 123 positive BFE changes, respectively). Lastly, the maximum BFE changes of the helix, sheet, and coils are 4.47 kcal/mol, 4.63 kcal/mol, and 4.52 kcal/mol, while the minimum BFE changes are -2.91 kcal/mol, -2.95 kcal/mol, and -3.33 kcal/mol, respectively.

Figure  indicates the BFE changes extreme values (maximal in blue and minimal in red) of the complexes of S protein ACE2 or antibodies following mutations. Many antibodies, such as CR3022 and CR3022_H11-D4, are not very sensitive to the current S protein mutations. However, some other antibodies, such as CV30, Fab 2-4, and EY6Z_Nb, can be dramatically affected by SARS-CoV-2 mutations.

# Mutation impacts on COVID-19 vaccines

The increasing number of affected and dead individuals, the global spread situation, and the lack of prophylactics and therapeutics give rise to the urgent demand for the prevention of COVID-19. Vaccination is the most effective and economical means to prevent and control pandemics . Currently, 213 vaccines are in various clinical trial stages, as reported in an online COVID-19 Treatment And Vaccine Tracker (<https://covid-19tracker.milkeninstitute.org/#vaccines_intro>). Broadly speaking, there are four types of coronavirus vaccines in progress: virus vaccines, viral-vector vaccines, nucleic-acid vaccines, and protein-based vaccines, as shown in . The first type of vaccine is the virus vaccine, which injects weakened or inactivate viruses to the human body. A virus is conventionally weakened by altering its genetic code to reduce its virulence and elicit a stronger immune response. A biotechnology company Codagenix is currently working on a "codon optimization" technology to weaken viruses, and it has weakened virus vaccine is in progress . Unlike a weakened virus, the inactivated virus cannot replicate in the host cell. A virus is inactivated by heating or using chemicals, which induces neutralizing antibody titers and has been proven to have its safety . At this stage, both Sinopharm, which works with the Beijing Institute of Biological Products and Wuhan Institute of Biological Products, and Sinovac, which works with Institute Butantan and Bio Farma is developing inactivate SARS-CoV-2 vaccines that are in Phase III clinical trials.

The second type of vaccine is the viral-vector vaccine, which is genetically engineered so that it can produce coronavirus surface proteins in the human body without causing diseases. There are two subtypes of viral-vector vaccines: the non-replicating viral vector and the replicating viral vector. There are 4 non-replicating viral vector vaccines in Phase III trials. AstraZeneca and the University of Oxford, whose vaccine is in phase III trials in many countries. It works by taking a chimpanzee virus and coating it with the S proteins of SARS-CoV-2. The chimp virus causes a harmless infection in humans, but the spike proteins will activate the immune system to recognize signs of a future SARS-CoV-2 invasion. Notably, the booster shots can be needed to keep long-lasting immunity. Moreover, at this stage, only one replicating viral-vector vaccine is in Phase I. Institut Pasteur Themis, in cooperating with the University of Pittsburgh CVR and Merck Sharp & Dohme is developing such a replicating viral vaccine, which tends to be safe and provoke a strong immune response .

The third type of vaccine is nucleic-acid vaccines, including two subtypes: DNA-based vaccines and RNA based vaccines. At least 40 teams are currently working on nucleic-acid vaccines since they are safe and easy to develop. The DNA-based vaccine works by inserting genetically engineered blueprints of the viral gene into small DNA molecules such as plasmids for injection. Moreover, the electroporation technique is employed to create pores in membranes to increase DNA uptake into cells. The injected DNA will produce mRNA by transcription with the help of the nucleus in human cells. Such an mRNA will translate viral proteins (mostly spike proteins), which are dutifully produced by cells in response to the genes, alarm the immune system, and should produce immunity. Currently, there are four DNA-based vaccines in Phase II. Similar to DNA-based vaccines, the RNA-based vaccines provide immunity through the introduction of RNA, which is encased in a lipid coat to ensure its entering into cells. Two RNA-based vaccines are in Phase III, and companies such as Moderna, Biontec, and Pfizer are working on the advanced development of RNA-based vaccines.

The fourth type of vaccine is the protein-based vaccines, which aims to inject viral proteins directly to human bodies to trigger immune readiness. Protein subunits vaccine is one of the subtypes of the protein-based vaccine. More than 60 teams are working on vaccines with viral protein subunits, such as spike proteins and membrane (M) proteins. Another subtype of the protein-based vaccine is the virus-like particle (VLP) vaccine. The VLP vaccines closely resemble viruses. However, they are not infectious since they do not contain viral genetic material. The non-replicating propriety provides a safer alternative to weakened virus vaccines, the HPV vaccine or newer flu vaccines are VLP vaccines. Currently, 16 teams are working on the VLP vaccines for the future prevention of COVID-19.

## Secondary structures of antigenic determinants

Since the structural basis of antibody CDRs, or paratope, is random coils, we hypothesize that CDRs favor antigenic random coils as complementary epitopes, i.e., antigenic determinants . depicts the 3D structure of S protein, where the random coils are drawn with green strings, and the other secondary structure is described with the purple surface. It shows that the RBD and the NTD mostly consist of random coils. The RBD is the antigenic determinant of 27 structurally-known SARS-CoV-2 antibodies; meanwhile, the NTD is the binding domain of antibody 4A8, which confirms our hypothesis.

marks the secondary structure of the S protein. The red, blue, and green colors represent helix, sheet, and random coils of S protein. It can be seen that the S protein mostly consists of random coils, which means there are many other potential antigenic epitopes on the S protein for antibody CDRs. We believe that the emphasis of direct binding competition with ACE2 in the past has led to the neglecting of many important antibodies that do not bind to the RBD. Therefore, we suggest that researchers pay more attention to antibodies that do not bind to the RBD.

## Statistical estimation of mutation impacts on COVID-19 vaccines

Vaccine efficacy is an essential issue for the control of the COVID-19 pandemic. S protein is one of the most popular surface proteins for the vaccine development. However, mutations accumulated on the S protein of SARS-CoV-2, which may reduce the vaccine efficacy. As we found in , mutations are more likely to happen on the random coils of S protein, which may have a devastating effect on vaccines in the development.

As shown in , mutations could considerably weaken the binding between the S protein and antibodies and thus pose a direct threat to reduce the efficacy of vaccines. However, there are a few obstacles in determining the exact impacts of mutations to COVID-19 vaccines. Firstly, the four types of vaccine platforms can produce very different virus peptides, which will result in different immune responses, as well as antibodies. Secondly, even for a given vaccine platform, the different peptides may be produced due to different immune responses caused by gender difference, age difference, race difference, etc. Therefore, in this work, we proposed to understand the impact of SARS-CoV-2 mutations on COVID-19 vaccines by the statistical analysis. By evaluating the binding affinity changes induced by 28 existing SARS-CoV-2 antibodies, as shown in to , we can notice that the K417N, Y505C, F456L, and F486L mutations enhance the binding of almost all of the 28 antibodies. In contrast, the R403K, L455F, and P491R mutations have weakened the binding of almost all of the existing antibodies. Moreover, mutation K378N enhances the binding of antibody EY6A, whereas mutation V395I weakens the binding of antibody S309. Furthermore, it is noticed that many mutations such as K417N, Q414R, and G585R considerably disrupt many antibodies and thus may bring a threat to future vaccines. depicts the maximal and minimal binding free energy changes for S protein complexes and 28 antibodies or ACE2. It can be seen that antibodies CR3022, CR3022_H11-D4, BD-629, BD-604, and BD-362-2 are not very sensitive to the current mutations on the S protein. However, other antibodies, such as CV30 and EY6A, are very sensitive to current mutations. In a nutshell, by setting up a SARS-CoV-2 antibody library with the statistical analysis based on the mutation-induced binding free energies changes, we can estimate the impacts of SARS-CoV-2 mutations on COVID-19 vaccines, which will provide a way to infer how a specific mutation will pose a threat to vaccines. This approach works better when more antibody structures become available.

Another important factor in prioritization is mutation frequency. Figures , , and have provided frequency information from our SNP calling. Once a mutation is identified as a potential threat, it can be incorporated into the next generation of vaccines in a cocktail approach. In principle, all four types of vaccine platforms allow the accommodation of new viral strains.

# Conclusion

Coronavirus disease 2019 (COVID-19) pandemic has gone out of control globally. There is no specific medicine and effective treatment for this viral infection at this point. Vaccination is widely anticipated to be the endgame for taming the viral rampant. Another promising treatment that is relatively easy to develop is antibody therapies. However, both vaccines and antibody therapies are prone to more than 18,000 unique mutations recorded in the [Mutation Tracker](https://users.math.msu.edu/users/weig/SARS-CoV-2_Mutation_Tracker.html).

We present a prediction of mutation threats to vaccines and antibody therapies. First, we identify existing mutations on the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spike (S) protein, which is the man target for both vaccines and antibody therapies. We analyze the mechanism, frequency, and ratio of mutations along with the secondary structures of the S protein. Additionally, we build a library of antibodies with structures available from the Protein Data Bank (PDB) and analyze their two-dimensional (2D) and three-dimensional (3D) characteristics by employing computational biophysics. We further predict the mutation-induced binding free energy (BFE) changes of S protein and antibody complexes by a model called TopNetTree based on deep learning and algebraic topology. From these studies, we infer that some of S protein mutations may disrupt the binding of antibodies and S protein, which will further affect the efficacy and reliability of vaccines. To prioritize mutation threats, we also take into consideration of mutation occurrence frequency. The resulting algorithm indicates that some high-frequency mutations such as M153I, S254F, and S255F with negative BFE changes may potentially disrupt the efficacy and reliability of vaccines and antibody therapies currently in the development. Our method can provide the efficient prioritization of mutations to guild the design of the next generation of vaccines and antibody therapies.

# Supporting material

Supporting material is available for: S1 Method; S2 Multiple sequence alignments of the antibodies and pairwise identity scores; and S3 Mutation-induced changes of binding free energies of antibody-SARS-CoV-2 spike protein complexes.

# Data availability

Detailed mutation information is available for download at [Mutation Tracker](https://users.math.msu.edu/users/weig/SARS-CoV-2_Mutation_Tracker.html)

# Acknowledgments

This work was supported in part by NIH grant GM126189, NSF Grants DMS-1721024, DMS-1761320, and IIS1900473, Michigan Economic Development Corporation, George Mason University award PD45722, Bristol-Myers Squibb, and Pfizer. The authors thank The IBM TJ Watson Research Center, The COVID-19 High Performance Computing Consortium, NVIDIA, and MSU HPCC for computational assistance. RW thanks Dr. Changchuan Yin for useful discussion.

# Competing Interests

The authors declare no competing interests.

[^1]: Corresponding author. E-mail: firstname.lastname@example.com


abstract: We propose a practical approach based on federated learning to solve out-of-domain issues with continuously running embedded speech-based models such as wake word detectors. We conduct an extensive empirical study of the federated averaging algorithm for the "Hey Snips" wake word based on a crowdsourced dataset that mimics a federation of wake word users. We empirically demonstrate that using an adaptive averaging strategy inspired from Adam in place of standard weighted model averaging highly reduces the number of communication rounds required to reach our target performance. The associated upstream communication costs per user are estimated at 8 MB, which is a reasonable in the context of smart home voice assistants. Additionally, the dataset used for these experiments is being open sourced with the aim of fostering further transparent research in the application of federated learning to speech data.
address: Snips, 18 rue Saint Marc, 75002 Paris, France
title: Federated Learning for Keyword spotting

# Introduction

Wake word detection is used to start an interaction with a voice assistant. A specific case of keyword spotting (KWS), it continuously listens to an audio stream to detect a predefined keyword or set of keywords. Well-known examples of wake words include Apple's "Hey Siri" or Google's "OK Google". Once the wake word is detected, voice input is activated and processed by a spoken language understanding engine, powering the perception abilities of the voice assistant .

Wake word detectors usually run on device in an always-on fashion, which brings two major difficulties. First, it should run with minimal memory footprint and computational cost. The resource constraints for our wake word detector are 200k parameters (based on the medium-sized model proposed in ), and 20 MFLOPS.

Secondly, the wake word detector should behave consistently in any usage setting, and show robustness to background noise. The audio signal is highly sensitive to recording proximity (close or far field), recording hardware, but also to the room configuration. Robustness also implies a strong speaker variability coverage. While the use of digital signal processing front-ends can help mitigate issues related to bad recording conditions, speaker variability remains a major challenge. High accuracy is all the more important since the model can be triggered at any time: it is therefore expected to capture most of the commands (high recall) while not triggering unintentionally (low false alarm rate).

Today, wake word detectors are typically trained on datasets collected in real usage setting e.g. users homes in the case of voice assistants. Speech data being by nature very sensitive, centralized collection raises major privacy concerns. In this work, we investigate the use of federated learning (FL) in the context of an embedded wake word detector. FL is a decentralized optimization procedure that enables to train a central model on the local data of many users without the need to ever upload this data to a central server. The training workload is moved towards the user's devices which perform training steps on the local data. Local updates from users are then averaged by a parameter server in order to create a global model.

# Related work

Most research around decentralized learning has historically been done in the context of a highly controlled cluster/data center setting, e.g. with a dataset evenly partitioned in an i.i.d fashion. The multi-core and multi-gpu distributed training setting has been specifically studied in the context of speech recognition in . Efforts on decentralized training with highly distributed, unbalanced and non-i.i.d data is relatively recent, as the foundations were laid down in with the introduction of the federated averaging (*FedAvg*) algorithm and its application to a set of computer vision (MNIST, CIFAR-10) and language modeling tasks (applied to the Shakespeare and Google Plus posts datasets). To our knowledge, the present work is the first experiment of its kind on user-specific speech data.

The federated optimization problem in the context of convex objective functions has been studied in . The authors proposed a stochastic variance-reduced gradient descent optimization procedure (*SVRG*) with both local and global per-coordinate gradient scaling to improve convergence. Their global per-coordinate gradient averaging strategy relies on a sparsity measure of the given coordinate in users local datasets and is only applicable in the context of sparse linear-in-the-features models. The latter assumption does not hold in the context of neural networks for speech-based applications.

Several improvements to the initial *FedAvg* algorithm have been suggested with a focus on client selection , budget-constrained optimization and upload cost reduction for clients . A dynamic model averaging strategy robust to concept drift based on a local model divergence criterion was recently introduced in . While these contributions present efficient strategies to reduce the communication costs inherent to federated optimization, the present work is as far as we know the first one introducing a dynamic per-coordinate gradient update in place of the global averaging step.

The next section describes the federated optimization procedure, and how its global averaging can be substituted by an adaptative averaging rule inspired from Adam. It is followed by the experiments section, where both the open-sourced crowdsourced data and model used to train our wake word detector are introduced. Results come next, and a communication cost analysis is provided. Finally, the next steps towards training a wake word detector on user data that is really decentralized are described.

# Federated optimization

We consider the standard supervised learning objective function $f_{i}(w) = l(x_i,y_i,w)$ that is the loss function for the prediction on example $(x_i, y_i)$ when using a model described by a real-valued parameter vector $w$ of dimension $d$. In a federated setting, we assume that the datapoints $i$ are partitioned across $K$ users, each user being assigned their own partition $\mathcal{P}_{k}$, $|\mathcal{P}_{k}|=n_k$. The optimization objective is therefore the following:

$$\begin{array}{c}
\underset{w \in \mathbb{R}^{d}}{\mathrm{min}} f(w) \ \ \mathrm{where} \ \ f(w) \overset{\mathrm{def}}{=}\sum_{k=1}^{K}\frac{n_k}{n}\times F_k(w) \\ \\
\mathrm{with} \ \ F_{k}(w) = \frac{1}{n_k}\sum_{i=1}^{n_k}f_i(w)
\end{array}
\label{equ:opt_objective}$$

The *FedAvg* algorithm introduced in aims at minimizing the objective function assuming a synchronous update scheme and a generic non-convex neural network loss function. The model is initialized with a given architecture on a central *parameter server* with weights $w_{0}$. Once initialized, the parameter server and the user's devices interact synchronously with each other during *communication rounds*. A communication round at time $t \in [1, ..,T]$ is described below:

1.  The central model $w_{t-1}$ is shared with a subset of users $\mathcal{S}_{t}$ that are randomly selected from the pool of *K* users given a participation ratio $\textit{C}$.

2.  Each user $k \in \mathcal{S}_{t}$ performs one or several training steps on their local data based on the minimization of their local objective $F_{k}$ using mini-batch stochastic gradient descent (SGD) with a local learning rate $\eta_{local}$. The number of steps performed locally is
    $E\times max(ceil(\frac{n_k}{B}),1)$, $n_k$ being the number of datapoints available locally, *E* the number of local epochs and *B* the local batch size.

3.  Users from $\mathcal{S}_{t}$ send back their model updates $w_{t,k}, k\in \mathcal{S}_{t}$ to the parameter server once local training is finished.

4.  The server computes an average model $w_{t}$ based on the user's individual updates $w_{t,k} , k\in \mathcal{S}_{t}$, each user's update being weighted by $\frac{n_k}{n_r}$, '''where''' $n_r=\sum_{k \in \mathcal{S}_{t}}n_k \approx  C\times \sum_{k=1}^{K}n_{k}$.

When $B=\infty$ (i.e the batch size is equal to the local dataset size) and $E=1$, then a single gradient update is performed on each user's data. It is strictly equivalent to doing a single gradient computation on a batch including all of selected user data points. This specific case is called *FedSGD*, e.g. stochastic gradient descent with each batch being the data of the federation of selected users at a given round. *FedAvg* (Federated averaging) is the generic case when more than one update is performed locally for each user.

The global averaging step can be written as follows, using a global update rate $\eta_{global}$.

$$w_{t} \leftarrow w_{t-1} - \eta_{global}\sum_{k \in \mathcal{S}_{t}}\frac{n_k}{n}( w_{t-1} - w_{t,k} )
\label{equ:global_optimization}$$

Setting the global update rate $\eta_{global}$ to 1 is equivalent to a weighted averaging case without moving average. Equation highlights the parallel between global averaging and a gradient update $\mathcal{G}_{t} = \sum_{k \in \mathcal{S}_{t}}\frac{n_k}{n}( w_{t-1} - w_{t,k} )$. This parallel motivates the use of adaptive per-coordinate updates for $\mathcal{G}_{t}$ that have proven successful for centralized deep neural networks optimization such as Adam . Moment-based averaging allows to smooth the averaged model by taking into account the previous rounds updates that were computed on different user subsets. We conjecture that the exponentially-decayed first and second order moments perform the same kind of regularization that occurs in the mini-batch gradient descent setting, where Adam has proven to be successful on a wide range of tasks with various neural-network based architectures. In this work, we set the exponential decay rates for the moment estimates to $\beta_{1}=0.9$ and $\beta_{2}=0.999$ and $\epsilon = 10^{-8}$ as initially suggested by the authors of .

In the federated setting, model evaluation is also done in a distributed fashion on a set of users that are kept specifically for this purpose. Model metrics are averaged on the *parameter server* with a similar weighting scheme as parameter averaging.

# Experiments

## Dataset

Unlike generic speech recognition tasks, there is no reference dataset for wake word detection. The reference dataset for multi-class keyword spotting is the speech command dataset , but the speech command task is generally preceded by a wake word detector and is focused on minimizing the confusion across classes, not robustness to false alarms. We constituted a crowdsourced dataset for the *Hey Snips* wake word. We are releasing publicly [^1] this dataset in the hope it can be useful to the keyword spotting community.

| Train set   | Dev set    | Test set   | Total       |
|:------------|:-----------|:-----------|:------------|
| 1,374 users | 200 users  | 200 users  | 1774 users  |
| 53,991 utt. | 8,337 utt. | 7,854 utt. | 69,582 utt. |

Dataset statistics for the *Hey Snips* wake word - 18% of utterances are positive, with strong per user imbalance in the number of utterances *(mean: 39, standard dev: 32)*

The data used here was collected from 1.8k contributors that recorded themselves on their device with their own microphone while saying several occurrences of the *Hey Snips* wake word along with negative short sentences from various text sources (e.g. subtitles).

This crowdsourcing-induced data distribution mimicks a real-world non-i.i.d, unbalanced and highly distributed setting, and a parallel is therefore drawn in the following work between a crowdsourcing contributor and a voice assistant user. The statistics about the dataset comforting this analogy are summarized in Table . The train, dev and test splits are built purposely using distinct users, 77% of users being used solely for training while the remaining are used for parameter tuning and final evaluation, measuring the generalization power of the model to new users.

## Model

Acoustic features are generated based on 40-dimensional mel-frequency cepstrum coefficients (MFCC) computed every 10ms over a window of 25ms. The input window consists in 32 stacked frames, symmetrically distributed in left and right contexts. The architecture is a CNN with 5 stacked dilated convolutional layers of increasing dilation rate, followed by two fully-connected layers and a softmax inspired from . The total number of parameters is 190,852, initialized using Xavier initialization . The model is trained using cross-entropy loss on frames prediction. The neural network has 4 output labels, assigned via a custom aligner specialized on the target utterance "Hey Snips": "Hey", "sni", "ps", and "filler" that accounts for all other cases (silence, noise and other words). A posterior handling generates a confidence score for every frame by combining the smoothed label posteriors. The model triggers if the confidence score reaches a certain threshold $\tau$, defining the operating point that maximizes recall for a certain amount of False Alarms per Hour (FAH). We set the number of false alarms per hour to 5 as a stopping criterion on the dev set. The dev set is a "hard" dataset when it comes to false alarms since it belongs to the same domain as data used for training. The model recall is finally evaluated on the test set positive data, while false alarms are computed on both the test set negative data and various background negative audio sets. See section for further details about evaluation.

## Results

We conduct an extensive empirical study of the federated averaging algorithm for the *Hey Snips* wake word based on crowdsourced data from Table . Federated optimization results are compared with a *standard setting* e.g. centralized mini-batch SGD with data from train set users being randomly shuffled. Our aim is to evaluate the number of communication rounds that are required in order to reach our stopping criterion on the dev set. For the purpose of this experiment early stopping is evaluated in a centralized fashion, and we assume that the dev set users agreed to share their data with the parameter server. In an actual product setting, early stopping estimation would be run locally on the devices of the dev users, they would download the latest version of the central model at the end of each round and evaluate the early stopping criterion based on prediction scores for their own utterances. These individual metrics would then be averaged by the parameter server to obtain the global model criterion estimation. Final evaluation on test users would be done in the same distributed fashion once training is finished.

**Standard baseline**: Our baseline e.g. a standard centralized data setting with a single training server and the Adam optimizer reaches the early stopping target in 400 steps ($\sim2$ epochs) and is considered as our upper bound of performance. Adam provides a strong convergence speedup in comparison with standard SGD that remains under 87% after 28 epochs despite learning rate and gradient clipping tuning on the dev set.

**User parallelism**: The higher the ratio of users selected at each round *C*, the more data is used for distributed local training, and the faster the convergence is expected, assuming that local training does not diverge too much. Figure shows the impact of *C* on convergence: the gain of using half of users is limited with comparison with using 10%, specifically in the later stages of convergence. A fraction of 10% of users per round is also more realistic in a practical setup as selected users have to be online. With lower participation ratios ($C=1\%$), the gradients are much more sensitive and might require the use of learning rate smoothing strategy. *C* is therefore set to 10%.

**Global averaging**: Global adaptive learning rates based on Adam accelerates convergence when compared with standard averaging strategies with or without moving averages. Table summarizes experimental results in the *FedSGD* setting with optimized local learning rates. Applying standard global averaging yields poor performances even after 400 communication rounds when compared with adaptive per-parameter averaging.

| Avg. Strategy         | 100 rounds | 400 rounds |
|:----------------------|:----------:|:----------:|
| $\eta_{global}=1.0$   |   29.9%    |   67.3%    |
| Adam                  |            |            |
| $\eta_{global}=0.001$ |   93.50%   |   98.29%   |

Dev set recall / 5 FAH for various averaging strategies - *FedSGD*, $\textit{C}=10\%$

**Local training**: Our results show consistency across local training configurations, with limited improvements coming from increasing the load of local training. The number of communication rounds required to reach the stopping criterion on the dev set ranges between 63 and 112 communication rounds for $\textit{E} \in [1,3]$ and $\textit{B}\in [20, 50, \infty]$, using $C=10\%$, *Adam* global averaging with $\eta_{global}= 0.001$, and a local learning rate of 0.01. In our experiments, the best performances are obtained for $E=1$ and $B=20$ for an average of 2.4 local updates per worker taking part in a round, yielding a 80% speedup with comparison to *FedSGD*. The total number of training steps needed to reach the stopping criterion amounts to approximately $3300$ for $100$ communication rounds e.g. $8.25$ times the number of steps required in the standard setting, with much smaller batches. Nevertheless we observed variability across experiments with regard to random weight initialization and early stage behaviour. Unlike some experiments presented in , the speedup coming from increasing the amount of local training steps does not lead to order of magnitude improvements on convergence speed, while local learning rate and global averaging tuning proved to be crucial in our work. We conjecture that this difference is related to the input semantic variability across users. In the MNIST and CIFAR experiments from the input semantics are the same across emulated users. In the wake word setting, each user has their own vocalization of the same wake word utterance with significant differences in pitch and accent than can lead to diverging lower stage representations that might perform poorly when averaged.

**Evaluation**: We evaluate the false alarm rates of the best model ($E=1$ and $B=20$) for a fixed recall of 95% on the test set. We observe 3.2 FAH on the negative test data, 3.9 FAH on Librispeech , and respectively 0.2 and 0.6 FAH on our internal news and collected TV datasets. Unsurprisingly, false alarms are more common on close-field continuous datasets than they are on background negative audio sets.

## Communication cost analysis

Communication cost is a strong constraint when learning from decentralized data, especially when user's devices have limited connectivity and bandwidth. Considering the asymmetrical nature of broadband speeds, the communication bottleneck for federated learning is the updated weights transfer from clients to the parameter server . We assume that the upstream communication cost associated with users involved in model evaluation at each communication round is marginal, as they would only be uploading a few floating point metrics per round that is much smaller than the model size. The total client upload bandwidth requirement is provided in the equation below: $$\rm{ClientUploadCost} = \rm{modelSize}_{f32}\times C \times N_{rounds}
\label{equ:com_cost}$$

Based on our results, this would yield a cost of 8MB per client when the stopping criterion is reached within 100 communication rounds. The server receives 137 updates per round when $\textit{C}=10\%$, amounting for 110GB over the course of the whole optimization process with 1.4k users involved during training. Further experiments with latter convergence stages (400 rounds) yielded 98% recall / 0.5 FAH on the test set for an upload budget of 32 MB per user.

# Conclusion and future Work

In this work, we investigate the use of federated learning on crowdsourced speech data to learn a resource-constrained wake word detector. We show that a revisited *Federated Averaging* algorithm with per-coordinate averaging based on Adam in place of standard global averaging allows the training to reach a target stopping criterion of 95% recall per 5 FAH within 100 communication rounds on our crowdsourced dataset for an associated upstream communication costs per client of 8MB. We also open source the *Hey Snips* wake word dataset.

The next step towards a real-life implementation is to design a system for local data collection and labeling as the wake word task requires data supervision. The frame labeling strategy used in this work relies on an aligner, which cannot be easily embedded. The use of of memory-efficient end-to-end models in place of the presented class-based model could ease the work of local data labelling.

[^1]: <http://research.snips.ai/datasets/keyword-spotting>


abstract: In the coming era of exascale supercomputing, in-situ visualization will be a crucial approach for reducing the output data size. A problem of in-situ visualization is that it loses interactivity if a steering method is not adopted. In this paper, we propose a new method for the interactive analysis of in-situ visualization images produced by a batch simulation job. A key idea is to apply numerous (thousands to millions) in-situ visualizations simultaneously. The viewer then analyzes the image database interactively during postprocessing. If each movie can be compressed to 100 MB, one million movies will only require 100 TB, which is smaller than the size of the raw numerical data in exascale supercomputing. We performed a feasibility study using the proposed method. Multiple movie files were produced by a simulation and they were analyzed using a specially designed movie player. The user could change the viewing angle, the visualization method, and the parameters interactively by retrieving an appropriate sequence of images from the movie dataset.
address: Graduate School of System Informatics, Kobe University, Japan
author: Akira Kageyama; Tomoki Yamada
title: An Approach to Exascale Visualization:
  Interactive Viewing of In-Situ Visualization

# Introduction

In supercomputer simulations, postprocessing has become a bottleneck during the overall research cycle. This is because the disk I/O and the network bandwidths cannot keep pace with the exponential growth in the computer processor speed. To resolve this imbalance, the user has to apply a data reduction method to the output data in supercomputer simulations. One approach is to compress the raw numerical data directly. Discrete Fourier transforms and wavelet transforms are often used for this purpose. Another approach to output data reduction is to apply the visualization during runtime. This approach, which is known as in-situ visualization, is effective for reducing the data size because images are two-dimensional.

In-situ visualization poses new challenges for visualization researchers. New difficulties arise because today's supercomputers are massively parallel machines. The target data for visualization are divided into pieces, which are distributed in network-connected memories. Thus, new visualization methods and algorithms need to be implemented on supercomputers for in-situ visualization. They require good parallel scaling to avoid degradation of the simulation's scale. These challenges have focused the attention of the High Performance Computing (HPC) community on in-situ visualization. Recent studies of in-situ visualization for HPC are reviewed in Section .

A key point of in-situ visualization is the correct settings for two types of parameters: visualization parameters and camera parameters. Visualization parameters include applied visualization methods and their states and values. The camera parameters include the viewpoint position, direction, field of view, and viewing frustum culling.

A simple in-situ visualization with one fixed camera but without steering is described conceptually in Fig. a, where a three-dimensional (3-D) space is spanned by three bases: the camera parameters, the visualization parameters, and the simulation time. The white ball in this figure represents a visualization shot taken in the 3-D space. After the simulation task is finished, the visualized images are sent to a local disk drive on the data analyzer's PC. The white arrow in the figure indicates data transfer. After this in-situ visualization, the sequential images are shown as a movie on the PC's display.

In most simulations, the key phenomena or events usually appear in localized, discrete spots in the whole simulation region. If this type of "hotspot" is next to the range of the camera view, the user has to reset the camera positions in the so-called scenario file and resubmit a simulation job, unless the steering simulation approach is taken.

In the steering simulation, the analyzer can control the simulation and the visualization parameters dynamically during runtime. A steering simulation with an in-situ visualization is illustrated in Fig b. The visualized images are transferred to the analyzer during the runtime of the simulation; see the white arrow in Fig. b. By observing the images on the PC monitor, the analyzer can change the visualization parameters dynamically to focus on a hotspot. The dashed line with an arrow in Fig. b represents the feedback control from the analyzer to the simulation and visualization. Although this kind of the steering simulation is an effective approach for HPC, it is not always possible because supercomputer simulations are usually performed in batch jobs.

Without steering, the in-situ visualization loses interactivity completely, which is critically important for obtaining insights from the simulation.

This paper proposes a method that facilitates the interactive viewing of the visualization results generated by the in-situ visualization of a batch-job supercomputer simulation.

# Interactive View of In-Situ Visualization Data

The key concept used by the method proposed in this paper is to produce a cluster of in-situ visualizations with different visualization and camera parameters at once and to analyze the output image data interactively during postprocessing.

The proposed method can be regarded, in some sense, as a generalization of the so-called bullet-time method, which is used in the film industry, where cameras are placed along a specified path and pictures are taken sequentially with short time intervals. This method is used to make a slow motion movie of the target object, which appears to be taken by a camera moving at an impossibly high speed. Figure c shows a conceptual description of the bullet-time method where the visualization is obtained using four different cameras. Similar to Fig. a, the output data is a sequence of images, which is then sent to the analyzer.

In our method, we use as many in-situ visualizations as possible and analyze the output image data later in an interactive manner, as shown in Fig. d. Any information on the simulated phenomena is expected to be in the image dataset so the analyzer can explore the image space dynamically.

When an output movie from an in-situ visualization is compressed to a reasonable size, such as 100 MB, the total output data size is *only* 100 TB, even if the number of applied in-situ visualization is one million. This is still smaller than the size of the raw numerical data in exascale simulations, which would be in the order of PB.

When we select only the camera position among the camera parameters, our proposed method requires that we place many cameras inside and outside the simulation region and that we use them all in the in-situ visualization in parallel, as shown in Fig. . This configuration of cameras reminds us of the "3D Dome" or "3D Room" constructed by Kanade et al. for capturing human motion in a room-sized space to construct a virtual reality . Our method can be regarded as an extension of the 3D room concept to scientific visualization. An important difference is that we explore not only in the camera space, but also in the visualization space, in our in-situ visualization method.

The output data of our proposed method is a dataset containing many (thousands to millions) movie files. When the camera number is sufficiently high, we can find practically any image from any position in the dataset—although a large part of the dataset would be useless. Therefore, the user can "rotate" a visualized object with the mouse, by extracting an appropriate sequence of image files from the movie dataset.

To demonstrate the feasibility of the proposed method, we performed a batch simulation on a PC cluster system where $130$ cameras were placed around the simulation region. Three types of in-situ visualization methods were used by all of the cameras during runtime. The output data comprised $130 \times 3 = 390$ sets of MPEG movie files. We also developed an interactive movie player so a sequence of images can be extracted from the dataset.

# Related Work

In-situ visualization has a long history. Johnson et al.  conducted a general review of steering simulation and visualization. Examples of in-situ visualization in peta-scale simulations and their technical challenges (especially those caused by massively parallel processing) are summarized in . They show that in-situ visualization is a promising solution for peta and exascale simulations. A natural extension of the online dynamical control of simulation is the end-to-end approach , where even mesh generation, which is usually conducted in the preprocessing stage, is performed on supercomputers. They developed a steering simulation for seismic wave propagation where the visualization images were shown in real time.

Huang et al. developed a steering simulation and visualization framework for environmental science where dynamical control over the Internet was implemented . Their user interface was constructed on web browsers. Ellsworth et al. developed an in-situ visualization system on the Colombia supercomputer for running a weather forecasting model . (Our experimental system described in section  is similar to their system because multiple MPEG files are generated by multiple in-situ visualizations. However, their movies are shown separately in each panel of a tiled display system, whereas ours are used as a database.) Esnard et al. developed an in-situ visualization system where the parallel visualization processing can run on a different computer system from the simulation computer system . In their development, special emphasis was placed on producing a steering environment with existing simulation codes.

Many in-situ visualizations, with or without steering, have been designed and developed for particular simulation problems. However, general visualization frameworks with high parallel scalability have also become available recently. Whitlock et al.  developed a library *Libsim* that facilitates in-situ visualizations using VisIt, which is one of the most sophisticated parallel visualization tools available today. Their paper also contained a concise review of the history and the latest status of the in-situ visualization research. Fabian et al.  reported the development of a coprocessing library for *ParaView*, which is another sophisticated parallel visualization tool. Using that library, it is possible to utilize the various visualization functions provided by *ParaView* during runtime, which is decoupled from the simulation. The latest case studies of in-situ visualization using *VisIt* and *ParaView* can be found in .

# Configuration of Viewpoints

One of the key points of our approach to interactive in-situ visualization is the configuration of multiple cameras. In an extreme case, cameras can be placed in a 3-D distribution as densely as possible inside and around the simulation region. In our feasibility test in the present study, we placed only $130$ cameras with a 2-D distribution on a spherical surface at a fixed radius from the center of the simulation region, to ensure that the test was at a moderate level. All of the cameras pointed at the center.

There are only five regular polyhedra so it is not a straightforward task to place more than 20 cameras on a spherical surface in a uniform manner.

Placing cameras on each grid point of a spherical coordinate system, or the so-called latitude-longitude mesh, does not work well because it leads to concentrations of cameras around the poles. Reducing the number of cameras at higher latitudes is not a good idea because camera motion along a constant longitude will lead to a lack of viewpoints (Fig. ). To avoid this problem, we use a Yin-Yang grid  where two congruent component grids, which are a part of low latitude region of the latitude-longitude mesh, are combined with a partial overlap to cover a spherical surface (Fig. ).

# Interactive View of Many Movies

The camera position is not the only parameter that can be explored using our method, e.g., see the parameter axis in Fig. d. Different visualization methods and their internal parameters such as the isosurface level, slice position, and color functions can also be explored using our method. An ultimate aim of our method is to obtain many (hopefully thousands) different visualization parameters from each camera. In this test, however, we only used three different visualization parameters. The full details are provided in the next section. The total number of output movie files is $390$ $(=130 \hbox{ cameras} \times 3 )$.

These movie files contain a lot of information that would be useless without a proper player. A movie player for PCs usually performs a simple task, i.e., it reads a source movie file and shows the sequence of images contained in the source file on the screen. Thus, we need a generalized player that can read multiple ($390$ in the present test and millions in the future) source files and display a sequence of images derived from them. The play should be interactive, i.e., the source should be instantly and smoothly changed based on a user's input such as a mouse motion or keystroke.

We have developed an interactive movie player using the OpenCV library. A snapshot is shown in Fig. . Our movie player can retrieve a sequence of images from different movie files and display them on the window as a single movie. The player we developed has three functions: (i) to stop and play the motion; (ii) to change the source movie; and (iii) to zoom in/out of a scene.

The zoom in/out function was implemented using the image magnification function provided by OpenCV. If we placed many more cameras in a fully 3-D distribution, we would be able to perform the zoom in/out function by changing the source cameras.

The aim of the method proposed in this paper is to cover the whole visualization parameter space by applying the in-situ visualizations at once. As shown in Fig. d, the visualization parameter space is covered not only by the camera position but also by the applied visualization methods and their internal parameters. In this experiment, we have used just one visualization method (volume rendering) with three different types of target data. In exascale applications, we will apply $10^3$ different visualization methods and parameters for each camera. The movie player that we have developed in this study can extract any image sequence from different movie files in the loaded movie dataset, so there are no technical restrictions on interactive changes in the visualization parameters, even in the present version of the movie player. For example, we can smoothly change the isosurface level by a keystroke or mouse drag if we have isosurface visualizations for each camera with sufficient different isosurface levels.

As for the volume rendering, it is difficult to specify a proper range of the transfer function beforehand. We would not, however, take a special care for the range setting, by accepting the possibility that large part of the volume rendering images is useless due to unsuitable transfer functions, as we have already accepted the possibility that many cameras are taking images from useless angles in the proposed method. A point in this method is to find valuable information afterwards in a large amount of movie database.

# Application to Seismic Wave Propagation Simulation

In our feasibility study, we performed a seismic wave propagation simulation using our proposed in-situ visualization method. The simulation and visualization codes were developed by Furumura and Chen. The details of the codes are described in their paper , so we briefly summarize only the key features related to our in-situ visualization experiment.

This simulation solves the time development of seismic waves in Cartesian coordinates $(x,y,z)$. The basic variables are the displacement field ${\bf u}(x,y,z)$ of a viscoelastic medium and its velocity ${\bf v}(x,y,z)$. The viscoelastic equations of motion are solved for them using a high-order finite difference method. A pseudo-spectral method can also be selected in their code if the spatial accuracy is critically important. However, we used only the finite difference method in this test.

One-dimensional domain decomposition is applied on the $z$ (vertical) axis for the parallel simulation. The rectangular simulation region is divided into horizontal slates. This code has good, almost linear, scaling up to more than 1000 processors in the Earth Simulator supercomputer . Hybrid parallelization is used with MPI and OpenMP.

Furumura and Chen also developed visualization tools for their seismic wave propagation simulation. These tools were derived from a highly optimized parallel visualization package , which was developed in the GeoFEM framework  for the Earth Simulator supercomputer. The original visualization package was designed to use unstructured cell data. In this visualization tool for seismic wave propagation, however, they were applied to a Cartesian structured mesh. Various visualization methods based on parallel ray castings were implemented in the original package of GeoFEM. In this test, we used only one type of visualization method, i.e., volume rendering for scalar fields.

Although the visualization tools used in this experiment originate from the GeoFEM package for unstructured cell data, the source code has been converted into a version that can be applied directly to Cartesian structured mesh data. Therefore, we could apply the visualization to the simulation data without converting the data format.

More sophisticated visualization packages such as VisIt and ParaView are now available. These modern packages provide highly scalable parallel rendering that can be integrated with the simulation code, so they would be a much better choice for the rendering engine of the in-situ visualization method proposed in this paper for general simulations. However, we used the GeoFEM-derived visualization tool in this prototype experiment simply because it was strongly coupled to the seismic wave propagation simulation. We will implement our method with ParaView in the future.

Seismic waves have two modes: s-waves (shear mode) and p-waves (compressional mode). We defined two scalar fields $\phi_s\equiv |\nabla\times{\bf v}|$ (for s-wave) and $\phi_p\equiv \nabla\cdot{\bf v}$ (for p-wave), and we applied the in-situ volume rendering visualization to the two fields $\phi_p$ and $\phi_s$.

We performed test runs on a PC cluster system where the simulation grid size was $256\times 512 \times 160$. The number of MPI processes was 40. As mentioned above, 130 cameras were placed spherically on the Yin-Yang grid points, i.e., 65 for Yin and 65 for Yang. Each camera performed three visualization tasks: volume rendering for the s-wave, the p-wave, and both of them. Visualization snapshots were captured 60 times during one simulation task. The images were saved in the PPM format and the size of each image was $2.5~$MB. The total size of all the images was $57\hbox{ GB} \ (=
2.5 \hbox{ MB} \times 130 \hbox{ cameras}
\times 3 \hbox{ tasks} \times 60 \hbox{ frames}$). After the simulation task, each set of sequential images captured by a single camera was combined into a movie file in the MPEG format using the FFmpeg library. (We will implement runtime compression in the future.) The compression rate was about $3~\%$; the total PPM image set of $57$ GB was converted into MPEG files with a total size of $1.7$ GB.

If we would save the raw numerical data from $\phi_s$ and $\phi_p$ with single-point precision for postprocessing, it would amount to $10\hbox{ GB}\  (=256\times 512 \times 160 \hbox{ grids} \times 4 \hbox{ B} \times 2 \times 60 \hbox{ snapshots}$). Our movie data ($1.7$ GB) was an order of magnitude smaller than this. This gap will increase further in larger scale simulations.

The total number of movie files produced during this experiment was $390$ $(=130 \hbox{ cameras} \times 3 \hbox{ tasks})$. The 390 MPEG files of $1.7$ GB were transferred to a PC, as shown in Fig. , and they were loaded into the interactive movie player described in Section .

Figure  shows a sequence of snapshots with an interactive view of the in-situ visualization movie data. The propagation of the p-wave (orange) and s-wave (green) can be observed in the movie displayed in a PC window. Typing a key on the keyboard sends a signal to the movie player, which changes the source movie file (or changes the camera angle). The user can observe these phenomena from any one of the spherically distributed 130 cameras shown in Fig. . Image retrieval from the movie dataset is smooth and fast. The user can change the camera position while the seismic wave propagates in the window. Despite the relatively small number of cameras, the change of view is so smooth that the user might think that this is a normal postprocessing visualization.

Figure  shows snapshots of the interactive switching of visualization tasks. At the beginning, the p-wave and s-wave are visualized in the window. At this point, the movie player is retrieving an image sequence from a movie file, which contains images of the p-wave and s-wave. When the viewer types a key, the movie player changes the source file and starts to retrieve image sequence from the new source, which may contain only the s-wave images. Thus, the window will show wave propagation. The user does not experience any delay during this switch. Another keystroke produces an instant switch in the visualization task to focus on p-wave propagation.

Figure  shows the zoom function available in the movie player. As mentioned in Section , we will place a high density of cameras in the most uniform 3-D configuration possible in our final implementation in the future. Thus, if we need to focus on a small area in the simulation region, our interactive movie player will extract other image sequences captured by a camera located closer to the target region. In the current test, however, we implemented the zoom function by magnifying images captured by a fixed camera.

# Concluding Remarks

In this paper, we have proposed a new in-situ visualization method for exascale simulations. This method can generate numerous (hopefully millions) in-situ visualizations simultaneously based on thousands of different visualizations captured by many (possibly thousands) different cameras. The output of this simulation strategy is movies, rather than numbers.

When a simulation produces one million movies, the total output data size is only 100 TB if each movie is compressed to 100 MB. It should be noted that a size of 100 TB will be quite small in the coming exascale era. The output data would be much larger if a user tries to save raw numerical data with the same temporal resolution for standard postprocessing visualization.

In our experiment, each in-situ visualization movie was compressed to $2.5~$MB for each visualization parameter (including the camera position) and visualization method. The pixel size of a movie was $920 \times 896$ and the frame number was 60. The frame number of 60 would need to be increased in future exascale applications and each image would require finer resolution because the phenomena simulated in future simulations are expected to have higher complexity. However, we would not have to produce visualizations of "ultra-fine" resolution movies, even if "ultra-fine" simulations are performed. This is because if we need to analyze a fine structure in a simulation region, there would be a camera close to that spot and visualization movies could be captured at a moderate resolution, e.g., $1600 \times 1200$ pixels (UXGA), which would provide the necessary information. Therefore, a movie file size of $100$ MB would be a reasonable estimate.

An output of one million movies can be analyzed during postprocessing using our proposed method. Our specially designed movie player reads the one million movie files and displays a sequence of images in a window. After obtaining an appropriate image sequence from different movie files, we can "rotate" the visualization objects effectively while the dynamic phenomena are displayed in the window. We can also change the visualization method by using different internal parameters, provided the corresponding in-situ visualizations have been applied during the simulation task.

To demonstrate the feasibility of our proposed method, we performed a simulation where 390 in-situ visualizations were generated. We also developed a movie player that reads 390 movie files and displays a movie in a PC window. Using this player, we confirmed that it was possible to "rotate" a visualized object interactively and to change the visualization task in real time by a single keystroke while the movie was playing.

Clearly, the size of our experiment was far from the exascale range. The major difference is in the size of the output dataset. In our experiment, the number of the movie files was only 390, whereas that would be $10^6$ in exascale tasks because 1000 cameras will be placed in and around the simulation region, and each camera will capture 1000 different visualizations. Since a set of visualizations by different cameras is an embarrassingly parallel problem, the scalability for the number of cameras does not matter as long as plenty number of computer nodes are available.

The size of each movie file will not increase dramatically even with the development of exascale applications, provided that researchers view the movies on monitor windows with 2k to 4k pixel widths. In our experiment, reading 390 movie files (1.7 GB) required 1.88 s when using an Apple iMac (SATA 7200 rpm) and 1.42 s with an HP Z800 (SATA 7200 rpm). Extrapolating the read time in a linear manner, it will take $3600\sim 4800$ s if we load $10^6$ movie files in the memory and the memory size required is $4.4$TB. We cannot predict the main memory size available and the I/O bandwidth of storage devices in the exascale era, but it is possible that we will need to take special care when handling such a large dataset. A possible solution is a prefetch mechanism, which would allow the dynamic loading of part of the movie dataset that is expected to be required by the user's next request. We will implement this prefetch mechanism in our movie player.

Our proposed method can be summarized as follows: (i) identify the potential ranges for the visualization and the camera parameters that will be of interest to the user; (ii) discretize the parameter subspace at as fine a level as possible; (iii) apply in-situ visualizations to all of the discrete parameters (the white balls in Fig. d) and save the output movies; and (iv) explore the movie data space using an interactive movie player. Any information required for the analysis will be present in the movie dataset.

In conclusion, our experiment suggests that this method of in-situ visualization with interactive viewing will be practical for peta and exascale supercomputing.

# Acknowledgements

We thank Professor Furumura for providing the simulation and volume rendering codes for seismic wave propagation. This work was supported by Grant-in-Aid for Scientific Research (KAKENHI) 23340128 and the Takahashi Industrial and Economic Research Foundation.