author: Jukka P. Pekola; Juha J. Vartiainen; Mikko Möttönen; Olli-Pentti Saira; Matthias Meschke; Dmitri V. Averin
title: Hybrid single-electron transistor as a source of quantized electric current

**The basis of synchronous manipulation of individual electrons in solid-state devices was laid by the rise of single-electronics about two decades ago . Ultra-small structures in a low temperature environment form an ideal domain of addressing electrons one by one. A long-standing challenge in this field has been the realization of a source of electric current that is accurately related to the operation frequency $f$ . There is an urgent call for a quantum standard of electric current and for the so-called metrological triangle, where voltage from Josephson effect and resistance from quantum Hall effect are tested against current via Ohm's law for a consistency check of the fundamental constants of Nature, $\hbar$ and $e$ . Several attempts to create a metrological current source that would comply with the demanding criteria of extreme accuracy, high yield, and implementation with not too many control parameters have been reported. However, no satisfactory solution exists as yet despite many ingenious achievements that have been witnessed over the years . Here we propose and prove the unexpected concept of a hybrid metal-superconductor turnstile in the form of a one-island single-electron transistor with one gate, which demonstrates robust current plateaus at multiple levels of $ef$ within the uncertainty of our current measurement. Our theoretical estimates show that the errors of the present system can be efficiently suppressed by further optimizations of design and proper choice of the device parameters and therefore we expect it to eventually meet the stringent specifications of quantum metrology.**

Synchronized sources, where current $I$ is related to frequency by $I=Nef$ and $N$ is the integer number of electrons injected in one period, are the prime candidates for the devices to define ampere in quantum metrology. The accuracy of these devices is based on the discreteness of the electron charge and the high accuracy of frequency determined from atomic clocks. Modern methods are replacing classical definitions of electrical quantities; voltage can be derived based on the AC Josephson effect of superconductivity  and resistance by quantum Hall effect , but one ampere still needs to be determined via the mutual force exerted by leads carrying the current. Early proposals of current pumps for quantum metrology were based on arrays of mesoscopic metallic tunnel junctions , in which small currents could eventually be pumped at very low error rates . However, these multijunction devices are hard to control and relatively slow . Thus, the quest for feasible implementation with a possibility of parallel architecture for higher yield have lead to alternative solutions such as surface-acoustic wave driven one-dimensional channels , superconducting devices , and semiconducting quantum dots . These do produce large currents in the nano-ampere range but their potential accuracy is still limited.

Somewhat surprisingly, a simple hybrid single-electron transistor, with normal metal (N) leads and a small superconducting (S) island, see Fig. , has been overlooked in this context. As it has turned out in the present work, an SNS-transistor, or alternatively an NSN-transistor, presents ideally a robust turnstile for electrons showing accurately positioned current plateaus. We emphasize here that a one island turnstile does not work even in principle without the hybrid design. An important feature in the present system is that hybrid tunnel junctions forbid tunnelling in an energy range determined by the gap $\Delta$ in the density of states of the superconductor, see Fig. d inset; current through a junction vanishes as long as $|V_J| \lesssim
\Delta/e$.

Figure a shows the simple electric configuration to operate a hybrid turnstile. A DC bias voltage $V$ is applied between the source and drain of the transistor and a voltage $V_g$ with DC and AC components at the gate. To understand the operation of the turnstile on a more quantitative level, let us follow a basic operation cycle shown in Fig. c. In general, a sinusoidal AC gate voltage is superposed on the DC offset such that the total instantaneous voltage on the gate, normalized into charge in units of $e$, reads $n_g \equiv C_g V_g/e = n_{g0}+A_g \sin(2\pi
f t)$ at frequency $f$. Here $C_g$ is the capacitance of the gate electrode to the transistor island. In Fig. c we have chosen the gate offset $n_g$ and amplitude $A_g$ to be $n_{g0}=A_g=0.5$, and the bias voltage across the transistor is set at $V=\Delta/e$ to suppress tunnelling errors, as will be discussed below. The key point in the operation of the hybrid turnstile is that the charge state locks to a fixed value in any part of the operational cycle except at the moment when a desired tunnelling event occurs. This is the key feature of the device, originating from the interplay of the superconducting gap in the energy spectrum and Coulomb blockade of single-electron tunnelling. It renders this structure to work as an accurate turnstile where errors can be suppressed efficiently by decreasing temperature and by choosing the bias point properly within the superconducting gap. This locking mechanism is illustrated and explained in Fig. c for one operational cycle. On the contrary, in the biased NNN transistor with Coulomb blockade alone, non-synchronized almost frequency independent DC current through the device is observed . Likewise, a corresponding fully superconducting SSS device is not favourable either, because inevitable supercurrent of Cooper pairs induces significant leakage errors .

Several hybrid turnstiles with aluminium as the superconductor, copper as the normal metal, and aluminium oxide as the tunnel barrier in between were fabricated by standard electron beam lithography. Both the aluminium and the copper films were 50 nm thick. Figure a shows the NSN sample whose data we present here. The charging energy of the aluminium island, $E_C
=e^2/(2C_\Sigma)$, is $E_C/k_B \simeq 2$ K, where $C_\Sigma$ is the total capacitance. The sum of the tunnel resistances of the two junctions is 700 k$\Omega$, i.e., 350 k$\Omega$ per junction on the average. The current-voltage ($IV$) characteristics of the transistor are shown in Fig. d at various values of the DC gate voltage and with no AC gate voltage applied. The superconducting gap suppresses the current strongly in the bias region $|V| \lesssim 0.4$ mV. Outside this region the typical gate modulation pattern shows up . The charging energy of the device was determined based on the envelopes of these $IV$ curves.

The turnstile experiments were carried out by voltage biasing the transistor at $V\simeq \Delta/e$, highlighted by an arrow in Fig. d. Figure shows the current through the NSN turnstile under varying parameters $n_{g0},A_g$, and $V$ at a fixed frequency $f=20$ MHz. Figure c shows cross-sections of the 3D plot in Fig. a along different constant values of $n_{g0}$ against the gate amplitude $A_g$. The corresponding prediction based on sequential tunnelling theory  is shown by the dashed lines in the same plot. The experimental data follow the theoretical prediction very closely. Moreover, the wide flat plateaus at $I=Nef$ seem indeed promising for metrological purposes. The magnitude of the pumped current is robust against fluctuations in relevant parameters. It is not sensitive to exact dimensions or symmetry of the device, operational temperature, gate offset or its amplitude, or the exact form of the driving signal in general. Some of these dependencies are demonstrated in Fig. d based on our present measurements.

Figure illustrates the frequency dependence of the NSN turnstile. In Fig. a we show the $IV$ curves measured for $n_{g0}\simeq A_g\simeq 0.5$ at various frequencies. The frequency dependence of the current corresponding to the first plateau in measurements of the type that were shown in Fig. b is plotted in Fig. b against $ef$ in the frequency regime up to 80 MHz. The predicted $I=ef$ relation is followed closely within smaller than 1% deviations in absolute current throughout this range. We stress here that in the present measurement, using just a room temperature current amplifier, we cannot test the agreement between the prediction and the absolute value of the measured current to any higher degree than this.

Next we discuss the choice of the operating conditions of a hybrid turnstile and the potential accuracy of this device. Within the classical model of sequential single-electron tunnelling, the bias voltage $V$ across the turnstile is a trade-off: small bias leads to tunnelling events in the backward direction and large $V$ to errors due to replacement of the tunnelled charge by another one tunnelling in the forward direction through the other junction. Unwanted events of the first type occur at the relative rate of $\simeq
\exp(-\frac{eV}{k_BT_N})$, where $T_N$ is the temperature of the normal metal electrodes. Errors of the second type occur at the relative rate of $\sim \exp(-\frac{2\Delta -eV}{k_BT_N})$. The prefactor of this expression is of the order of unity in relevant cases of interest. Minimizing these errors thus yields $eV \simeq
\Delta$, which is chosen as the operation point in the experiments. At this bias point the two errors are of order $\exp(-\frac{\Delta}{k_BT_N})$. For $\Delta \simeq 200$ $\mu$eV (aluminium) and $T_N < 100$ mK that is a standard range of operation temperature, we obtain an error rate of $\sim 10^{-10}$, which is sufficiently small as compared to the requested $\sim 10^{-8}$ accuracy of the metrological source .

The analysis above neglects several types of errors. High operation frequency is one source of error: it leads to missed tunnelling events and to enhanced tunnelling in the wrong direction. These errors are suppressed approximately as $\exp(-\frac{\Delta}{2\pi f
e^2 R_T})$. For typical parameters, $\Delta$ for aluminium and $R_T
= 50$ k$\Omega$, we then request $f\ll 4$ GHz for accurate operation. Such a small value of $R_T$ seems acceptable because of sufficiently strong suppression of co-tunnelling effects in this system as will be discussed below. With exponential suppression of errors in $f$, the metrological accuracy limits then the frequency to $\sim 100$ MHz for turnstiles with aluminium as the superconductor. A possible way to increase the speed of the device is to use niobium as the superconductor, with almost an order of magnitude larger gap. With ultrasmall junctions, to keep $E_C\sim \Delta$, which is another criterion to satisfy in order not to miss any tunnelling events at the chosen bias point, one would be able to increase the frequency, and the synchronized current, by the same order of magnitude. This is an attractive yet unexplored possibility. Additional improvement, about factor of 3 increase in current, could possibly be achieved by shaping the ac gate voltage to have rectangular waveform.

Another source of potential errors is the co-tunnelling , i.e., higher order quantum tunnelling processes, which are limiting the use of short arrays in normal-metal-based devices . In a hybrid turnstile, the lowest order quasiparticle co-tunnelling errors are, however, suppressed ideally to zero within the superconducting gap as has been shown for a fully superconducting case in the past . Another process of the same order which is not suppressed by the superconducting gap is the tunnelling of Cooper pairs, also called Andreev reflection. In junctions without pinholes in the barriers, the rate $\Gamma_A$ of this tunnelling should be quite small, $\Gamma_A/\Gamma_0 \sim
\hbar/\mathcal{N} e^2 R_T$, where the effective number of the transport modes in the junction of area $A$ can be estimated as $\mathcal{N}/A \simeq 10^7$ $\mu$m$^2$ . If the charging energy is large, $E_C>\Delta$, it suppresses direct tunnelling of Cooper pairs. Analysis shows that the lowest-order process that limits the accuracy of the hybrid turnstiles with ideal superconducting electrodes is then the co-tunnelling of one electron and one Cooper pair, the rate $\Gamma_{CPE}$ of which can be estimated roughly as $\Gamma_{CPE}/\Gamma_0 \sim
(1/\mathcal{N})(\hbar/e^2 R_T)^2$. These processes are suppressed to the level mandated by the metrological accuracy, but one may need to increase the junction resistance to $R_T > 50$ k$\Omega$. Further compromise in the operation frequency would not be necessary even in this case if one benefits from the improvements in the turnstile operation discussed above.

Sub-gap leakage, due, for instance, to non-zero density of quasiparticle states within the gap, introduces a material- and fabrication-specific source of errors into our system. This effect is demonstrated by the non-vanishing slope of the $IV$ curve in the top inset of Fig. d and by an equal slope in the bias dependence of the bottom panel in Fig. d showing the current on the first plateau. Our estimates show that as far as co-tunnelling is concerned, such errors are small already in the present devices. For sequential tunnelling sub-gap leakage causes a substantial additional contribution to current, of order $10^{-3}$ in the present device. With high quality tunnel junctions, possibly by an improved fabrication process, its influence can be suppressed further. Furthermore, the separation of the current plateaus, with one bias polarity only, is ideally not sensitive to this effect, unlike the absolute value of current on a single plateau. Yet we find the sub-gap leakage as the main issue to be solved in order to realize a metrologically compatible turnstile. We would also like to point out that a series connection of a few SN junctions would present an improved version of a multijunction electron pump  in terms of leakage and co-tunnelling errors, since this device can be operated without external bias voltage.

There is a difference between the performance of the SNS and NSN structures, which was not emphasized in the discussion above. The charge transport in these systems is associated with non-trivial heat flux: based on the same strategy as discussed here a single-electron refrigerator can be realized . In this device superconductor is always heated, but under proper bias conditions heat flows out from the normal metal. Therefore, in a single island realization, an SNS configuration is more favourable, at least theoretically, as compared to the NSN turnstile. It turns out that with quite realistic parameters it is possible to refrigerate the small, thermally well isolated N island of an SNS turnstile substantially, and hence the error rates can be further suppressed. The source and drain leads can be thermalized close to the bath temperature by proper choice of geometry and materials. Cooling or overheating effects were, however, not identified definitely in the present experiment, where small errors ($\ll
10^{-3}$) could not be assessed.

One of the key advantages of the single-island turnstile, as compared to multi-island pumps is that the influence of the background charges  can be compensated by adjusting just a single DC gate voltage, e.g., to maximize the width or minimize the slope of the current plateaus. Therefore the level of the current can be increased by a relatively straightforward parallelization of several turnstiles. If an upgrade of current by, for example, an order of magnitude is necessary, DC-gate setting of each of the ten turnstiles can be adjusted individually whereafter the currents of them can be combined. The whole device can then be operated with common-to-all DC bias and AC gate drive.

**Acknowledgements** We thank Mikko Paalanen and Antti Manninen for fruitful discussions, and Antti Kemppinen for assistance in the measurements. The work was financially supported by Technology Industries of Finland Centennial Foundation and by Academy of Finland. M. M. acknowledges the Finnish Cultural Foundation for financial support.


abstract: Over the past decades, deep learning (DL) systems have achieved tremendous success and gained great popularity in various applications, such as intelligent machines, image processing, speech processing, and medical diagnostics. Deep neural networks are the key driving force behind its recent success, but still seem to be a magic black box lacking interpretability and understanding. This brings up many open safety and security issues with enormous and urgent demands on rigorous methodologies and engineering practice for quality enhancement. A plethora of studies have shown that the state-of-the-art DL systems suffer from defects and vulnerabilities that can lead to severe loss and tragedies, especially when applied to real-world safety-critical applications.
  .
  In this paper, we perform a large-scale study and construct a paper repository of 223 relevant works to the quality assurance, security, and interpretation of deep learning. We, from a software quality assurance perspective, pinpoint challenges and future opportunities towards universal secure deep learning engineering. We hope this work and the accompanied paper repository can pave the path for the software engineering community towards addressing the pressing industrial demand of secure intelligent applications.
author: Lei Ma; Felix Juefei-Xu; Minhui Xue; Qiang Hu; Sen Chen;
Bo Li; Yang Liu; Jianjun Zhao; Jianxiong Yin; Simon See
bibliography: sample-bibliography.bib
title: Secure Deep Learning Engineering:
  A Software Quality Assurance Perspective

# Introduction

In company with massive data explosion and powerful computational hardware enhancement, deep learning (DL) has recently achieved substantial strides in cutting-edge intelligent applications, ranging from virtual assistant (e.g., Alex, Siri), art design , autonomous vehicles , to medical diagnoses  – tasks that until a few years ago could be done only by humans. DL has become the innovation driving force of many next generation's technologies. We have been witnessing on the increasing trend of industry stakeholders' continuous investment on DL based intelligent system , penetrating almost every application domain, revolutionizing industry manufacturing as well as reshaping our daily life.

However, current DL system development still lacks systematic engineering guidance, quality assurance standards, as well as mature toolchain support. The *magic box*, such as DL training procedure and logic encoding (as high dimensional weight matrices and complex neural network structures), further poses challenges to interpret and understand behaviors of derived DL systems . The latent software quality and security issues of current DL systems, already started emerging out as the major vendors, rush in pushing products with higher intelligence (e.g., Google/Uber car accident , Alexa and Siri could be manipulated with hidden command . A DL image classifier with high test accuracy is easily fooled by a single pixel perturbation ). Deploying such cocooned DL systems to real-world environment without quality and security assurance leaves high risks, where newly evolved cyber- and adversarial-attacks are inevitable.

To bridge the pressing industry demand and future research directions, this paper performs a large-scale empirical study on the most-recent curated 223 relevant works on deep learning engineering from a software quality assurance perspective. Based on this, we perform a quantitative and qualitative analysis to identify the common issues that the current research community most dedicated to. With an in-depth investigation on current works, and our in-company DL development experience obtained, we find that the development of secure and high quality deep learning systems requires enormous engineering effort, while most AI communities focus on the theoretical or algorithmic perspective of deep learning. Indeed, the development of modern complex deep learning systematic solutions could be a challenge for an individual research community alone. We propose the *Secure Deep Learning Engineering* (SDLE) development process specialized for DL software, which we believe is an interdisciplinary future direction (e.g., AI, SE, security) towards constructing DL applications, in a systematic method from theoretical foundations, software & system engineering, to security guarantees. We further discuss current challenges and opportunities in SDLE from a software quality assurance perspective.

To the best of our knowledge, our work is the first study to vision SDLE, from the quality assurance perspective, accompanied with a state-of-the-art literature curation. We hope this work facilitates drawing attention of the software engineering community on necessity and demands of quality assurance for SDLE, which altogether lays down the foundations and conquers technical barriers towards constructing robust and high quality DL applications. The repository website is available at:

<https://sdle2018.github.io/SDLE/V1.1/en/Home.html>

# Research Methodology

This section shows the research questions, and discusses the detail of paper collection procedure.

## Research Questions

This paper mainly focuses on following research questions.

-   **RQ1:** What are mostly research topics and the common challenges relevant to quality assurance of deep learning?

-   **RQ2:** What is secure deep learning engineering and its future direction in perspective of quality assurance?

The RQ 1 identifies the mostly concerned topics in the research community and their common challenges, while RQ2 concerns the key activities in SDLE life cycle, based on which we discuss our vision and future opportunities.

## Data Collection Methodology

Extensive research contributions are made on deep learning over the past decades, we adopt the following procedure to select works most relevant to the theme of our paper repository.

-   We first collect papers from conferences listed on the *Computer Science Rankings* within the scope of AI & machine learning, software engineering, and security.[^1] To automate the paper collection procedure, we develop a Python-based crawler to extract paper information of each listed conference since the year 2000 and filter with keywords.

-   To further reduce the search space for relevant topics, we use keywords (e.g., deep learning, AI, security, testing, verification, quality, robustness) to filter the collected papers.

-   Even though, scraping all the listed conferences may still be insufficient, we therefore crawl outwards – extract all the related work for each keyword-filtered paper and crawl one level down of these papers.

-   This finally results in 223 papers and we manually confirmed and labeled each paper to form a final categorized list of literature.

**Paper Category and Labeling.** To categorize the selected papers, we perform paper clustering by taking into account the title, abstract, and listed keywords. Based on further discussion of all authors (from both academia and industry with AI, SE, and security background), we eventually identify four main paper categories, and seven fine-grained categories in total (see Figure ). In the next step, three of the authors manually label each paper into a target category independently, and discuss the non-consensus cases until an agreement is reached.

**The Dataset and the Trend.** Figure  shows the general trends of publication on secure deep learning research area, where the publication number (i.e., both total paper as well as in each category) dramatically increases over years. Such booming trend becomes even more obvious accompanied with the milestones of DLs (e.g., DL won ImageNet Challenge in 2012, AlphaGo defeated human championship in 2016), which is highlighted in Fig. . For the four main categories, we find the most publications are relevant to Security and Privacy (SP, 86 papers), followed by Interpretability and Understanding (IU, 65), Testing and Verification (TV, 53), and Datasets (17).

The SP category with the highest paper publication number is not surprising. Since Goodfellow et al.  posted the security issues of DLs, it attracted both the AI and security communities to escalate and burst a research competition on defending and attacking techniques. Even though, it still lacks a complete understanding on why current DL systems are still vulnerable against adversarial attacks. This draws the attention of researchers on interpreting and understanding how DL works, which would be important for both application and construction of robust DLs. As the recent emerging investment blowout in DL applications to safety-critical scenarios (e.g., autonomous driving, medical diagnose), its software quality has become a big concern, where researchers find that the different programming paradigm of DL makes existing testing and verification techniques unable to directly handle DLs . Therefore, we have observed that many recent works are proposing novel testing and verification techniques for DLs, from testing criteria, test generation techniques, test data quality evaluation, to static analysis. Meanwhile, the dataset benchmarks of different DL application domains emerge to grow as well , in order to facilitate the study of solving domain-specific problems by DLs (e.g., image classification, 3D object recognition, autonomous driving, skin disease classification).

**Common Issues.** In contrast to traditional software of which the decision logic is mostly programmed by human developers, deep learning adopts a data-driven programming paradigms. Specifically, a DL developer's major effort is to prepare the training data (including knowledge to resolve a task) and neural network architecture, after which the decision logic is automatically obtained through the training procedure. On one hand, this paradigm reduces the burden of a developer who manually crafts the decision logic. On the other hand, for a DL developer, the logic training procedure is almost like a magic-box driven by an optimization technique. Due to the decision logic of DL is encoded into a DNN with high dimensional matrices, the interpretation and understanding, training procedure, as well as the obtained decision logic are all very difficult , which could be a root cause and a common challenge among all categories. For example, without completely understanding the decision logic of DL, it is hard to know in what particular case an adversarial attack could penetrate, and how we could defend against such attacks. In the case of testing, extensive studies are performed on analysis of traditional software bugs, their relations to software development activities, as well as techniques for defect detection. However, a comprehensive empirical study and understanding on why DL bugs occur still could not be well explained, let alone the root case analysis.

# Secure Deep Learning Engineering Life Cycle

Due to the fundamental different programming paradigms of deep learning and traditional software, the secure deep learning engineering practice and techniques are largely different with traditional software engineering, although the major development phases could still be shared.

Figure  shows the key development phases of SDLE. In the rest of this section, we first describe each of the key development phases, their uniqueness and difference compared with traditional practices in software engineering, and then we discuss the security issues in current SDLE. In the next section, we explain the quality assurance necessity in SDLE life cycle, and highlight the challenges and opportunities.

**Requirement Analysis.** Requirement analysis investigates the needs, determines, and creates detailed functional documents for the DL products. DL-based software decision logic is learned from the training data and generalized to the testing data. Therefore, the requirement is usually measured in terms of an expected prediction performance, which is often a statistics-based requirement, as opposed to the rule-based one in traditional SE.

**Data-Label Pair Collection.** After the requirements of the DL software become available, a DL developer (potentially with domain experts for supervision and labeling) tries to collect representative data that incorporate the knowledge on the specific target task. For traditional software, a human developer needs to understand the specific task, figures out a set of algorithmic operations to solve the task, and programs such operations in the form of source code for execution. On the other hand, one of the most important sources of DL software is training data, where the DL software automatically distills the computational solutions of a specific task.

**DNN Design and Training Program Implementation.** When the training data become available, a DL developer designs the DNN architecture, taking into account of requirement, data complexity, as well as the problem domain. For example, when addressing a general purpose image processing task, convolutional layer components are often included in the DNN model design, while recurrent layers are often used to process natural language tasks. To concretely implement the desired DNN architecture, a DL developer often leverages an existing DL framework to encode the designed DNN into a training program. Furthermore, he needs to specify the runtime training behaviors through the APIs provided by the DL framework (e.g., training epochs, learning rate, GPU/CPU configurations).

**Runtime Training.** After the DL programming ingredients (i.e., training data and training program) are ready. The runtime training procedure starts and systematically evolves the decision logic learning towards effectively resolving a target task. The training procedure and training program adjustment might go back-and-forth several rounds until a satisfying performance is achieved. Although the training program itself is often written as traditional software (e.g., in `Python`, `Java`), the obtained DL software is often encoded in a DNN model, consisting of the DNN architecture and weight matrices. The training process plays a central role in the DL software learning, to distill knowledge and solution from the sources. It involves quite a lot of software and system engineering effort to realize the learning theory to DL software (see Figure ) over years.

**Testing & Verification.** When the DNN model completes training with its decision logic determined, it goes through the systematic evaluation of its generality and quality through testing (or verification). Note that the testing activity in the AI community mainly considers whether the obtained DL model generalizes to the prepared test dataset, to obtain high test accuracy. On the other hand, the testing activity (or verification) in SDLE considers a more general evaluation scope, such as generality, robustness, defect detection, as well as other nonfunctional requirement (e.g., efficiency). The early weakness detection of the DL software provides valuable feedback to a DL developer for solution enhancement.

**Deployment.** A DL software passed the testing phase reaches a certain level of quality standard, and is ready to be deployed to a target platform. However, due to the platform diversity, DL framework supportability, and computation limitations of a target device, the DL software often needs to go through the platform calibration (e.g., compression, quantization, DL framework migration) procedure for deployment on a target platform. For example, once a DL software is trained and obtained on the `Tensorflow` framework, it needs to be successfully transformed to its counterpart of `TensorflowLite` (resp. `CoreML`) framework to `Android` (resp. `iOS`) platform. It still needs to go through on device testing after deployment, and we omit the testing phase after deployment for simplicity.

**Evolution and Maintenance.** After a DL product is deployed, it might experience the procedure of modification for bug correction, performance and feature enhancements, or other attributes. The major effort in evolution and maintenance phases relies on the manually revision on design, source code, documentation, or other software artifacts. On the other hand, DL software focuses more on comprehensive data collection, DL model continuous learning (e.g., re-fitting, retro-fitting, fine tuning, and re-engineering).

**Security Issues in DL.** The current practice of security in deep learning has fallen into the trap that many other domains have experienced. Almost every month new attacks are identified  followed by new countermeasures  which are subsequently broken , and so on ad-infinitum. There is a broad and pressing need for a frontier-level effort on trustworthiness and security in DL to break this cycle of attacks and defenses. We have a unique opportunity at this time—before deep learning is widely deployed in critical systems—to develop the theory and practice needed for robust learning algorithms that provide rigorous and meaningful guarantees. If we rethink the SDLE life cycle (see Figure ), security vulnerabilities can happen in almost every step. For instance, for the training related steps such as *Requirement Analysis, Data-Label Pair Collection* and *DNN design and training*, poisoning attacks can easily happen via manipulating training data. In the testing related steps, such as *testing & verification*and *deployment*, evasion attacks can take place by perturbing the testing data slightly (e.g. adversarial examples). In addition, when deploying the DL software to different platforms or with different implementation frameworks, there will always be opportunities for adversaries to generate attacks from one to the other.

We believe many of these security issues are highly intertwined the quality of current DL software, lacking systematic quality assurance solutions over the entire SDLE process which is largely missed in research works as described in the next section.

# Towards Future Quality Assurance of SDLE

Over the past decades, software quality assurance discipline has been well-established for traditional software, with many experiences and practices widely applied in software industry. However, the fundamentally different programming paradigm and decision logic representation of DL software make existing quality assurance techniques unable to be directly applied, forcing us to renovate the entire quality assurance procedure for SDLE. In this section, we pose our vision and challenges on quality assurance in SDLE to guide future research.

From the very beginning of SDLE, we need to rethink how to accurately define, specify, and document the of DL software requirement, especially for the functional requirements. This leaves us a question whether we should follow a statistical based approach, a rule based approach, or their combination, which has not been well investigated yet.

The training data play a key role in shaping the learning process and DL decision logic. However, most current research treats the training data as high quality for granted, without a systematic quality control, inspection and evaluation process. As poisoning attacks show, many incorrect behaviors and security issues could be introduced with the maliciously tweaked training data. How to select the suitable size while representative data would be an important question. In addition, data supervision and labeling process is also labor intensive and error prone. For example, ImageNet dataset contains more than one million general purpose images. We also need to provide assistance and quality control for the labeling procedure.

It becomes even more challenging, when it comes to the training program implementation and runtime training. Most state-of-the-art DL frameworks are implemented as traditional software on top of the DL software stack. Even the learning theory is perfect, it still has a big gap to transfer such ideally designed DL models to a DL application encoded on top of the DL framework. One big challenge is how to ensure the software stack (e.g., hardware drivers, DL library, DL framework) correctly implements the learning algorithm. Another challenge is to provide useful interactive support of the training process. Most current DL training procedure only shows training loss (accuracy), validation loss (accuracy), which is mostly a black box to a DL developer. When, the training procedure goes beyond expectation, the root cause analysis becomes extremely difficult, which may come from the DL architecture issue, training program implementation issue, or the hardware configuration issue. Hence, the software engineering community needs to provide the novel debugging, runtime monitoring, and profiling support for the training procedure, which is involved with non-determinism and runtime properties hard to specify.

The large input space has already been a challenge for testing and verifying traditional software. Such a challenge is further escalated for DL software, due to its high dimensional input space and the internal latent space. Even though, traditional software testing has already explored many testing criteria as the goal to guide testing. How to design a suitable testing criteria to capture the testing confidence still remains unclear. Even with some preliminary progress on testing criteria designed for DLs , there are many more testing issues needed to be addressed, such as how to effectively generate tests , how to measure the test data quality , and how to test DL robustness and vulnerabilities .

Further DL challenge comes up with current deployment process: (1) target device computation limitations, and (2) DL framework compatibility across platforms. The DL software is mostly developed and trained on severs or PCs with GPU support. When it needs to be deployed on a mobile device or edge device with limited computation power, the DL software must be compressed or quantized for computation/energy efficiency, which could introduce defects. How to ensure the quality and detect the potential issues during this process is an important problem. In addition, the current DL frameworks might not always be supported by different platforms. For example, the Tensorflow is not directly supported by Android or iOS, and how to make DL software cross-platform compatible would be an important direction. Finally the quality assurance concerns in DL software evolution and maintenance are mostly focused on avoiding introducing defects during change, which might rely on regression testing. However, how to effectively evolve the DL software still remains unknown, which we leave as an open question for further study.

# Conclusion

Considering deep learning is likely to be one of the most transformative technologies in the 21st century, it appears essential that we begin to think how to design fully fledged deep learning systems under a well-tested development discipline. This paper defines the secure deep learning engineering and discusses the current challenges, opportunities, and puts forward open questions from the quality assurance perspective, accompanied with a paper repository. We hope our work can inspire future studies towards constructing robust and high quality DL software.

[^1]: <http://csrankings.org/#/index?all>


abstract: Many viruses evolve rapidly. For example, hemagglutinin of the H3N2 influenza A virus evolves to escape antibody binding. This evolution of the H3N2 virus means that people who have previously been exposed to an influenza strain may be infected by a newly emerged virus. In this paper, we use Shannon entropy and relative entropy to measure the diversity and selection pressure by antibody in each amino acid site of H3 hemagglutinin between the 1992–1993 season and the 2009–2010 season. Shannon entropy and relative entropy are two independent state variables that we use to characterize H3N2 evolution. The entropy method estimates future H3N2 evolution and migration using currently available H3 hemagglutinin sequences. First, we show that the rate of evolution increases with the virus diversity in the current season. The Shannon entropy of the sequence in the current season predicts relative entropy between sequences in the current season and those in the next season. Second, a global migration pattern of H3N2 is assembled by comparing the relative entropy flows of sequences sampled in China, Japan, the USA, and Europe. We verify this entropy method by describing two aspects of historical H3N2 evolution. First, we identify 54 amino acid sites in hemagglutinin that have evolved in the past to evade the immune system. Second, the entropy method shows that epitopes A and B in the top of hemagglutinin evolve most vigorously to escape antibody binding. Our work provides a novel entropy-based method to predict and quantify future H3N2 evolution and to describe the evolutionary history of H3N2.
author: $\mbox{Keyao Pan}^1$ and $\mbox{Michael W. Deem}^{1,2}$

Department of $^1$Bioengineering and $^2$Physics & Astronomy, Rice University,
6100 Main Street, Houston, TX 77005
bibliography: references.bib
title: Quantifying Selection and Diversity in Viruses by Entropy Methods, with Application to the Hemagglutinin of H3N2 Influenza

**Keywords:** Virus, Influenza, Evolution, Prediction, Selection, Diversity, Entropy

# Introduction

A common strategy by which viruses evade pressure from the immune system is to evolve and change their antigenic profile. Viruses with a low evolutionary rate that infect only humans, such as the small pox virus , can be effectively controlled by vaccinating the human population. By contrast, viruses with a high evolutionary rate, such as HIV, hepatitis B, and influenza A, resist being eliminated by the immune system by generating a plethora of mutated virus particles and causing chronic or repeated infection. In this study, we take subtype H3N2 influenza A virus as a model evolving virus. Influenza A virus circulates in the human population every year, typically causing 3–5 million severe illnesses and 250,000–500,000 fatalities all over the world . Hemagglutinin (HA) and neuraminidase (NA) are two kinds of virus surface glycoproteins encoded by the influenza genome. The subtype of influenza is jointly determined by the type of hemagglutinin ranging from H1 to H16, and that of neuraminidase ranging from N1 to N9. On the surface of the virus membrane, HA exists as a cylindrical trimer containing three HA monomers, and each monomer comprises two domains, HA1 and HA2. Hemagglutinin is also a key factor in virus evolution, because it is the major target of antibodies, and HA escape mutation changes the antigenic character of the virus presented to the immune system. The H3N2 virus causes the largest fraction of influenza illness. H3 hemagglutinin is under selection by the immune response mainly on the five epitope regions in the HA1 domain , labeled epitopes A to E, as shown in Figure . The immune pressure and the escape mutation drive the evolution of the H3N2 virus. The underlying mutation rate of the HA gene is $1.6 \times 10^{-5}/\mbox{amino acid position}/\mbox{day}$ , measured using the method modified from that in an earlier study on the HA mutation rate . Note that the mutation rate does not necessarily equal to the evolution rate, or the fixation rate. The mutation rate equals to the evolutionary rate only if the evolution is neutral. The non-neutrality of the HA evolution is shown in the Results section. Evolution of the hemagglutinin viral protein causes occasional mismatch between the virus and the vaccine and decreases vaccine effectiveness . As more amino acid substitutions are introduced into influenza sequences, the antigenic characteristics of influenza strains drift away , and influenza epidemic severity of subtype H1N1 and subtype H3N2 increases.

The H3N2 virus has a distinguished evolutionary history, largely affected by the immune pressure. The H3N2 virus emerged in the human population in 1968 and has been circulating in the population since 1968. The phylogenetic tree of H3 hemagglutinin since 1968 has a linear topology in which most sequences are close to the single trunk of the tree, and the lengths of the branches are short . Historical hemagglutinin sequences fall into a series of clusters, each of which has similar genetic or antigenic features and circulates for 2–8 years before being replaced by the next cluster . The evolution of different amino acid positions of hemagglutinin shows a remarkable heterogeneity: a subset of positions undergo frequent change, while some positions are conserved . This heterogeneity is quantified by the Shannon entropy at each position of the amino acid sequence of hemagglutinin . Shannon entropy has been used to locate protein regions with high diversity, such as the antigen binding sites of T-cell receptors . Shannon entropy has been used to identify antibody binding sites, or epitopes, which are under immune pressure and so are rapidly evolving . The heterogeneity of amino acid substitution suggests that point mutations randomly occurring in distinct positions have different contributions to the virus fitness.

The selection pressure on the H3N2 virus to evolve is reflected in the difference between the H3 hemagglutinin sequences in two consecutive seasons. We consider Northern hemisphere strains. When the epidemic initiates in a new season, we assume that each position of an HA sequence inherits the amino acid from a sequence of the previous season or has a different amino acid due to random mutation and selection. This assumption comes from the fact that the H3N2 virus circulating in each influenza season migrates from a certain geographic region in which the virus is preserved between two influenza seasons . In the absence of selection, the histogram of the 20 amino acid usage in one position in the current season is similar to that in the same position in the previous season except for changes due to the small random mutation rate. The difference between the two histograms beyond that expected due to mutation quantifies selection.

Synthesizing these factors, we introduce an entropy method to describe the evolution of influenza. The entropy method extracts an evolutionary pattern from aligned sequences. Shannon entropy quantifies the amount of sequence information in each position of aligned sequences . The sequence information reflects the variation, which is equivalently diversity, in each position, and so Shannon entropy has been used to measure the diversity in each position . Shannon entropy has also been used to measure the structural conservation in the protein folding dynamics . See for a detailed review of the applications of Shannon entropy. On the other hand, relative entropy measures gain of sequence information at each position and requires a background amino acid frequency distribution . Relative entropy was also used as a sequence conservation measure to detect functional protein sites . Further, a dimension reduction technique using relative entropy has identified sectors in proteins . As an extension of these previous works, we apply Shannon entropy and relative entropy to jointly measure two quantities in each position: sequence information in one season and gain of sequence information from one season to the next season. Simultaneous analysis of Shannon entropy and relative entropy sheds light on the evolutionary pattern of the H3N2 virus evolution when data from multiple seasons are available. In the HA1 domain, positions in the epitope regions have increased Shannon entropy, and this feature was applied to locate the epitopes of H1 hemagglutinin . We here use Shannon entropy to quantify the virus diversity in each amino acid position in each season. The entropy relative to the previous season is also used to analyze the evolution of the HA1 domain in one single season and to quantify the selection pressure on the virus in each amino acid position in each season. The selection and the virus diversity are two significant state variables determining the dynamics of evolution.

The article is organized as follows. The Materials and Methods section presents the data used in this work and details of the entropy model. The Results section uses Shannon entropy of the sequence in one season to predict the evolution quantified by relative entropy from this season to the next season. Results are also presented for the flow of virus migration between the four geographic regions of China, Japan, the USA, and Europe. In the Result section, we demonstrate the entropy method in two applications, the results of which agree with prior knowledge on H3N2 evolution. Finally, we discuss our results and present our conclusions.

# Materials and Methods

## Sequence Data

The hemagglutinin sequences considered in this work are the amino acid sequences of the hemagglutinin of human influenza A H3N2 virus. We only use Northern Hemisphere sequences because 90% of the world population lives in Northern Hemisphere. The influenza season in Northern Hemisphere is defined as the time interval from October in one year to September in the next year. We downloaded 5309 Northern Hemisphere sequences labeled with month of collection from the NCBI Influenza Virus Resource on 16 January 2011. Sequences too short to cover residues 1–329 of hemagglutinin were removed, and the remaining sequences were trimmed to only keep residues 1–329 in the HA1 domain. Any sequence with an undefined amino acid in residues 1–329 was removed. We consider 18 seasons from 1992–1993 to 2009–2010 during which most available sequences were collected. In total, we preserved and aligned 4292 sequences in these 18 seasons containing amino acids 1–329.

## Histograms of 20 Amino Acids

The first step is to quantify the alignment of the amino acid sequences. The aligned historical H3 hemagglutinin sequences form a matrix $\mathbf{A}$ with 4292 rows and 329 columns. The element $\mathbf{A}_{l,j}$ denotes the identity of the amino acid in sequence $l$ and position $j$. The 4292 sequences were clustered into 18 groups by the seasons of sampling from the 1992–1993 season to the 2009–2010 season. Note that most of the sequences before the 1992–1993 season were not labeled with month of collection and are excluded from this study. We denote by $i=0,1,\dots,17$ the seasons between 1992–1993 and 2009–2010. For position $j$ in season $i$, the relative frequency of each amino acid $k$, $f\left(k,i,j\right), k=1,\dots,20$, was calculated from the vector $\mathbf{A}_{\vec{l}\left(i\right),j}$ where the index array $\vec{l}\left(i\right)$ holds the indices of sequences sampled in season $i$.

## Shannon Entropy as Diversity

The Shannon entropy is one useful quantification of the diversity in single position. Large Shannon entropy has the physical meaning that the amino acid in the given position is prone to be substituted. This physical meaning was also applied in . The diversity at a single position takes the format of Shannon entropy because of the sensitivity of Shannon entropy to diversity.

This physical meaning of the Shannon entropy does not necessarily involve the joint frequency distributions for two and more positions, and we do not consider the joint frequency in the manuscript. Rather, we define the diversity only in the level of single amino acid position. Consequently, the defined diversity is additive for a number of positions. The idea of adding diversity in each position of the sequence comes from classic works such as , which added the Shannon entropy in each position to measure the total diversity in an aligned binding site.

Therefore, diversity of the virus in each position in each season is represented by the Shannon entropy that quantifies the amount of information in the histogram or distribution under study. For the sequences sampled in all the seasons, positions with high evolutionary rate have a higher Shannon entropy compared to the conserved positions . The sequences in each season are assumed to be collected concurrently. The Shannon entropy is a quantification of diversity of amino acids in one position, and so the diversity in position $j$ in season $i$ is calculated from the histogram $\mathbf{f}\left(i,j\right) = \left[f\left(1,i,j\right), \dots, f\left(20,i,j\right)\right]^\mathrm{T}$ by Shannon entropy $$\label{eq:diversity}
D_{i,j} = -\sum_{k=1}^{20} f\left(k,i,j\right) \log f\left(k,i,j\right)$$ in which $k = 1, \dots, 20$ is the identity of the amino acid in position $j$ in season $i$.

## Relative Entropy as Selection Pressure

Selection in each position $j$ in season $i$ is reflected by the significant difference between the 20-bin histogram in the current season $\mathbf{f}\left(i,j\right)$ and that in the previous season $\mathbf{f}\left(i-1,j\right)$. In the absence of selection, random mutation and genetic drift are the dominant forces generating $\mathbf{f}\left(i,j\right)$ from $\mathbf{f}\left(i-1,j\right)$. In each position, random mutation creates a slightly modified histogram, from which amino acids are randomly chosen to appear in season $i$ by the effect of genetic drift.

The source of random mutation is the spontaneous error of the RNA polymerase replicating the influenza virus RNA. The random mutation rate in different regions of hemagglutinin is thought to be homogeneous, regardless whether the regions are in antigenic sites or not . Therefore random mutation is modeled as a Poisson process $\mathbf{M}\left(\mu, g\left(k\right)\right)$ equally affecting all the positions. Here $\mu$ is the mutation rate of influenza A virus that equals to $5.8 \times 10^{-3}/\mathrm{residue}/\mathrm{season}$ , and $g\left(k\right), k=1,\dots,20$ is the relative frequency of each amino acid in the whole alignment $\mathrm{A}$. The probability that the original amino acid $t$ mutates to amino acid $u$ is $$\label{eq:mutation}
\mathbf{M}_{u,t} \left(\mu, g\right) = \frac{\mu g\left(u\right)}{1 - g\left(t\right)}.$$ So after mutating for one season, the histogram in position $j$ in season $i-1$ is obtained by $$\label{eq:background}
\hat{\mathbf{f}}\left(i,j\right) = \mathbf{M} \left(\mu, g\right) \mathbf{f}\left(i-1,j\right).$$ This histogram serves as the background distribution for season $i$ from which the sequences in season $i$ are built.

If selection is absent, the effect of genetic drift is to create sequences in the current season by randomly choosing amino acids in each position from a background distribution $\hat{\mathbf{f}}\left(i,j\right)$. We denote by $N_i$ the number of sequence in season $i$. The probability that $N_i$ amino acids in position $j$ have the histogram $\mathbf{f}\left(i,j\right)$ is $$\label{eq:sample_hist}
\mathrm{Pr}\{\mathbf{f}\left(i,j\right)\} \approx \exp\left(-N_i S_{i,j}\right)$$ where $$\label{eq:selection}
S_{i,j} = \sum_{k=1}^{20} f\left(k,i,j\right) \log\frac{f\left(k,i,j\right)}{\hat{f}\left(k,i,j\right)}$$ is the relative entropy between the observed histogram, $f\left(k,i,j\right)$, and the background histogram, $\hat{f}\left(k,i,j\right)$ .

The null hypothesis that selection is absent in the evolution is rejected if the relative entropy $S_{i,j}$ is great enough such that the probability in Equation is less than 0.05, that is, the relative entropy $S_{i,j}$ is greater than $-\log\left(0.05\right) / N_i$ in season $i$. Note that the majority of residues were stable in most of the seasons, and in this case the relative entropy is $S_{i,j} = \log\left(1/\left(1-\mu\right)\right) \approx \mu$. To avoid classifying these stable residues erroneously as positions under selection, a proper threshold of relative entropy needs to be larger than the mutation rate $\mu$. Additionally, a fraction $\lambda$ of the circulating HA1 sequences were not deposited in the database because of the sampling bias of the HA1 sequences. In an extreme case, in a stable position $j$ with the real histogram of 20 amino acids $\left[1-\lambda,\lambda,\dots,0\right]^\mathrm{T}$ in all the seasons, the histograms of the sequences sampled in two consecutive seasons $i-1$ and $i$ are $\mathbf{f}\left(i-1,j\right) = \left[1,0,\dots,0\right]^\mathrm{T}$ and $\mathbf{f}\left(i,j\right) = \left[1-\lambda,\lambda,\dots,0\right]^\mathrm{T}$, respectively, and so the relative entropy introduced by the sampling bias is $$S^\mathrm{bias} \approx \left(1-\lambda\right) \log\frac{1-\lambda}{1-\mu} + \lambda \log\frac{\lambda}{\mu/19}$$ in spite of the absence of selection in position $j$ in season $i$. The relative entropy $S^\mathrm{bias}$ equals to 0.1 if a sampling bias $\lambda = 2.5\%$ exists in the HA1 database sequences. We fix the threshold of the relative entropy in season $i$ to $$\label{eq:threshold}
S_i^\mathrm{thres} = \max\left\{-\log\left(0.05\right) / N_i,\left(1-\lambda\right) \log\frac{1-\lambda}{1-\mu} + \lambda \log\frac{\lambda}{\mu/19}\right\} \approx \max\left\{3/N_i, 0.1\right\}.$$ The numbers of collected HA1 sequences $N_i$ were fewer than 30 only in the 1995–1996 season $\left(i=3\right)$ with $N_3 = 25$. The thresholds $S_3^\mathrm{thres} = 0.12 > 0.1$ in the 1995–1996 season due to the small numbers of HA1 sequences. In all the other 17 seasons, the numbers of sequences $N_i$ were greater than 30, and so the thresholds $S_i^\mathrm{thres} = 0.1$.

# Results

In this section, we show the positive correlation between the Shannon entropy in season $i$ and the relative entropy from season $i$ to season $i+1$. This correlation means that the larger the virus diversity in one season, the higher the virus evolutionary rate from this season to the next season. We draw the H3N2 migration pattern by comparing the relative entropy. The migration pattern reveals a novel migration path from the USA to Europe and shows that the virus evolutionary rate is higher in the epicenter, China, than in the migration paths. We also demonstrate the entropy method in two applications. First, we compute average Shannon entropy and relative entropy in each position over the past 17 seasons to identify positions under selection pressure. Second, we compare Shannon entropy and relative entropy in epitope regions to find the contribution of each epitope to the H3N2 evolution. Results of these two applications agree with previous studies and additionally show the heterogeneity of the selection pressure over different amino acid positions of hemagglutinin, with increased pressure in the epitopes, as well as the dominance of epitopes A and B.

## Correlation between Shannon entropy and relative entropy

Relative entropy $S_{i+1,j}$ in amino acid position $j$ from season $i$ to season $i+1$ linearly increases with Shannon entropy $D_{i,j}$ in position $j$ in season $i$. For the sequences sampled from the 1992–1993 season ($i = 0$) to the 2008–2009 season ($i = 16$), $329 \times 17 = 5593$ ordered pairs $(D_{i,j}, S_{i+1,j})$ are calculated. All except two pairs fall into 8 bins in which the values of $D_{i,j}$ belong to 8 intervals $[0, 0.1)$, $[0.1, 0.2)$, $\dots$, $[0.7, 0.8)$, respectively. The first bin with $D_{i,j}$ in $[0, 0.1)$ is discarded because it contains numerous conserved amino acid positions. The values of $S_{i+1,j}$ are averaged respectively in each of the 7 remaining bins. As described in Figure , average relative entropy in each bin shows positive correlation with midpoints of the $D_{i,j}$ interval, $R^2 = 0.70$. An amino acid position $j$ with high Shannon entropy $D_{i,j}$ in season $i$ is expected to present high relative entropy $S_{i+1,j}$ from season $i$ to $i+1$. The evolution in position $j$ from season $i$ to $i+1$ quantified by relative entropy is therefore predicted using the mean and standard error of $S_{i+1,j}$ in the bin chosen by $D_{i,j}$ in season $i$.

Positive correlation is also observed between the mean values of $D_{i,j}$ and $S_{i+1,j}$ in a variety of positions $j$ in each season $i$. In each season between 1992–1993 ($i = 0$) and 2008–2009 ($i = 16$), we average the Shannon entropy $D_{i,j}$ and relative entropy $S_{i+1,j}$ over the positions $j$ with $D_{i,j} > 0.1$. The data point with $i=1$, $\left(0.22, 1.38\right)$, has a large standard error of the relative entropy $S_{i+1,j}$ and is excluded in the analysis below. The remaining average Shannon entropy $\langle D \rangle_{i}$ ($i = 0, 2, \dots, 16$) correlates with average relative entropy $\langle S \rangle_{i+1}$ ($i = 0, 2, \dots, 16$) with $R^2 = 0.50$, as shown in Figure . A least squares fit gives $\langle S \rangle_{i+1} = 1.82 \langle D \rangle_{i} - 0.23$. Thus, the expected average relative entropy from the current season $i$ to the next season $i+1$ can be calculated from the average Shannon entropy $\langle D \rangle_{i}$ in the current season $i$.

The above relationships between Shannon entropy and relative entropy cannot be generated by a neutral evolution model. To demonstrate this result, we create an ensemble of 1000 identical sequences with 50 amino acid positions. Each iteration of the model simulates H3N2 evolution during one season. In each iteration, the number of mutated amino acids $N_\mathrm{mut}$ in each sequence follows a Poisson distribution with mean $\mu = 2.0$, which is the annual substitution rate in history. The $N_\mathrm{mut}$ mutated positions are then randomly assigned in the corresponding sequence. We randomly select $p_\mathrm{cut} = 10\%$ of the sequences to build the sequence ensemble in the next iteration. The Shannon entropy $D_{i,j}$ and relative entropy $S_{i+1,j}$ generated in iteration $i =$ 51–100 are processed using the same method as for H3 sequences in history. First, as shown in Figure , no increasing trend appears in the means of $S_{i+1,j}$ in the 7 bins from $[0.1, 0.2)$ to $[0.7, 0.8)$. Second, no correlation ($R^2 = 0.003$) is observed between $\langle D \rangle_{i}$ and $\langle S \rangle_{i+1}$, see Figure . When we change the parameters $\mu$ between 1.0 and 10 and $p_\mathrm{cut}$ between 1% and 100% in the algorithm, the simulation still does not yield the visible increasing trend in Figure or the correlation observed in Figure . As a result, we conclude that neutral evolution alone is not able to generate the pattern between Shannon entropy and relation entropy of H3 sequences. It was previously shown that the fixation rate of H3N2 evolution cannot be explained only by neutral evolution . In this study, the monotonically increasing linear relationship between relative entropy and Shannon entropy in Figure and suggests that selection pressure substantially contributes to H3N2 evolution.

## Annual Virus Migration

The entropy method is also used to analyze the global migration pattern of the virus. Most of the Northern Hemisphere H3 sequences were collected in East-Southeast Asia, the USA, and Europe. East-Southeast Asia is suggested to be the reservoir of the annual H3N2 epidemic . To increase the geographic resolution in East-Southeast Asia, we use two regions, China and Japan, as the representative of East-Southeast Asia, because each of these regions has a population over 50 million and has a consistent time series of H3 sequence data from the 2001–2002 to the 2007–2008 season.

We select the sequences in four geographic regions that are China, Japan, the USA, and Europe in seven seasons from 2001–2002 to 2007–2008. In all the six pairs of consecutive seasons, we calculate for each region four average relative entropy values of the whole sequence. These four average relative entropy values are calculated using the sequences in each of the four regions in the previous season as the reference. The results are shown in Table . Sequences collected in China in the previous season yield the minimum relative entropy to the sequences in the current season collected in China ($p < 2.1 \times 10^{-5}$, Wilcoxon signed-rank test), Japan ($p < 0.0049$, Wilcoxon signed-rank test), and the USA ($p < 0.0012$, Wilcoxon signed-rank test). Sequences in the USA in the previous season have the minimum relative entropy to the sequences in Europe in the current season ($p < 0.15$, Wilcoxon signed-rank test). Relative entropy data in Table imply the virus migration from China, as the geographic reservoir, to Japan and the USA and suggest a migration from the USA to Europe. The result in Table also implies that the H3N2 virus circulating in China seed the virus in China, Japan, and the USA in the next season, and the virus in the USA probably seed the virus in Europe in the next season.

|  |  |  |  |  |  |
|:---|:---|:---|:---|:---|:---|
|  | Region of the |  |  |  |  |
|  | current season | China | Japan | USA | Europe |
| Region of the |  |  |  |  |  |
| previous season |  |  |  |  |  |
| China |  | **0.057** | **0.040** | **0.044** | 0.064 |
|  |  |  |  |  | (0.0017) |
| Japan |  | 0.114 | 0.094 | 0.076 | 0.059 |
|  |  | ($2.1 \times 10^{-5}$) | (0.0049) | (0.0012) | (0.032) |
| USA |  | 0.105 | 0.087 | 0.070 | **0.056** |
|  |  | ($2.1 \times 10^{-10}$) | ($3.3 \times 10^{-8}$) | ($6.4 \times 10^{-5}$) |  |
| Europe |  | 0.135 | 0.115 | 0.094 | 0.074 |
|  |  | ($3.8 \times 10^{-9}$) | ($3.4 \times 10^{-6}$) | ($4.8 \times 10^{-7}$) | (0.15) |

The relative entropy between hemagglutinin sequences in the different regions in the current influenza season and sequences in these regions in the previous season. The minimum relative entropy in each column is marked in bold. The $p$ values of the Wilcoxon signed-rank test between the minimum relative entropy and other relative entropy values in the same column are in the parentheses. Hemagglutinin sequences were collected from four geographic regions: China, Japan, the USA, and Europe. Seven seasons from 2001–2002 to 2007–2008 are used here. The relative entropy values listed in this table are averaged for all the sites and all the six pairs of consecutive seasons. These results imply that the H3N2 viruses in China, Japan, and the USA migrate from China, while the H3N2 virus in Europe migrates from USA.

Comparison of the relative entropy data in Table in the H3N2 reservoir and migration paths also reveals the H3N2 virus migration pattern. Using the Wilcoxon sign-rank test, the relative entropy data of the H3 hemagglutinin in China in two consecutive seasons is significantly greater than those in the three migration paths: from China to Japan ($p = 0.035$), from China to the USA ($p = 0.0030$), and from the USA to Europe ($p = 0.0017$). The relative entropy data, therefore, confirms China as the H3N2 reservoir and implies that novel H3N2 viruses are emerging in China, not during the migration process.

## Positions under Selection

The values of diversity as Shannon entropy $D_{i,j}$ and selection as relative entropy $S_{i,j}$ are available for the sequences collected from the 1993–1994 season to the 2009–2010 season. First we apply the mean field approximation to remove the variation of selection and diversity over the time, and only consider the variation of Shannon entropy and relative entropy in different positions and regions over the past 17 seasons. A profile of the pattern of Shannon entropy and relative entropy in position $j$ comprises the average selection, the number of seasons under selection, and the average diversity. The average selection $\bar{S}_j$ is expressed by the mean of relative entropy in each position over the 17 seasons $$\bar{S}_j = \frac{1}{17}\sum_{i=1}^{17} S_{i,j}$$ and is displayed in Figure . The number of seasons under selection in each position $j$ is calculated by $$\label{eq:n_select}
N_j = \sum_{i=1}^{17} H\left(S_{i,j} - S_i^\mathrm{thres}\right)$$ where $H$ is the Heaviside step function. The numbers are shown in Figure . The average diversity $\bar{D}_j$ in each position is calculated by averaging the Shannon entropy over the 17 seasons from 1993–1994 to 2009–2010 $$\bar{D}_j = \frac{1}{17}\sum_{i=1}^{17} D_{i,j}$$ and is displayed in Figure .

Figure presents the distribution of the selection. Around 76% of the amino acid positions 1–329 of hemagglutinin have an average selection close to zero and fall into the leftmost bin. The numbers of seasons when selection $S_{i,j}$ in these positions were greater than the threshold level $S_i^\mathrm{thres}$ are shown in Figure . The average diversities $\bar{D}_{i,j}$ in all the positions are shown in Figure . If position $j$ is under selection with $S_{i,j} > S_i^\mathrm{thres}$ in greater than two of the 17 seasons between 1993–1994 and 2009–2010, or $N_j > 2$, this position $j$ is classified as a position under selection in the evolutionary history of H3N2 virus. The 54 positions with $S_{i,j} > S_i^\mathrm{thres}$ in greater than two seasons are listed in Table .

Patterns of selection and diversity similar to those observed in historical sequences in Figure are generated by a Monte Carlo simulation model, as displayed in Figure S1. The basis of the Monte Carlo simulation is that antibody binds to one of two epitope regions on the surface of the HA1 domain , and the dominant epitope bound by antibody is under immune pressure and undergoes a higher substitution rate . The detailed description and discussion of the Monte Carlo model is in the Appendix.

| Region          | Amino acid positions                    |
|:----------------|:----------------------------------------|
| Epitope A       | 122 124 126 131 133 137 140 142 144 145 |
| Epitope B       | 128 155 156 157 158 159 189 192 193 197 |
| Epitope C       | 45 50 273 275 278 312                   |
| Epitope D       | 121 172 173 201 207 219 226 227 229 246 |
| Epitope E       | 57 62 75 78 83 92 260 262               |
| Out of epitopes | 3 5 25 33 49 106 202 222 225 271        |

Amino acid positions $j$ under selection. To be included, the positions must be under selection, $S_{i,j} > S_i^\mathrm{thres}$, in greater than two seasons.

## Comparison of Different Regions

A human antibody binds to five epitopes in the H3 hemagglutinin . The five epitopes are located in different parts of the HA1 domain of the cylinder-like structure of the H3 hemagglutinin. Epitopes A and B are on the top of the HA1 structure and are exposed in the HA trimer. Epitope D is on the top of HA1 structure and is partly buried inside the HA trimer. Epitopes C and E are at the central area of the exposed surface of the HA1 domain as shown in Figure . Using the entropy method, we will show that epitopes A and B are under the highest average selection over all the seasons. These results can be interpreted as the antibody binds mostly to the top exposed part of the structure of hemagglutinin trimer defined by epitopes A and B, and so the selection in these two epitopes is with higher intensity.

We divide the HA1 domain of the H3N2 hemagglutinin into six regions, namely epitopes A to E, and positions not in any of the epitopes. These regions show significantly distinct patterns of evolution. In each seasons from 1993–1994 to 2009–2010, we averaged selection and diversity in each epitope and the positions not in any of the epitopes. The fraction of positions $j$ under selection defined by $S_{i,j} > S_i^\mathrm{thres}$ was also calculated. The averages for 17 seasons are listed in Table . It is evident that the values in Table vary across the epitopes. The selection and diversity in epitopes A and B are greater than those in epitopes C, D, and E for each of selection, fraction of positions under selection, and diversity. The fraction of positions under selection is significantly greater than those in epitopes C, D, and E ($p < 0.038$, using Wilcoxon signed rank test). The values in epitopes C, D, and E are significantly greater than those not in any of the epitopes ($p < 0.0019$ for selection, $p < 0.0011$ for fraction of positions under selection, and $p < 6.0 \times 10^{-4}$ for diversity, using Wilcoxon signed rank test). Consequently, epitopes A and B display the highest level of selection and diversity.

|                    |           |                       |           |
|:-------------------|:----------|:----------------------|:----------|
| Region             | Selection | Fraction of positions | Diversity |
|                    |           | under selection       |           |
| Epitope A          | 0.187     | 0.152                 | 0.077     |
| Epitope B          | 0.157     | 0.134                 | 0.062     |
| Epitope C          | 0.077     | 0.087                 | 0.048     |
| Epitope D          | 0.100     | 0.072                 | 0.037     |
| Epitope E          | 0.111     | 0.094                 | 0.049     |
| Out of epitopes    | 0.021     | 0.019                 | 0.013     |
| The whole sequence | 0.060     | 0.051                 | 0.028     |

Annual selection, fraction of positions under selection, and diversity in epitopes A to E, positions not in any of the epitopes, and the whole HA1 sequence.

# Discussion

The Shannon entropy and the relative entropy are here introduced to quantify the diversity and the selection pressure of the evolving H3N2 virus. The foundation of the entropy calculation is the assumption that the virus sequences used in the entropy calculation are from a random unbiased sampling of the virus circulating in the human population. However, sampling density of the H3N2 virus varies in different geographic regions, hence creating a sampling bias.

We have addressed this issue at the continent level by analyzing data from different regions separately. We now additionally study the effect of sampling bias within one country. For example, among the H3 hemagglutinin sequences labeled with month of collection in the NCBI Influenza Virus Resource Database, the New York state sequences account for about one third of the USA sequences. In the contrast, New York state has only about 6.5% of the USA population. We chose eight seasons from 2001–2002 to 2008–2009 with abundant USA sequences collected in and out of New York state during this period of time available in the database. Using the procedure in the Materials and Methods section, we calculated the histogram of 20 amino acids in each amino acid position in each season for the New York state sequences, and that for the non-New York state sequences. Each of the $329 \times 8 = 2632$ pairs of histograms in and out of New York state were compared using the $\chi^2$ test for homogeneity. The $p$ values of 2581 pairs are greater than 0.05. That is, 98.1% of the pairs are not significantly different. The high sampling density in New York state does not affect the histograms of 20 amino acids in each position, implying that the sampling bias of the H3N2 virus is uncorrelated to the amino acid usage patterns.

By applying Shannon entropy and relative entropy to the aligned hemagglutinin sequences labeled with month of collection, we obtain the evolution and migration pattern of the H3N2 virus in the Results section. First, Shannon entropy and relative entropy quantify diversity of and selection pressure over the virus, relatively. Relative entropy from the current season $i$ to the next season $i+1$ linearly increased with the Shannon entropy in the current season $i$. See Figure and . Second, relative entropy quantifies the similarity of two groups of virus and implies the migration path of the H3N2 virus. See Table . In the following text we compare our methods and results to the literature.

The relative entropy reveals the H3N2 migration pattern. Previous studies applied phylogenetic methods in an attempt to locate the epicenter of the H3N2 epidemic in each season. studied the dynamic of influenza sequence diversity in the temperate regions in both hemispheres to imply that the H3N2 virus originates in the tropics and migrates to the temperate regions in both hemispheres. obtained the antigenic and genetic evolution rate in each region and distances of the H3N2 strains to the trunk of the phylogenic tree. This information indicated East and Southeast Asia as the epicenter, from which the H3N2 virus spreads to North America, Europe, and Oceania in each season . Recently, suggested the center of the H3N2 migration network being China, Southeast Asia, and the USA by estimating the migration rate between different regions in the world. Here the relative entropy, the gain of sequence information, is used as a novel measure of the sequence similarity. The H3N2 migration path is the directed graph in which each path has the minimum relative entropy, or the maximum sequence similarity. These studies reach a consensus that South China is located in the epicenter of influenza epidemics. Here, we additionally identify a novel migration path from the USA to Europe and show that virus evolutionary rate is higher in the epicenter than in the migration paths.

Previous studies have identified positions that have led to the immune escape of influenza, by resolving historical mutations. A 1997 study examining 254 H3 nucleotide sequences from 1984 to 1996 identified 14 positions that are under selection, using the dN/dS ratio method . In a 1999 study involving 357 nucleotide sequences from 1983 to 1997, 18 positions were identified to be under selection, using the dN/dS ratio method . A 2003 paper used the alignment of 525 nucleotide sequences from 1968 to 2000 to calculate the codon diversity and the amino acid diversity . This paper reported 25 positions with the largest codon diversity and amino acid diversity. A 2007 paper located 63 positions of positive selection by alignment of 2,248 sequences from 1968 to 2005 and considering substitutions at the amino acid level . Our study identified 54 positions under selection at the amino acid level, using 4,292 aligned sequences from the 1992–1993 season to the 2009–2010 season. Considering this historical body of work, it is apparent that the number of amino acid positions identified to be under selection has increased with the number and the time span of sequences used in the study and as the discriminating power of the data has increased. In addition, different criteria to identify the positions under selection have been introduced in the previous studies and in the present study. identified positions to be under selection when an amino acid substitution occurred during successive years. We here classify a position $j$ as under selection if its relative entropy $S_{i,j}$ is greater than the threshold $S_i^\mathrm{thres}$ in greater than two seasons. These two methods identify many identical positions as well as some distinct positions.

The heterogeneity of the methods also contributes to identification of different sets of positions under selection. We note that these methods fall into two categories. The first category operates at the codon level. The $\mathrm{d}N/\mathrm{d}S$ ratio method calculates the non-synonymous and synonymous mutation rate of the codon. The method comparing the codon diversity and the amino acid diversity is a variation of the $\mathrm{d}N/\mathrm{d}S$ ratio method. The second category operates at the amino acid level. identified the positions with amino acid switch occurring in history to be under selection. Our entropy method recognizes the positions with relative entropy higher than the threshold, $S_i^\mathrm{thres}$, in greater than two seasons. A large $\mathrm{d}N/\mathrm{d}S$ of a codon does not necessarily mean an amino acid switch in the same position because the amino acid substitution could be unfixed. Methods at the amino acid level, such as the amino acid switch and our entropy method, can identify positions with low $\mathrm{d}N/\mathrm{d}S$ to be under selection because these methods do not consider nucleotide substitutions. Positive selection does not necessarily lead to a fixed amino acid switch, and in this case the entropy method can still detect positive selection. Unlike the amino acid switch method , the entropy method applied in this study is able to detect unfixed amino acid substitutions arising from selection. Our entropy method releases the requirement of fixed amino acid substitution in but adds one requirement: the positions under selection need to present large relative entropy in greater than two seasons. Consequently, these methods identify slightly different sets of positions to be under selection.

# Conclusion

We use Shannon entropy and relative entropy as two state variables of H3N2 evolution. The entropy method is able to predict H3N2 evolution and migration in the next season. First, we show that the rate of evolution increases with the virus diversity in the current season. The Shannon entropy data in one season strongly correlate with the relative entropy data from that season to the next season. If higher Shannon entropy of the virus is observed in one season, higher virus evolutionary rate is expected from this season to the next season. Second, the relative entropy values between virus sequences from China, Japan, the USA, and Europe indicate that the H3N2 virus migration from China to Japan and the USA, and identify a novel migration path from the USA and Europe. The relative entropy values in and out of China, the epicenter, show that evolutionary rate is higher in China than in the migration paths. Moreover, the entropy method was demonstrated on two applications. First, selection pressure of the H3 hemagglutinin is mainly in 54 amino acid positions. Second, the top exposed part in the three-dimensional structure of HA trimer covered by epitopes A and B is under the highest level of selection. These results substantiate current thinking on H3N2 evolution, and show that the selection pressure is focused in a subset of amino acid positions in the epitopes, with epitopes A and B on the top of hemagglutinin being dominant and making the largest contribution to the H3N2 evolution. These predictions and applications show that the entropy method is not only predictive but also descriptive.

#### Acknowledgements

Keyao Pan's research was supported by a training fellowship from the Keck Center Nanobiology Training Program of the Gulf Coast Consortia (NIH Grant No. R90 DK071504). This project was also partially supported by DARPA grant HR 0011-09-1-0055.