abstract: Strategy evaluation schemes are a crucial factor in any agent-based market model, as they determine the agents' strategy preferences and consequently their behavioral pattern. This study investigates how the strategy evaluation schemes adopted by agents affect their performance in conjunction with the market circumstances. We observe the performance of three strategy evaluation schemes, the history-dependent wealth game, the trend-opposing minority game, and the trend-following majority game, in a stock market where the price is exogenously determined. The price is either directly adopted from the real stock market indices or generated with a Markov chain of order $\le 2$. Each scheme's success is quantified by average wealth accumulated by the traders equipped with the scheme. The wealth game, as it learns from the history, shows relatively good performance unless the market is highly unpredictable. The majority game is successful in a trendy market dominated by long periods of sustained price increase or decrease. On the other hand, the minority game is suitable for a market with persistent zig-zag price patterns. We also discuss the consequence of implementing finite memory in the scoring processes of strategies. Our findings suggest under which market circumstances each evaluation scheme is appropriate for modeling the behavior of real market traders.
author: Yongjoo Baek; Sang Hoon Lee; Hawoong Jeong
date: 2024-10-01
title: Market behavior and performance of different strategy evaluation schemes

# Introduction

*Bounded rationality*  is now widely accepted as a fundamental aspect of human decision-making process. Insufficient information and cognitive limitations force people to rely on lessons of experience rather than reason out the optimal solution. Hence, especially in the fields of behavioral economics and econophysics, traditional assumptions of *perfectly rational* agents making *a priori* optimal decisions have been replaced by agents of limited cognitive capacity who follow rules of thumb empirically validated. An archetypical implementation of bounded rationality is found in the famous *El Farol Bar* problem  and its simplified variant, the *minority game* . In the minority game, played by multiple agents for multiple turns, each player opts for one of two choices at each turn and wins the turn if the player is on the minority side. Since it is impossible to predict the choice of the other players, an optimal choice simply does not exist. Instead, each player forms a set of *strategies* which predict the behavior of the other players and advice the player which side to choose given the latest *choice pattern* of the other players. Using a *strategy evaluation scheme*, an agent compares the predictions of its strategies with the actual outcome of the game and evaluates the credibility of each strategy. At each turn, the advice of the most credible strategy is followed. In the standard minority game, strategies are evaluated in terms of their minority game score, *i.e.*, how often they correctly land on the minority side. Different strategy evaluation schemes may be used in other models of bounded rationality, but the structure of those models are essentially similar to the one outlined here.

The minority game has been readily applied in modeling financial markets . Its boundedly rational agents choosing from strategies bear some resemblance to real market participants. The fluctuating choice pattern shows features reminiscent of *stylized facts* of financial markets . Phase transition between symmetric and asymmetric regimes gives clues as to how markets self-organize themselves to be *marginally efficient* . Yet whether the minority game faithfully captures the behavior of financial market speculators has been put under question . One can observe that excess demand (supply) works to the advantage of sellers (buyers), thus favoring the minority, but this is merely a superficial similarity. Whether a strategy is successful is fully revealed only when an agent buys (short-sells) an item and sells (buys) it back *later* at a higher (lower) price. While the outcome of a single event determines the payoff in the minority game, in real markets we need at least two events separated in time to determine the payoff.

A variety of alternative models have been proposed to address this problem. One class of them assume that an agent buys (sells) an item and then immediately sells (buys) it back at the next time step. Marsili  showed that if agents *expect* price changes in the adjacent time steps to be negatively correlated, the minority game payoff is justified. But if agents expect the price changes to be positively correlated, they should use the *majority game* payoff to evaluate their strategies. In that case, the strategies opting for the majority decision obtain higher scores. Since both expectations about price behavior are equally justifiable, we are led to an alternative market model where agents of both minority and majority expectations coexist. Meanwhile, starting from the same buy-today-and-sell-tomorrow assumption, other studies  derived the \$-game payoff which can be roughly considered a *time-delayed* version of the majority game payoff .

Another class of alternatives involves wealth, the total value of cash and assets in an agent's possession. Besides being a natural measure of success, wealth can be updated at each time step without considering two temporally separated events. Furthermore, past buy or sell decisions continue to affect the way score changes, since wealth fluctuates according to the value of financial items accumulated through time. This is not the case for the minority game or the majority game, since the score change is dependent only on the decision made in the previous turn. Hence agents evaluating their strategies using wealth-based payoff are far more history-dependent, and their collective behavior often leads to quite regular price behavior which may explain how different price trends are formed. Yeung *et al.*  performed a comprehensive study on a market model incorporating wealth as the measure of both an agent's success and a strategy's credibility, which they termed the *wealth game*.

All these different payoff schemes grew out of efforts to capture the characteristics of real financial markets more closely. Now it is natural to ask which ones are more relevant. Taking a Darwinian perspective, market participants are likely to use strategy evaluation schemes that are most beneficial for them, *i.e.*, best at accumulating wealth . Thus we may let agents with different strategy evaluation schemes participate in a market, and compare their performance on the basis of wealth. For example, Andersen *et al.*  compared the wealth of the best minority game player with that of the worst in a market exclusively composed of minority game players, where the price is completely determined by the collective behavior of the minority game players. They found that the best player was actually poorer than the worst player in terms of wealth. This inconsistency implies that minority mechanism cannot dominate the *entire* market for long.

However, if minority game players account for only a small part of the market and are effectively decoupled from the price dynamics, there can be special situations when their strategy evaluation scheme proves profitable. In such cases, it is more suitable to follow the methodology set out by Yeung *et al.* , which compares the average wealth achieved by different strategy evaluation schemes when the price data are *exogenously* given. Yeung *et al.* used price data taken from real financial markets, such as the Hang Seng Index (HSI). Since real market data cannot be directly controlled and their complexity evades any simple quantitative description, if we experiment with such data, it is hard to draw any general conclusions about the relation between the market trend and the corresponding suitable behavior patterns of agents. In order to clarify the relation *systematically*, we use artificial price data generated with a Markov process characterized by at most two parameters. Since we can now try various kinds of price behavior controlled by as few parameters as possible, we hope our results are better established and have more general implications.

This paper is organized as follows. First, we describe our model in Sec. . To get some intuition about the problem, we compare the performance of different strategy evaluation schemes in real markets in Sec. , using the Korea Composite Stock Price Index (KOSPI) and the HSI as price data. Conclusions drawn from this section are systematically verified in Sec. , using artificial price data generated by the Markov process. While our studies mainly concern strategy evaluation schemes with infinite score memory, whether introducing finite score memory changes the result is discussed in Sec. . Finally, the summary of our results is presented in Sec. .

# Model

We use a variant of the original wealth game model , where the agents' buy or sell decisions have no influence on the price dynamics. This modification allows us to directly control the price dynamics, so that we can study the correlation between the price behavior and the profitability of each strategy evaluation scheme. Our "exogenized" model can be considered an approximation of the reality, if it concerns only a small fraction of the entire group of market participants. The situation is quite similar to the canonical ensemble in statistical mechanics, where we consider the temperature of the system a variable directly controlled by the external heat bath whose heat capacity is much larger than that of the system.

Consider $N$ agents participating in a stock market. As previously pointed out, they account for only a small part of the market. At time step $t$, agent $i$ decides whether to buy or sell a unit of stock, or to abstain from trade. Agent $i$'s each possible action is represented by $a_i\left(t\right)=\pm 1,~0$, respectively. Agent $i$'s *position* is the accumulation of the agent's past actions, written by $$\label{eq:position}
        k_i\left(t\right)=\sum^{t-1}_{t'=0}a_i\left(t'\right).$$ This indicates the amount of stock the agent owns (if positive) or owes (if negative). Agent $i$'s wealth is the sum of its cash $c_i\left(t\right)$ and stock, $$\label{eq:wealth}
        w_i\left(t\right)=c_i\left(t\right)+k_i\left(t\right)P\left(t\right),$$ where $P\left(t\right)$ is the price of a unit of stock. At each time step, the agent's action changes its cash by $$\label{eq:cash}
        c_i\left(t+1\right)=c_i\left(t\right)-a_i\left(t\right)P\left(t+1\right).$$ Therefore the agent's wealth is updated by the rule $$\label{eq:wealthupdate}
        w_i\left(t+1\right)=w_i\left(t\right)+k_i\left(t\right)\left[P\left(t+1\right)-P\left(t\right)\right].$$

The agents share an $m$-bit *market history*, which records the price increase (denoted by 1) and decrease (denoted by 0) for the latest $m$ time steps. Also each agent is provided with $s$ randomly drawn *strategies*. A strategy is a mapping from the set of $2^m$ possible market histories to the set of the agent's three possible actions (buy, sell, or abstain), the total number of possible strategies being $3^{2^m}$.

At every time step the agent updates the scores of its strategies according to a certain strategy evaluation scheme. The agent follows the suggestion of the highest-scoring strategy, although there is one exception to this rule. A suggestion that makes the agent's cash change sign from positive to negative, or decreases the cash that is already negative if followed is ignored and replaced with an abstention. This constraint, which we shall call the *non-negative cash constraint*, is implemented by introducing the position limitation $$\label{eq:maxpos}
    K_i\left(t\right) = \max \left[\frac{w_i\left(t\right)}{P\left(t\right)},0\right]$$ so that any action increasing $|k_i| - K_i$ when $k_i > K_i$ or $k_i < -K_i$ is forbidden.

We consider three strategy evaluation schemes as classified by Yeung *et al.*  The score of strategy $\sigma$ at time step $t$ is denoted by $u_{\sigma}\left(t\right)$.

1.  Wealth game (WG): the score of a strategy is updated in a manner similar to the way an agent's wealth is updated. That is, the score of strategy $\sigma$ is updated by $$u_{\sigma}\left(t+1\right)=u_{\sigma}\left(t\right)+k_{\sigma}\left(t\right)\left[P\left(t+1\right)-P\left(t\right)\right]
        \label{eq:wg}$$ where $k_\sigma\left(t\right)=\sum^{t-1}_{t'=0}a_\sigma\left(t'\right)$ is the *virtual position* of strategy $\sigma$. Now strategies are also subject to the non-negative cash constraint: any action $a_\sigma\left(t\right)$ of the strategy that makes the cash part of the strategy's score change sign from positive to negative, or decreases the cash part that is already negative, is replaced with an abstention. Note that while agent $i$'s wealth and its strategy $\sigma$'s score are updated in the same manner, they are *not* equal, since agent $i$'s action $a_i\left(t\right)$ and position $k_i\left(t\right)$ are different from strategy $\sigma$'s suggested action $a_\sigma\left(t\right)$ and virtual position $k_\sigma\left(t\right)$. We expect this position dependence to grant WG the strongest history dependence among the three schemes under consideration.

2.  Minority game (MinG): the score updating rule of MinG is given by $$\label{eq:ming}
        u_{\sigma}\left(t+1\right)=u_{\sigma}\left(t\right)-a_{\sigma}\left(t\right)\left[P\left(t+1\right)-P\left(t\right)\right].$$ Trend-opposing strategies are favorably scored, as buying (selling) increases the score when the price falls (rises). The buying (selling) party can be roughly considered the minority in a bearish (bullish) market, so we can say MinG favors the minority. Hence the name of the scheme, despite the fact that the agents do not necessarily have to be in the minority in the exact sense of the term to gain profits. As pointed out by Marsili , users of this scheme believe that the immediate price trend will be reversed soon. Also they are short-sighted (or eager to forget about the past) in the sense that only the action of the previous turn, rather than the position, affects the change of score.

3.  Majority game (MajG): the score updating rule of MajG is given by $$\label{eq:majg}
        u_{\sigma}\left(t+1\right)=u_{\sigma}\left(t\right)+a_{\sigma}\left(t\right)\left[P\left(t+1\right)-P\left(t\right)\right].$$ Trend-following strategies are favored, as buying (selling) increases the score when the price rises (falls). Just in the sense that MinG favors the minority, MajG favors the majority. MajG users expect that the current market trend would be sustained .

# Performance in real markets

We start by comparing the performance of the three strategy evaluation schemes in real financial markets. We use the closing price data of the HSI from December 31, 1986 to June 10, 2009 and the KOSPI from July 1, 1997 to June 10, 2009 .

## Adaptation to market trends

One measure of the agents' overall adaptation to the market trends is the number of strategy-switching agents. A large number of strategy-switching agents indicate that the agents are actively responding to changes in the market trend. As Figs. (a)–(c) show, the peaks of the number of strategy switchers decrease in size, *i.e.*, later trend changes do not induce as much response from the agents as early trend changes do. Cumulative scoring of each strategy means that the score gap between strategies will broaden over time, which makes agents more reluctant to abandon previously successful strategies. Thus, the agents increasingly settle on one strategy.

Decay of peaks is the most rapid for WG. Position dependence of the WG wealth update rule quickly broadens the score gap between successful and unprofitable strategies, and this gap is not easily reduced unless a new market trend persists for a sufficiently long period. Thus WG agents tend to settle their preferred strategies early on and stick to it as long as possible. MinG and MajG agents show similar behavior, but their strategy preferences are not as clear as those of WG agents. Hence, MinG and MajG agents are more sensitive to trend changes.

## Effect of trend changes on agents' wealth

In the previous subsection, we observed that WG agents quickly adapt to the initial market trend. They can optimize their positions for the initial trend and efficiently accumulate wealth. But since positions can be changed only by one at a time, they are very vulnerable to sudden trend changes. This is confirmed by the average wealth curves shown in Figs. (d) and (e), especially the KOSPI during the 1997 Asian financial crisis. While WG agents quickly realized the benefits of short-selling and prospered during the initial market crash, their advantage turned into a trap when the market began to "recover" in 1998. Combination of large negative positions (built up by short-selling) and positive price changes meant WG agents were particularly hard hit by this sudden trend reversal. It took some time for WG agents to switch their strategies and reverse the sign of their positions, and for a while their average wealth was the lowest among the three schemes. This observation shows that while wealth is a convenient measure of success, it has its own shortcomings. An agent cannot liquidate its own assets all at once, so if a large portion of the agent's wealth comes from assets, the amount of wealth is largely at the mercy of price fluctuations.

MajG agents suffer similar difficulties in 1998, but their adaptation to the initial trend was not as complete as that of WG agents, *i.e.*, their positions were not sufficiently negative. Hence, their initial wealth gain on average was less than that of WG agents, but so was their loss due to the trend reversal.

On the other hand, MinG agents are always least affected by trend reversals. They always try to move against the market trend, and consequently their actions are severely restricted by the position constraint in Eq. (), limiting their positions to near-zero region. Hence MinG agents may temporarily attain the highest average wealth after some trend changes. But they cannot keep the lead for long if the new trend turns out to be stable, as in the KOSPI where WG eventually catches up with MinG.

## Initial trend significance

Agents are most flexible in the initial stage and grow increasingly conservative over time. As time passes, the gaps between the scores of strategies broaden and wealth gained or lost in the initial phase limits the freedom of the agents, both making it harder for agents to switch to a different strategy. Figure (d) shows that MinG agents are the second most successful in the HSI despite the general increase of price. This is because the initial trend, somewhat periodical increase and decrease of price, worked to MinG agents' advantage. This helped MinG agents attain good position and high average wealth early on, and MajG never overcame this initial disparity. A similar observation holds for the KOSPI, where WG was never able to make up for the advantage of MajG formed by the trend reversal in 1998.

This initial trend dependence may be seen as a weakness of the strategy evaluation schemes considered in this study. But while real markets show a complex mixture of *heterogeneous* trends, the strategy evaluation schemes are designed as a *simplified* model of the behavior of market participants. The extent to which those simplified agents can cope with ever-changing market trends is bound to be limited, and hence the lingering influence of the initial trend. Traders in real markets are certainly more adaptive than those model agents and initial trends would be less crucial for them. This is yet another reason why we experiment with artificially generated prices in the next section, so that the complexity of market trends is reduced and initial trend dependence becomes less significant.

In summary, WG agents quickly adapt to the initial market trend and draw the most profits from the trend. At the same time, however, WG agents suffer the most from sudden trend changes, while MG agents are least swayed by them. In addition, the initial trend is significant, since it affects the strategy preference and the freedom of choice for later periods.

# Performance in artificial markets

## Simulation settings

To clarify the relationship between the price pattern and the performance of strategy evaluation schemes remains, we extend our study to include artificially generated prices whose behavior can be described by a few parameters. Let $p_\uparrow (\mu)$ be the probability that the price will increase in the next time step given the latest $m$-bit market history $\mu$. At each time step the price can increase or decrease by 1. Starting from the random initial $\mu$ and the initial price $P(0) = 1000$, we generate all the subsequent price data.

## History-independent price behavior

Consider the case when the price data are generated using only a single probability $p_\uparrow$, the history-independent probability of price increase. In other words, the price dynamics is a biased random walk. If $p_\uparrow$ is sufficiently larger or smaller than $1/2$, the price reliably increases or decreases from the beginning to the end. If $p_\uparrow \simeq 1/2$, the dynamics gets close to an unbiased random walk and the price behavior becomes unpredictable.

Figure  shows the average wealth of agents for each strategy evaluation scheme at the 1000th time step, for each value of $p_\uparrow$. As the market becomes more predictable ($p_\uparrow$ farther away from $1/2$), WG and MajG are similarly more successful than MinG. Higher predictability means that WG agents' lessons from history are more useful. Since there is only one probability involved, higher predictability is synonymous with stability of trends, a favorable condition for characteristic MajG agents. On the other hand, no strategy evaluation scheme is significantly better than the others when the market is unpredictable.

Note that the WG and MajG curves in Fig.  are highly asymmetric. These curves indicate that agents are better off when $p_\uparrow$ is low rather than high, which is a natural consequence of our model. When $p_\uparrow$ is high, agents accumulate wealth by maintaining large positive positions. But as the agents run out of cash, the position constraint Eq. () prevents them from buying stock any further. Hence the average wealth increases only at a limited rate. When $p_\uparrow$ is low, successful agents quickly build up large negative positions. Such an agent's wealth has positive contribution from cash and negative contribution from the amount of sold stock. While the amount of cash keeps increasing, the fraction of stock in the agent's total wealth continues to decrease as the price falls.

Therefore, there is effectively no lower bound on the negative position, which leads to ever-accelerating wealth gain of agents. This result is against our intuition that bullish market is more profitable than bearish market, but it should be recalled that we are considering only a small fraction of the entire market participants with negligible influence on the price dynamics, who can freely buy or sell. Under realistic circumstances, agents cannot keep selling stock and build negative position, since they have trouble finding buyers.

Note that this asymmetry is also reflected in the real market simulation results shown in Fig. . Since the overall increment of the HSI is larger than that of the KOSPI during the observation period, the eventual ratio of the agents' average wealth to the stock index is much larger in case of the KOSPI than the HSI.

## Price generated by two-bit market history

Now we generate the price data using four probability parameters, $p_\uparrow \left(\downarrow\downarrow\right)$, $p_\uparrow \left(\downarrow\uparrow\right)$, $p_\uparrow \left(\uparrow\downarrow\right)$, and $p_\uparrow \left(\uparrow\uparrow\right)$. Each parameter corresponds to the probability of price increase given the latest two-bit market history, where the direction of each arrow represents the direction of price change. In this case, we can say the price is generated by a Markov chain of order two.

Previous simulations show that long-term price increase or decrease favors WG and MajG over MinG. Now we shall only consider the cases when general price increase or decrease from the beginning to the end is suppressed. For this we introduce two new constraints,

$$p_\uparrow \left(\downarrow\downarrow\right) + p_\uparrow \left(\uparrow\uparrow\right) = p_\uparrow \left(\downarrow\uparrow\right) + p_\uparrow \left(\uparrow\downarrow\right) = 1.
\label{prob_constraint}$$

This reduces the number of parameters to two. We define the long-term parameter $p_L$ and the short-term parameter $p_S$ by

$$\begin{split}
    p_L &\equiv p_\uparrow \left(\uparrow\uparrow\right) - 0.5 = 0.5 - p_\uparrow \left(\downarrow\downarrow\right), \\
    p_S &\equiv p_\uparrow \left(\uparrow\downarrow\right) - 0.5 = 0.5 - p_\uparrow \left(\downarrow\uparrow\right).
\end{split}$$

If $p_L > 0$, price is likely to increase or decrease for three consecutive time steps or longer. If $p_L < 0$, price increase or decrease is not likely to continue for more than two time steps. Thus $p_L$ controls the likelihood of long-term price trends. Meanwhile, if $p_S > 0$, price is likely to show sustained zig-zag oscillations of period two. If $p_S < 0$, price increase or decrease is likely to continue for at least two time steps. Hence $p_S$ controls how rapid price oscillations are likely to be.

Examples of generated price data for different values of $p_L$ and $p_S$ are shown in Fig. . Figure  visualizes relative performance of the three evaluation schemes, in terms of the average wealth and the chance of achieving the highest average wealth. We can simplify our observations in terms of four extreme cases.

1.  $p_L > 0$ and $p_S > 0$
    The price behavior is dominated by long-term trends with intermittent zig-zag oscillations, as illustrated by the $p_L = 0.4$ and $p_S = 0.4$ case shown in Fig. . WG and MajG outperform MinG since trend-following strategies are more suitable for a trendy market. However, zig-zag oscillations are also likely to persist, which is disadvantageous for MajG. Hence, WG tends to be more successful than MajG.

2.  $p_L > 0$ and $p_S < 0$
    The price behavior is completely dominated by long-term trends without zig-zag oscillations, as illustrated by the $p_L = 0.4$ and $p_S = -0.4$ case shown in Fig. . WG and MajG are more successful than MinG, with MajG closely in the lead. It is not obvious why MajG should be better than WG, but note that there are abrupt trend reversals in the price pattern. As pointed out in Sec. , WG is particularly prone to initial trend reversals. Thus WG agents start with a slight disadvantage, which they find difficult to make up for later even if they eventually adapt themselves to the market.

3.  $p_L < 0$ and $p_S > 0$
    Long-term trends are suppressed and zig-zag oscillations of period 2 ($\uparrow\downarrow\uparrow\downarrow\ldots$, for example) become dominant, as illustrated by the $p_L = -0.4$ and $p_S = 0.4$ case shown in Fig. . MinG is the most successful since its fundamentalistic expectations turn out to be correct. WG still adapts well to the price behavior and its average wealth is only slightly less than that of MinG. MajG gets the lowest average wealth due to its chartistic nature.

4.  $p_L < 0$ and $p_S < 0$
    Zig-zag oscillations of period 4 ($\uparrow\uparrow\downarrow\downarrow\ldots$, for example) become dominant, as illustrated by the $p_L = -0.4$ and $p_S = -0.4$ case shown in Fig. . This condition is still more favorable for MinG than for MajG, but the advantage of MinG is not as strong as in the case $p_S > 0$. Thus, WG outperforms MinG.

It should be noted that although MajG tends to show the worst performance for $p_L < 0$, there is an exception when $p_S \simeq 0$. Whenever $p_S$ is far from zero while $p_L < 0$, the price dynamics is dominated by regular zig-zag price oscillations. But if $p_S \simeq 0$, regularity of the oscillations is broken, giving some chance for MajG while impairing the performance of MinG.

Even if we change the values of the number of agents $N$ and the length of market history $m$ to check the stability of our results, qualitatively the same results as explained above are observed. Interestingly, the area of the $p_L$ – $p_S$ diagram in which MinG or MajG is the most successful increases as $m$ is increased, as shown in Fig. . This is not because greater memory $m$ enhances the performance of MajG and MinG. Rather, all three schemes tend to show worse performance for greater $m$, with the impairment being the severest for WG. We suspect that when $m > 2$, agents are trying hard to find some spurious causal relations between market history and price behavior, which can cause undesirable inefficiency.

The price dynamics is dependent only upon the latest two time steps, but agents are considering longer time spans to make their decisions. Thus they are likely to make false conclusions about the market trends, which deteriorates their average wealth. Also greater $m$ leads to rapidly growing number of possible strategies ($= 3^{2^m}$), but we have fixed the number of strategies available for each agent to two. This further hinders the agents from making correct decisions, since only a few agents would be given strategies suitable for the market trends. Since WG has the strongest dependence on history, they suffer the heaviest loss from these problems.

In most cases, WG manages to be the most successful scheme, otherwise closely follows the scheme of the highest average wealth. Although they may suffer temporarily from trend changes, eventually they learn to make up for their loss as long as the market shows sufficient predictability. Even if the market totally lacks predictability, all schemes show similar performance, so even in such cases we cannot say WG is worse than other schemes. We can say WG is the most "versatile" among the three schemes.

With this concept of Markov chain, we are able to *measure* the probabilities of each movement $p_\uparrow \left(\downarrow\downarrow\right)$, $p_\uparrow \left(\downarrow\uparrow\right)$, $p_\uparrow \left(\uparrow\downarrow\right)$, and $p_\uparrow \left(\uparrow\uparrow\right)$ for the real stock market data, in retrospect. If the probabilities are measured for each consecutive two-step and averaged for the entire observation period, it turns out that for the HSI, $p_\uparrow \left(\downarrow\downarrow\right)=0.52$, $p_\uparrow \left(\downarrow\uparrow\right)=0.53$, $p_\uparrow \left(\uparrow\downarrow\right)=0.50$, and $p_\uparrow \left(\uparrow\uparrow\right)=0.53$. Similar values are observed for the KOSPI with $p_\uparrow \left(\downarrow\downarrow\right)=0.51$, $p_\uparrow \left(\downarrow\uparrow\right)=0.54$, $p_\uparrow \left(\uparrow\downarrow\right)=0.53$, and $p_\uparrow \left(\uparrow\uparrow\right)=0.56$. Note that the previous constraints in Eq. () do not exactly hold in real data . However, all the probabilities are slightly above $0.5$, $p_\uparrow \left(\uparrow\uparrow\right) \gtrsim p_\uparrow \left(\downarrow\downarrow\right)$, and $p_\uparrow \left(\downarrow\uparrow\right) \gtrsim p_\uparrow \left(\uparrow\downarrow\right)$, which may give slight overall advantage to WG and MajG according to Fig. (a) ($m = 2$ case), reasonably consistent with our observation.

# Finite Score Memory

Thus far, we have assumed that the scores of the strategies have infinite memory. That is, a strategy's suggestions made in the past and at present equally contribute to its score. As time passes, the sheer size of the past experience overwhelms contributions from the recent experience. Thus agents grow increasingly unresponsive to trend changes, as the decrease of strategy switchers in Figs. (a)–(c) clearly indicates. For the agents to stay highly adaptive to market changes, our model should implement finite score memory so that the emphasis is put upon the "present state of affairs."

Finite score memory can be implemented by evaluating the performance of a strategy in terms of the score accumulated over the latest $T$ time steps . If $u_\sigma\left(t\right)$ denotes the infinite-memory score of a strategy as originally defined by one of the three evaluation schemes explained in Sec. , we define the corresponding finite-memory score $u_\sigma^T\left(t\right)$ by $$u_\sigma^T \left(t\right) = u_\sigma \left(t\right) - u_\sigma \left(t-T\right).$$ It is straightforward to apply the above formula to MinG, MajG, and WG. For convenience, let these finite-score-memory variants of the original three strategy evaluation schemes be called the delta minority game (DMinG), the delta majority game (DMajG), and the delta wealth game (DWG).

A note of caution is in order in the case of DWG. The virtual wealth $u_\sigma \left(t\right)$ should obey the non-negative cash constraint explained in Sec. , if $u_\sigma^T \left(t\right)$ is to be interpreted as the *change* of virtual wealth. Since the score $u_\sigma^T \left(t\right)$ is indirectly subject to the constraint through $u_\sigma \left(t\right)$, the DWG score is still under the influence of the entire history. In this sense, the score memory of DWG is not truly finite.

## Real markets

Figure  shows the average wealth achieved by DMinG, DMajG, and DWG in real markets (KOSPI and HSI) on the last day of the period of observation, for different score memory sizes $T$. The performance of each strategy evaluation scheme strongly depends on the score memory size, especially in the case of KOSPI where the overall increase of price is much less dominant. While it is hard to find any general tendencies in the fluctuating average wealth curves, they do suggest that the real market behavior is a complex mixture of market trends with different time scales.

## Artificial markets

Let us first consider the performance of WG, DMinG, and DMajG in an artificial market where the price is generated by biased random walk. We can observe that finite score memory does not make much difference in this case: DMajG exhibits the worst performance among the three except for the unpredictable case $p_{\uparrow} \sim 0.5$, while DWG and DMajG are almost equally successful, in a manner very similar to the result shown in Fig. . Finite score memory is intended to improve the performance of agents when the market trends change from time to time, but biased random walk generates very stable market trends. Thus score memory is not so relevant in this case.

If the price is generated by a Markov chain of order 2 subject to the constraint Eq. (), then finite score memory causes some visible changes, as shown in Fig. . Figures (a)–(c) show the relative performances of DMinG, DMajG, and the original WG. The original WG, whose score memory is fixed to be infinite, is unaffected by the score memory size $T$. Thus Figs. (a)–(c) show how the *absolute* performances of DMinG and DMajG are affected by score memory. Note that the performances of DMinG and DMajG are worse than their infinite score memory counterparts, as indicaed by the greater area in the $p_L$ - $p_S$ diagram dominated by the original WG. As the score memory is increased, the shape of the $p_L$ - $p_S$ diagram approaches the infinite score memory result shown in Fig. . If we compare the performances of DMinG, DMajG, and DWG as shown in Figs. (d)–(f), we observe that the performance of DWG is worse than that of the original WG, but again approaches the original infinite score memory result as $T$ becomes large.

Our observations indicate that finite score memory does not improve the agents' performance if the price is generated by a Markov process of order $\le 2$, while keeping the qualitative features of the observations made in the infinite score memory cases. The price series thus generated exhibit simple and steady market trends, while finite score memory is intended for more complex market trends. If the market trend is steady, agents would better recognize the trend if they observe it for a sufficiently long time rather than throw away their distant past experiences.

# Summary and conclusions

We have studied average performances of three strategy evaluation schemes, WG, MinG, and MajG, in a market whose price dynamics is exogenously determined by real market data or the Markov process. We believe that our simplified version of WG effectively captures the behavior of a small portion of traders in a large stock market, and helps determine which evaluation scheme is useful for individual traders given a particular market situation. We can expect that individuals whose influence on the market is negligible will use tests similar to the one presented in this paper, and choose the most successful scheme to evaluate their strategies. Thus we can tell which scheme most suitably describes the behavior of agents under different market circumstances. While incorporation of finite score memory does make a difference in real markets, the basic conclusions drawn from the artificial market observations are qualitatively still valid.

Note that MajG is most successful when the price pattern is completely dominated by long-term trends, while MinG is the best choice if the price tends to show rapid oscillations. In other words, these schemes prosper when their expectations are fulfilled. Combined with Marsili's observation  that the price behavior follows the expectations of either MinG or MajG depending on which side is more dominant in the market, we get an idea of how certain price patterns maintain themselves through positive feedback. For instance, a bubble can maintain itself because long-term trends make MajG traders a dominant force in the market and MajG traders' expectations fulfill themselves. Whether we can model some *negative* counterpart of this feedback mechanism would be an interesting issue for further studies.

We used average wealth to assess the viability of each evaluation scheme, which at first glance may seem biased toward WG. But as our simulation results show, other schemes can be more successful than WG depending on the market behavior, even though wealth is the only measure of success in our model. Trend reversals and lingering influence of initial market trends are the main reasons why the performance of WG is impaired. WG agents are especially adept at building up positions optimal for the given price pattern, which means that a large portion of their wealth comes from their assets. Pattern changes can be too rapid for WG agents to follow up by moving to a new optimal position, and in such cases the agents are simply at the mercy of the market's whim.

Still, we cannot ignore the advantages of WG. Its main strength lies in its versatility, as shown by tests using generated price data. WG agents learns from the history, so they eventually adapt to any market trends and make up for the losses from initial "mistakes." Thus, we can justify using WG to model the behavior of traders in financial markets, whenever there is some degree of predictability and persistence in price patterns.


abstract: Development combines three basic processes — asymmetric cell division, signaling and gene regulation — in a multitude of ways to create an overwhelming diversity of multicellular life-forms. Here, we attempt to chart this diversity using a generative model. We sample millions of biologically feasible developmental schemes, allowing us to comment on the statistical properties of cell-differentiation trajectories they produce. Our results indicate that, in contrast to common views, cell-type lineage graphs are unlikely to be tree-like. Instead, they are more likely to be directed acyclic graphs, with multiple lineages converging on the same terminal cell-type. Additionally, in line with the hypothesis that whole body regeneration is an epiphenomenon of development, a majority of the 'organisms' generated by our model can regenerate using pluripotent cells. The generative framework is modular and flexible, and can be adapted to test additional hypotheses about general features of development.
author: Somya Mani; Tsvi Tlusty
bibliography: references.bib
title: Large-Scale Survey of Cell-Differentiation Programs in a Generative Model Reveals Regeneration as an Epiphenomenon of Development

# Supplementary material

## Dissection of the effect of parameters on graph size

We compared the properties of lineage graphs in our data with those generated with Erdos-Renyi random graphs (ER graphs) of similar size. ER graphs are generated using a fixed probability, $p$ of any two nodes in the graph being connected by an edge . Therefore, on average, the number of edges in a graph with $n$ nodes is proportional to $n^2$. Increasing the number of nodes to $c*n$ increases the number of edges to $c^2 * n^2$. We determined the number of edges in lineage graphs with $n = [1,2,3,4,5,6,7,8,9,10]$ nodes, and calculated the number of edges expected in ER graphs with $n = [2,4,6,8,10,12,14,16,18,20]$ nodes. Compared to ER graphs, the rate of growth of number of edges in lineage graphs in our data is noticeably slower (Fig.(A)).

The number of nodes in lineage graphs decreases sharply with the parameter $P_\mathrm{adj}$ (Fig.(E)). We show here that this occurs because even at low values of $P_\mathrm{adj}$, cell-types in organisms are connected enough that the fraction of cell-types receiving all signals produced in the organisms reaches a maximum (Fig.(B)).

The effect of the parameter $P_\mathrm{asym}$ on the number of nodes in lineage graphs can be explained in terms of its effect on the number of distinct daughter cell-types produced (Fig.(C)). The number of distinct daughter cells produced at different values of $P_\mathrm{asym}$ is related to the average fraction of genes in a 1 state in these daughter cells. Among all possible cell-types with $N$ genes, most cell-types tend to have about half their genes in a 1 state, and very few cell-types contain fewer, or more genes in a 1 state (Fig.(D:inset)). Therefore, when $0.2 < P_\mathrm{asym}< 0.6$, where on average, daughter cells have about half their genes in a 1 state, organisms produce the most number of distinct daughter cells, and at $P_\mathrm{asym}< 0.2$ and $P_\mathrm{asym}> 0.6$, fewer distinct daughter cells are produced (Fig.(D)).

## Randomization of lineage graphs

We randomized lineage graphs generated with our model while keeping node in-degrees and out-degrees unchanged. Topology distribution largely remains unchanged upon randomization (Fig.(A,B)), and not many graphs change their topology upon randomization (Fig.(C)). Although, we find that the proportion of acyclic graphs decreases slightly, from 24% in model generated graphs, to 19% in randomized graphs.

## Characteristics of lineage graphs with different topologies: graph size

Different graph topologies are different in their graph size distributions. While SCC and cyclic graphs span a large range of graph sizes (Fig.(A,B)), trees and chains tend to be notably small (Fig.(C,E)). DAG type graphs can have moderately large number of nodes (Fig.(D)).

## Characteristics of tree-type lineage graphs

Tree-type graphs can be further characterized as divergent or convergent trees. We call graph nodes with in-degrees \> 1 convergent, and nodes with out-degrees \> 1 divergent. Note that by this definition, the same node is allowed to be both convergent and divergent. For some tree-like graph with $n$ nodes and $n_e$ edges, let $in_i$ and $out_i$ be the in-degree and out-degree of the $i^{th}$ node respectively. We define for this graph a number $g_c$ as the sum of in-degrees of all convergent nodes, and a number $g_d$ as the sum of out-degrees of all divergent nodes, i.e.;

$g_c = \Sigma in_i$, $\forall i$ s.t. $in_i > 1$,

$g_d = \Sigma out_i$, $\forall i$ s.t. $out_i > 1$

We define the degree of divergence of this graph as $(g_d - g_c)/n_e$. For a perfectly divergent tree, such as the tree to the left in Fig.(A), the degree of divergence is 1. And for a perfectly convergent tree (e.g. the tree to the right in Fig.(A)), degree of convergence is -1. We find that most tree-like graphs in our data tend to be more convergent than divergent (Fig.(B)). Lineage graphs that are divergent lead to an increase in cell-type diversity starting from a few initial cell-types. Lineage graphs of real organisms are believed to be divergent trees. Larger trees tend to be more divergent (Fig.(C)). Degree of divergence decreases as $P_\mathrm{asym}$ increases, it is relatively insensitive to $P_\mathrm{sig}$ and $P_\mathrm{adj}$.

## Characteristics of DAG-type lineage graphs

DAG-type graphs differ from tree-like graphs in having edges that link different branches. If the edges in the DAG are rendered undirected, these edges are parts of cycles, or loops (Fig.(A)). The number of such edges in DAGs can be determined by subtracting the number of edges in the spanning tree of the graph from the total number of edges. For a graph with $n$ nodes, the spanning tree has $n-1$ edges. For a given DAG-type graph, we call the fraction of its edges that forms loops, its loop-fraction. Loop-fractions of DAG-type lineage graphs indicate the level of trans-differentiation. DAG-type graphs in our data have high loop-fractions (Fig.(B)), and loop-fraction increases with graph size (Fig.(C)).

## Distribution of regenerative capacities

In order to infer whether a lineage graph is regenerative, we only look at whether its regenerative capacity is greater than 1, or not. In Fig.(A), we show the spread of regenerative capacities for different topologies. For most topologies, median regenerative capacity is greater than 1. The actual value of regenerative capacity is less meaningful, except in the case of tree-type graphs, where most trees have a regenerative capacity of 0. This implies that most trees contain no pluripotent cells. We also find that while median regenerative capacity decreases with $N$, the range of regenerative capacities increases with $N$ (Fig.(B)).

## Organisms in the model have short regeneration trajectories

Regeneration trajectories from pluripotent cells to the homeostatic organism tend to be short (Fig.). On average, trajectory lengths do not depend on whether the pluripotent cell is independent or not. At first, the observation that trajectory lengths decrease with graph size (Fig.(D)), might seem incongruous. But it can be understood intuitively by seeing that number of nodes in lineage graphs is proportional to the number of distinct daughter cells produced in every step of development. Since the same rules are followed for development and regeneration in the model, we can expect the the number of new cell-types added at each step of the regeneration trajectory is larger for organisms with more nodes in their lineage graphs.

## Intrinsically independent cell-types are enriched in lineage graphs

We find in the model that the cell-fate of most pluripotent cells is independent of cellular context (Fig.). We wondered whether the large number of independent cell-types in lineage graphs in our data could be attributed to an insensitivity of these cell-types to signals from other cell-types. Alternatively, these cell-types could be independent despite being responsive to signals from other cell-types. We find that the former case tends to be true. We call a cell-type *intrinsically independent* if the full set of signals that can potentially be received by each of its daughter cells is already satisfied by signaling among these daughter cells themselves. In other words, no further external signals can influence the fates of the daughter cells of intrinsically independent cell-types. We calculated the fraction of intrinsically independent cell types across all $2^N$ possible cell-types across all systems in our data. We find that cell-types that are part of lineage graphs are much more likely to be intrinsically independent irrespective of parameter region (Fig.). Thus cell-types in lineage graphs are predisposed to be independent.But, not all independent cell-types in lineage graphs are intrinsically independent (overall, about 20% the independent cells across all lineage graphs are not intrinsically independent), and not all independent cell-types are pluripotent (Fig.(F)).

## Drosophila segement polarity network expressed in terms of the generative model

In Drosophila embryos, segment polarity genes maintain borders of parasegments, which are 4 cells wide. Within each parasegment, the polarity genes are expressed in characteristic stripes. In , the authors demonstrated that the gene regulatory network responsible for the pattern of gene expression in this system can be modeled as a Boolean logical network. In the following, we examine the Drosophila segment polarity network in terms of our generative model.

The network consists of 15 nodes: en, wg, hh, ptc and ci represent mRNAs, and SLP, EN, WG, HH, PTC, SMO, PH, CI, CIA and CIR represent proteins. Of these, WG, hh and HH act as signals. Signaling molecules HH and WG do not participate in regulation within the cells that produce them, rather they act only in cells that receive them as signals. In order to incorporate this feature, we represent each cell in the parasegment as two model cells; production of all non-signal molecules takes place in one of the cells, and molecules responsible for regulation of signal molecule production are exported to the second cell, from which signal molecules are secreted (Fig.(A,B)). In this sense, the second cell acts as a special compartment which insulates the gene network in the first cell from regulation by signal molecules produced within the same cell.

In this system, signals are exchanged only between neighbouring cells. Accordingly, in our model, cell positions can be expressed as additional 'genes', whose states do not change. For example, to express the positions of the $4 \times 2$ cells in this system, we use 3 additional 'genes' (Fig.(C)). In this system, signal exchange only depends on these 'positional genes', and does not depend on the states of the other genes.

Thus, our model is capable of expressing spatial arrangement of cells, and complex signaling mechanisms, although, it comes at the cost of an increase in system size. In Fig , we show the signaling vector $\mathrm{SG}$, and portions of the cellular adjacency matrix $A$, and gene regulation matrix $GR$ relevant to the steady state of the wild-type segment polarity network. The authors assume symmetric cell-division in , and we do the same; therefore we do not show the cell-division matrix $\mathrm{CD}$ here.


abstract: Unmanned aerial vehicle (UAV) wireless communications have experienced an upsurge of interest in both military and civilian applications, due to its high mobility, low cost, on-demand deployment, and inherent line-of-sight (LoS) air-to-ground channels. However, these benefits also make UAV wireless communication systems vulnerable to malicious eavesdropping attacks. In this article, we aim to examine the physical layer security issues in UAV systems. In particular, passive and active eavesdroppings are two primary attacks in UAV systems. We provide an overview on emerging techniques, such as trajectory design, resource allocation, and cooperative UAVs, to fight against both types of eavesdroppings in UAV wireless communication systems. Moreover, the applications of non-orthogonal multiple access, multiple-input and multiple-output, and millimeter wave in UAV systems are also proposed to improve the system spectral efficiency and to guarantee security simultaneously. Finally, we discuss some potential research directions and challenges in terms of physical layer security in UAV systems.
author: Xiaofang Sun,  Derrick Wing Kwan Ng,
Zhiguo Ding,  Yanqing Xu,
and Zhangdui Zhong,  [^1] [^2] [^3]
bibliography: IEEEabrv.bib; Reference.bib
title: Physical Layer Security in UAV Systems: Challenges and Opportunities

# Introduction

In the past few years, heterogeneous data applications have emerged in wireless communications, such as video streaming, e-health monitoring, video conferencing, etc. However, traditional communication networks with fixed infrastructures are unable to meet the increasingly stringent quality-of-service (QoS) requirements in the fifth-generation (5G) and beyond 5G networks. As a result, the development of unmanned aerial vehicle (UAV) has created a fundamental paradigm shift in wireless communication systems to facilitate fast and highly flexible deployment of communication infrastructures. In particular, by exploiting the high maneuverability of UAV, communication links can be established ubiquitously, especially in temporary hotspots, disaster areas, and complex terrains. Compared to traditional terrestrial wireless communications, UAV wireless communications have the following unique features :

-   **On-demand and swift deployment:** UAV enables fast establishment of temporal communication infrastructures in emergency scenarios where legacy or fixed infrastructures are destroyed or do not exist, such as disaster rescue, remote sensing, firefighting, and others. The deployment of UAV facilitates cost-efficient and uninterrupted communications.

-   **High flexibility:** Due to the fully controllable three-dimensional (3D) mobility, UAV can either stay quasi-stationarily or cruise continuously to a dedicated location, depending on the requirements of wireless communication systems. The movement of UAVs provides a new degree of freedom to offer efficient communications.

-   **High probability of line-of-sight (LoS) air-to-ground channel:** According to the field trial results , the LoS component dominates the air-to-ground channels in many practical scenarios, especially for rural areas or moderately height UAV altitude. This channel characteristic leads to the fact that the channel state information (CSI) can be directly determined by the position of each node and used to facilitate the design of high-speed communication systems.

-   **Limited resources:** It is noted that onboard batteries of UAVs have limited energy storage. Moreover, the sizes of UAVs are usually small for on-demand deployment, and the weights of UAVs for communications are typically not exceeding $25$ kg for both safety and energy consumption issues . These limitations directly restrict the communication, computation, and cruising duration capabilities of UAV.

Despite the promising gains brought by UAVs, the open nature of air-to-ground wireless channels makes secure information transfer a challenging issue. Specifically, on the one hand, information signals transmitted over wireless LoS channels are likely to be intercepted by some undesired receivers which leads to a risk of information leakage. On the other hand, wireless UAV transceivers are vulnerable to malicious jamming attacks. Hence, security plays an extremely important role in UAV wireless communications. Unfortunately, traditional encryption techniques require high computational complexity leading to a large amount of energy consumption which may not be suitable to UAV systems. As an alternative, physical layer security is computationally efficient and effective in safeguarding wireless communication networks via exploiting the inherent randomness of wireless channels. As a result, various physical layer techniques have been proposed in the literature for guaranteeing communication security, e.g., .

Since the channel characteristics determine the performance of physical layer security, the LoS channels as well as the mobility and flexibility of UAVs bring both opportunities and challenges into the physical layer security design in UAV systems. In particular, on the one hand, the LoS channel condition in UAV-based communication systems may increase the vulnerability to eavesdropping. On the other hand, the fully controllable mobility of UAVs can be exploited to enhance communication security via adjusting its trajectory. For example, if the locations of eavesdroppers are known, the UAV can avoid flying close to them to reduce the potential of information leakage for guaranteeing secure transmission. Hence, introducing UAV to wireless communication systems is a double-edged sword which requires a careful system design and thorough research.

Motivated and inspired by the unique challenges and opportunities brought by UAV, this paper aims to provide an important overview on physical layer security in UAV wireless communication systems, which sheds light on potential researches in this field. In particular, the potential security attacks specifically in UAV wireless communication systems are discussed. Moreover, corresponding solutions are provided in terms of emerging techniques by considering both the advantages and limitations of UAV. To further enhance the physical layer security, the applications of advanced 5G technologies, e.g., NOMA, 3D beamforming, and mmWave, are investigated in UAV wireless communication systems. Finally, potential research challenges that have not been addressed in the literature and some future directions in terms of secure communications in UAV systems are envisioned.

# Security Attacks in UAV Systems

A typical physical layer security communication problem comprises a minimum of three nodes, i.e., a legitimate transmitter, a legitimate receiver, and a potential eavesdropper, which can be modeled by a wiretap channel . In general, secrecy capacity and secrecy outage probability are the two fundamental metrics to evaluate the physical layer security performance. Typically, denote $C_{\mathrm{M}}$ and $C_{\mathrm{E}}$ as the Shannon capacities of the main channel and eavesdropping channel, respectively. Consequently, the secrecy capacity $C_{\mathrm{S}}$ is defined by $$\begin{aligned}
\label{eq:Secrecy_Capacity}
C_{\mathrm{S}} = [C_{\mathrm{M}}-C_{\mathrm{E}}]^+,
\end{aligned}$$ where $[x]^+\triangleq \max\{x,0\}$. Notably, perfectly secure communication between a legitimate pair is possible, when the eavesdropper's channel is a degraded version of the main channel.

In UAV-based communication systems, a UAV can act as either a transmitter or a receiver. On the one hand, it is noted that when the UAV is a legitimate transmitter, the associated air-to-ground LoS channels facilitate the signal reception at both legitimate receiver and the eavesdropper, which may increase the vulnerability to potential eavesdropping. On the other hand, when the UAV acts as a legitimate receiver, the existence of LoS channels enhance the physical layer security especially against passive eavesdropping. As such, to investigate the security issues in the presence of eavesdroppers, we mainly focus on the former scenario where the UAV acts as a legitimate transmitter. The potential security attacks in UAV systems are illustrated in Figure and will be described in the following sections.

## Passive Eavesdropping Scenarios

In this scenario, passive eavesdroppers intend to intercept some confidential data without degrading the received signal quality at the legitimate receiver. Since the eavesdroppers work in a passive manner and only intend to intercept the confidential messages, they usually remain silent and their position information is not easy to be obtained by the UAV, which makes the system vulnerable to eavesdropping. In the following, we discuss three cases with different availabilities of the eavesdroppers' position information.

### Full Position Information of Eavesdroppers

An optical camera or a synthetic aperture radar equipped at the UAV can help to detect and track the positions of potential external eavesdroppers. If the eavesdroppers stay stationary, obtaining the complete positions of these external eavesdroppers are possible at the UAV. As such, the security issues are likely to be addressed in such scenarios via proper resource allocation design and taking advantage of the flexibility of UAV. However, obtaining such precise position information requires a high hardware cost. Besides, the required new equipments impose an extra load to UAV which may increase its energy consumption.

### Partial Position Information of Eavesdroppers

When the eavesdroppers stay quasi-stationary or move in certain areas, obtaining precise position information of eavesdroppers is challenging and expensive in terms of hardware cost and energy consumption. Then, with the aid of the equipped camera or radar on the UAV, partial position information of these eavesdroppers is possible to be obtained at the UAV via sensing and tracking. In the literature, worst-case secrecy capacity and secrecy outage probability are the two common metrics to evaluate the performance of UAV systems with partial position information of external eavesdroppers .

### Absence of Position Information of Eavesdroppers

When eavesdroppers hide themselves physically, it is difficult for the UAV to detect and track them. This raises a problem that the completely absence of position information makes the UAV system extremely vulnerable to eavesdropping. In such scenarios, perfectly secure communications cannot be always guaranteed.

## Active Eavesdropping Scenarios

Compared to passive eavesdroppings, active ones are more dangerous. The reasons are twofold. On the one hand, the active eavesdropping intends to attack the main channel by degrading the channel capacity. For instance, active eavesdroppers transmit jamming signals to the legitimate receiver for degrading the capacity of the main channel, i.e., $C_{\mathrm{M}}$. On the other hand, the active eavesdropping may also aim to improve the capacity of the eavesdropping channel. As such, eavesdropping attacks caused by an active eavesdropper is more harmful than that of passive ones. Specifically, the LoS channel characteristic can improve the capacity of the eavesdropping channel when the UAV is a legitimate transmitter. According to the duplexing mode, we discuss the following two types of jamming attacks in UAV wireless communication systems:

-   *Full-duplex (FD) eavesdropper:* A FD active eavesdropper transmits jamming noise and intercepts confidential signals simultaneously and independently . In such scenarios, the UAV would increase its transmit power to improve the quality of the jammed channel which not only facilitates the eavesdropping at the FD eavesdropper due to the LoS channel characteristics, but also consumes more energy to combat the active eavesdroppings.

-   *Cooperative half-duplex (HD) eavesdroppers:* Multiple HD eavesdroppers can mimic a FD eavesdropper via cooperation. For instance, some eavesdroppers transmit jamming to the legitimate receiver, while the others intercept the confidential signal.

In both situations, when the CSI of the legitimate receiver is available at the active eavesdroppers, the UAV systems become severely vulnerable, as the active eavesdroppers can efficiently interfere the legitimate receiver.

# Physical Layer Security Techniques for UAV Systems

To enhance the physical layer security, the unique properties of UAVs, e.g., high mobility and flexibility in positioning, can be exploited. Moreover, by using advanced resource allocation techniques, the secrecy performance can be further improved. In the following, we discuss the specific effective physical layer security strategies against the corresponding eavesdroppers listed in Section in air-to-ground channels, as shown in Table .

Specific Security Attacks and Possible Solutions in UAV Wireless Communication Systems

Security attacks
Passive eavesdroppings
FD/Cooperative HD active eavesdroppings

3-8
Full PI
Partial PI
Absence of PI
Full PI
Partial PI
Absence of PI

Techniques
Joint design

Robust design

Artificial noise

Multi-UAV CoMP

1-8
Multi-antenna

2-8
NOMA

2-8
mmWave

Specific Security Attacks and Possible Solutions in UAV Wireless Communication Systems

PI: Position information

Joint design: Joint trajectory and resource allocation design

## Anti-Eavesdropping Techniques

### Joint Trajectory and Resource Allocation Design

In UAV systems, there are various resources needed to be allocated, such as transmit power, cruising speed, time slot, and frequency bandwidth. It is noted that resource allocation affects the signal strength received at not only the legitimate receiver, but also the eavesdroppers. In the following, we first discuss how trajectory design can be exploited to improve the physical layer security in UAV systems. We then provide an overview on the joint design of resource allocation and trajectory for physical layer security provisioning.

#### Trajectory Design Approach

In practice, the initial and final locations of the UAV are usually predetermined. Besides, the UAV is also constrained by the maximum cruising speed and duration. These define the maximum service area. As a result, UAV trajectory design has become an important research topic for realizing efficient UAV-based communication systems . In terms of communication security, the UAV can fly close to the legitimate ground node and away from the eavesdropper if it is possible. The principle is to carefully design the trajectory of UAV, such that the legitimate link can be enhanced while the eavesdropping link becomes weaken.

#### Resource Allocation

Specifically, a joint design of trajectory and resource allocation is a promising approach to further enhance physical layer security, e.g., . The basic principle of the joint design is that when a UAV has to fly close to the eavesdropper, the UAV can decrease or shut down the transmission power to reduce the potential of information leakage. At the same time, the UAV flies away from the eavesdropper with its full speed for saving more time slots for the future. In contrast, when the UAV flies close to the legitimate receiver, the UAV usually slows down and increases its transmit power for the confidential information transmission.

We provide an example to show how UAV can be adopted as a mobile relay to guarantee secure communications via joint trajectory and resource allocation design. In the considered scenario, a legitimate transmitter intends to serve a legitimate receiver in the presence of a passive eavesdropper with the aid of a UAV. The UAV is introduced as a mobile relay to complete the data delivery and to enable physical layer security. The ultimate goal of the system design is to maximize the spectral efficiency of the system, while guaranteeing secure communications and subject to certain practical constraints as in . The joint trajectory and resource allocation design can be obtained by iteratively solving the corresponding approximated optimization problem.

Figure provides some interesting insights into the trajectory and transmission performance during the operation period. In this figure, the legitimate transmitter and its receiver locate at $(0, 0, 0)$ and $(5$ km, $0,0)$, respectively, in a 3D Cartesian coordinate system. The position of the eavesdropper is assumed to be available at the UAV for the optimal design. The maximum cruising speed of the UAV is $20$ km/h.

Figure (a) and Figure (b) depict the optimized trajectories of a UAV for two interesting scenarios, where the eavesdropper dynamically moves anticlockwise and clockwise, respectively, around the legitimate receiver with radius $1$ km and velocity $50$ km/h. Variables $\mathbf{s}_t$, $\mathbf{s}_r$, and $\mathbf{s}_e$ represent locations of the legitimate transmitter, legitimate receiver, and eavesdropper, respectively. $\mathbf{q}_0$ and $\mathbf{q}_N$ represent the predetermined initial and final locations of UAV, respectively. From this figure, we find that the UAV first flies close to the transmitter for caching enough data. Then it approaches the legitimate receiver and avoids any possible trajectories leading close to the eavesdropper. Figure (c) and Figure (d) depict the capacities and the corresponding cruising speed of the UAV, respectively, via the optimal joint design. In these two figures, the eavesdropper locates stationary at ($4$ km, $0, 0)$. $C_\mathrm{U}$ denotes the instantaneous achievable rate at the UAV. From Figure (c) and Figure (d), we find that when the UAV locates closer to the eavesdropper than to the legitimate receiver, the UAV either caches data from the legitimate transmitter by slowing down and then hovering above it or keeps silent and moves away from the eavesdropper. Differently, when the UAV is closer to the legitimate receiver, the UAV transmits confidential signal with a positive secrecy rate and tries to hover above the desired receiver as long as possible. These indicate that joint trajectory and resource allocation design in UAV systems is an effective way to enhance the physical layer security and improve spectral efficiency.

### Robust Joint Design

When only partial or statistical position information of eavesdroppers is available, robust joint design can be exploited to facilitate secure UAV communications by considering the worst case scenario. For example, the uncertain location area of the potential eavesdropper can be modeled by a region with a center which is the exact location of the eavesdropper where the length of its radius is related to the amount of uncertainty. Then, the robust design for the worst-case communication security is to guarantee the QoS of the system whenever the eavesdropper is located within the region. In general, the UAV with optimized trajectory would fly as close as possible to the legitimate receiver for enhancing the capacity of the legitimate channel while cruises away from the uncertain area of the eavesdropper(s) as much as possible. Also, a higher transmit power from the UAV is adopted when the UAV is close to the legitimate user to exploit the short distance communication between the transceivers. However, when the UAV has to approach the uncertain area of the eavesdropper(s), it decreases or shuts down the transmit power and accelerates away from this area to reduce the potential of information leakage.

### Artificial Noise

To address the security issues caused by passive eavesdroppers with the absence of their position information, one effective approach is to transmit artificial noise to the null space of the legitimate pair's channels via performing cooperation among multiple UAVs and to transmit confidential data only when UAVs are close enough to the legitimate receiver via joint trajectory and resource allocation design. To this end, confidential data and artificial noise are available among the cooperating UAVs. As such, the capacity of the confidential signal through the legitimate channel, i.e., $C_{\mathrm{M}}$ in Eq. , is increased but that through the eavesdropping channel, i.e., $C_{\mathrm{E}}$, is reduced. However, since artificial noise consumes some transmit power, this would leave a smaller amount of system power for allocating to the confidential information signal. Therefore, power allocation is an important issue in UAV-based wireless communications for improving the system secrecy capacity. However, finding the optimal power allocation for UAV-based typically means solving non-trivial NP-hard optimization problems which are generally intractable. Hence, computationally efficient suboptimal solutions should be proposed.

## Anti-Jamming Techniques

To address the security issues caused by jamming attacks, cooperation among multiple UAVs can be exploited to enhance the physical layer security, since when multiple UAVs are available, cf. Figure , the degrees of freedom for optimizing the system resources are increased. The cooperation approaches can be summarized into the following two cases.

For instance, cooperative multi-point (CoMP) transmission technique can be employed at UAVs. In particular, these multiple UAVs can form a virtual antenna-array to enhance the received signal strength at legitimate receivers while degrade that at the eavesdroppers. Furthermore, with multiple UAVs in the system, one can optimize their trajectories and resource allocations such that some UAVs transmit confidential signal to the ground legitimate receivers while the others send jamming signals to confuse the eavesdroppers. According to the availabilities of eavesdroppers' positions, UAV can adopt different transmission schemes against active eavesdroppings:

-   When UAVs have the complete position information of the eavesdroppers, multiple UAVs can be employed to facilitate a joint trajectory and resource allocation design via forming a virtual antenna-array. Consequently, the confidential signal transmission can be enhanced by focusing the information energy beams to the ground legitimate receiver while reducing the possibility of information leakage.

-   When UAVs have only partial position or statistical information of the eavesdropper, robust joint trajectory and resource allocation design can be introduced into multi-UAV networks to improve the secrecy rate of the system or to reduce the secrecy outage probability below a certain level .

-   When the position information of the active eavesdropper is completely absent at UAVs, multi-UAV applying cooperative jamming to the orthogonal space of the legitimate receiver's channel and position adjustment are effective methods to degrade the received signals at eavesdroppers and enhance that at the legitimate receiver . Moreover, multi-antenna techniques can be adopted to utilize spatial degrees of freedom, hence enhance the quality of the received signal at the legitimate receiver, and reduce the potential information leakage to active eavesdroppers via precise beamformer design. Furthermore, one can exploit the huge bandwidth of mmWave frequency band to avoid potential eavesdropping and take advantages of line-of-sight dominated channels in mmWave systems for realizing highly directional transmission.

*Remark:* We note that the the aforementioned anti-jamming techniques can be also adopted to reduce information leakage against passive eavesdroppings. Besides, the techniques used for addressing security issues in air-to-ground channels can be also applied to ground-to-air ones where the UAV is a legitimate receiver. For instance, the flexibility of the UAV can be leveraged against eavesdropping and jamming attacks via proper trajectory planning or cooperative receiving.

# Advanced Approaches for UAV Systems

To further enhance the system secrecy performance, some advanced techniques can also be incorporated into the UAV systems . In the sequel, we will discuss the potential applications of NOMA, beamforming, and mmWave techniques in UAV systems for improving the performance of physical layer security.

## Enhancing Physical Layer Security by NOMA

NOMA is viewed as a promising technique to provide superior spectral efficiency by multiplexing information signals at different power levels . Hence, it is expected that NOMA can bring additional rate and robustness to enhance the achievable rate in UAV physical layer security communications. Consider a scenario where a UAV acts as a relay to facilitate data delivery to two receivers with different security clearance levels within a maximum cruising duration $T$. The receiver with a lower security clearance level is a potential eavesdropper since it has a strong motivation in intercepting signals intended to a receiver with a higher security clearance. Then, when the eavesdropper suffers from a bad channel condition, NOMA is adopted to forward both confidential and public information simultaneously. Otherwise, UAV only broadcasts the public information for security issues. The mode selection between NOMA and unicast is chosen based on the results of the proposed resource allocation optimization. In particular, for maximizing the spectral efficiency, one needs to jointly optimize the transmission scheme, resource allocation, and UAV's trajectory. However, the coupled optimization variables generally result in non-convex optimization problems which are difficult to solve optimally. As an alternative, an iterative suboptimal algorithm based on successive convex approximation can be employed to facilitate a computationally efficient joint design .

Under the aforementioned optimization framework, Figure (a) depicts the optimal rate region achieved by NOMA and OMA, respectively, with different cruising durations. Figure (b) depicts the maximum accumulated public data rate achieved at the eavesdropper versus the horizontal displacement of the eavesdropper from the legitimate receiver. From Figure , we find that NOMA scheme always outperforms OMA in all the considered scenarios, which demonstrates the spectral efficiency advantage brought by NOMA in striking a balance between public data rate and confidential data rate. Moreover, based on Figure (b), we find that the accumulated rate drops rapidly when the eavesdropper is sufficiently close to the legitimate receiver. In fact, when the secrecy rate requirement cannot be satisfied, we set the accumulated public data rate as zero to account for the penalty of the failure in guaranteeing secure communications.

## Enhancing Physical Layer Security by Multi-antenna Technology

Multi-antenna technologies have been widely considered in wireless communications due to spatial degrees of freedom for achieving high spectral efficiency . Recently, in order to concurrently improve the received signal power at the legitimate receiver and degrade the signal strength received at eavesdroppers, multi-antenna technologies have also been investigated from the physical layer security aspect to enhance the secrecy performance and robustness of the system. In speak of the UAV wireless communication systems, the applications of multi-antenna technology can be realized from the following two approaches, cf. Figure .

### Traditional Two-Dimensional (2D) Beamforming

The traditional 2D beamforming is an effective way to improve the physical layer security. For example, when the UAV has the complete knowledge of the CSI, the beamforming direction can be simply set pointing towards the orthogonal space of eavesdropper's channel and hence the information can be safely conveyed. However, it is obvious that this approach may not achieve the best secrecy performance of the system as the degrees of freedom are not fully utilized to improve the received signal powers at the legitimate users. Therefore, there exists a trade-off between improving the received signal strength of the legitimate users and degrading the signal qualities of the eavesdroppers.

### Three-Dimensional (3D) Beamforming

Compared to the traditional 2D beamforming, 3D beamforming can generate separated beams in the 3D space simultaneously to provide a better service coverage . Thus the 3D beamforming yields a higher system throughput and can support more legitimate users than that of the 2D beamforming. Generally, the 3D beamforming technique is more suitable for scenarios where the users are distributed in 3D space with different elevation angles to their transmitter. Due to the high altitude of UAVs, the legitimate receivers and the potential eavesdroppers can be easily separated by their different altitudes and elevation angles to the UAVs. Furthermore, LoS channel characteristic in UAV systems enables effective beamforming in both azimuth and elevation domains. Specifically, narrow and precise beamformer can be created to improve the transmission efficiency with respect to the desired legitimate receivers while reduces the possibilities of information leakage to the potential eavesdroppers.

## Enhancing Physical Layer Security by mmWave

mmWave communications have been widely studied in the literature , since they are able to support high data rate by utilizing the abundant frequency bands. One of the major challenges for mmWave communications is that its performance depends on the availability of LoS channels. Thus, compared to the conventional terrestrial communications, the inherent LoS channels in UAV systems facilitate the use of mmWave for high-data rate communications.

As demonstrated in , the mmWave channels in UAV systems are very sparse in the angular domain, as there are generally not many scatters around UAVs in the sky. Hence, one can take advantages of the specific channel characteristics in mmWave UAV systems for realizing highly directional transmission and exploit the huge bandwidth of mmWave frequency spectrum to avoid potential eavesdropping. For instance, active eavesdropping attacks in UAV systems can be addressed by adopting frequency hopping (FH) technique to hide confidential data over an ultra-wide mmWave frequency spectrum. There are two main advantages of this technique. First, confidential data can be hided from the eavesdropper by frequently hopping to different carrier frequencies without exploiting the CSI. Second, frequency diversity can be exploited in FH to improve the robustness against jamming active attacks.

# Open Issues and Challenges

UAV-based wireless communications is a growing research area and handling the associated security issues is the key to unlock its potential. Despite the fruitful research in this area, there are variety of challenges to be tackled, as shown in Figure . In this section, we provide and list several open issues and challenges for future works.

## Practical UAV Channel Modeling and Position Acquisition

In the literature, air-to-ground channel is generally modeled by the LoS channel. However, this model may not be accurate for some scenarios, such as urban areas. Consequently, more efforts should be devoted to realistic channel modelings and to be verified by practice field test measurements. Moreover, the mobility of UAV has significant impacts on the CSI acquisition for both the legitimate users and eavesdroppers. This brings challenges from the physical layer security perspective, as efficient CSI acquisition algorithms usually exploit the physical properties of the wireless channels. Therefore, it is necessary to design a pragmatic CSI acquisition mechanism to detect and track positions of the legitimate receivers and eavesdroppers.

## Physical Layer Security Against Pilot Contamination Attacks

In practice, efficient beamforming for secure UAV communications requires accurate CSI which can be obtained by exploiting the pilot signals. However, in some scenario, active eavesdroppers intentionally transmit deterministic pilot samples which are identical to that transmitted by the legitimate transmitter to deceive the UAV for facilitating eavesdroppings . As a result, the UAV designs an inappropriate transmission strategy which benefits the signal reception at the eavesdroppers. For example, the UAV may misjudge the actual network environment and fly close to the eavesdropper for confidential data delivery which increases the possibility of information leakage. Hence, investigating some efficient policies to eliminate the impact of pilot contamination attack is challenging but important for safeguarding UAV systems.

## Physical Layer Security Against Malicious UAV Attacks

The high mobility and flexibility of UAVs can be exploited not only to enhance physical layer security, but also to intercept the confidential data and even perform jamming to reduce the quality of the legitimate link, which may bring more challenges than handling conventional terrestrial eavesdroppers for safeguarding UAV systems. However, only limited researches have been devoted in this important aspect from communication theory perspective. Therefore, it is desired to investigate advanced techniques in terms of the physical layer security to protect against malicious UAVs.

## Optimal Joint Trajectory and Resource Allocation Design

Compare to terrestrial systems, the UAV-based system brings a new dimension to enhance physical layer security via designing the trajectory of a UAV. However, the designs of trajectory and resource allocation are generally coupled together, which would make the design optimization problem intractable. Consequently, most of existing designs are suboptimal. Unfortunately, the performance gaps between the optimal solution and existing suboptimal solutions are unclear. Therefore, to improve the performance of physical layer security in UAV systems for mission critical applications, efficient algorithms are desired to strike a balance between computational complexity and system performance.

## Limited Onboard Resources

One critical obstacle in UAV wireless communication systems arises from the restricted flight duration, limited onboard energy and computational capability, etc, as the battery capacities, sizes, and weights of UAVs are all limited. As such, investigating advanced techniques to enable sustainable secure UAV communication are desired. For example, energy harvesting from solar and laser provides a viable solution to supply energy to UAVs on-the-fly. Besides, UAV cooperation can effectively leverage the onboard resources among all the cooperative UAVs to enable secure communication.

# Conclusion

This article provides an overview and comprehensive discussions on the specific security issues in UAV systems. Challenges and opportunities brought by UAVs for safeguarding UAV-based communication systems via physical layer security are fully exploited. First, key problems in terms of security attacks are revealed. Then, physical layer security approaches in UAV systems are proposed to effectively prevent from both passive and active eavesdroppings. Particularly, we provide an illustrative example on guaranteeing security provisioning by jointly designing UAV's trajectory and resource allocation, which demonstrates the advantages brought by the flexibility of UAVs. Moreover, we propose to apply NOMA, MIMO, and mmWave techniques in UAV systems to further enhance the physical layer security and the spectral efficiency. Finally, some potential research directions and challenges are envisioned.

# Acknowledgment

This work was supported in part by funding from the UNSW Digital Grid Futures Institute, UNSW, Sydney, under a cross-disciplinary fund scheme, in part by the Australian Research Council's Discovery Project (DP190101363), in part by the UK EPSRC under grant number EP/P009719/2, and in part by H2020-MSCA-RISE-2015 under grant number 690750.

[^1]: X. Sun, Y. Xu, and Z. Zhong are with the State Key Lab of Rail Traffic Control and Safety and the Beijing Engineering Research Center of High-speed Railway Broadband Mobile Communications, Beijing Jiaotong University, Beijing 100044, China (emails: {xiaofangsun, yanqing_xu, email@example.com).

[^2]: D. W. K. Ng is with the School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia (Email: firstname.lastname@example.com).

[^3]: Z. Ding is with the School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UK (Email: firstname.lastname@example.com).