Question: Represent the article for finding a single-sentence TLDR

Question: A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel – from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others’ updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner’s Dilemma. Although experimentally successful, we show that LOLA agents can exhibit ‘arrogant’ behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally. Problem Setting. While machine learning has traditionally focused on optimising single objectives, generative adversarial nets (GANs) BID9 have showcased the potential of architectures dealing with multiple interacting goals. They have since then proliferated substantially, including intrinsic curiosity BID19, imaginative agents BID20, synthetic gradients, hierarchical reinforcement learning (RL) BID23 BID22 and multi-agent RL in general BID2.These can effectively be viewed as differentiable games played by cooperating and competing agents -which may simply be different internal components of a single system, like the generator and discriminator in GANs. The difficulty is that each loss depends on all parameters, including those of other agents. While gradient descent on single functions has been widely successful, converging to local minima under rather mild conditions BID13, its simultaneous generalisation can fail even in simple two-player, two-parameter zero-sum games. No algorithm has yet been shown to converge, even locally, in all differentiable games. Related Work. Convergence has widely been studied in convex n-player games, see especially BID21; BID5. However, the recent success of non-convex games exemplified by GANs calls for a better understanding of this general class where comparatively little is known. BID14 recently prove local convergence of no-regreat learning to variationally stable equilibria, though under a number of regularity assumptions. Conversely, a number of algorithms have been successful in the non-convex setting for restricted classes of games. These include policy prediction in two-player two-action bimatrix games BID24; WoLF in two-player two-action games BID1; AWESOME in repeated games BID3; Optimistic Mirror Descent in two-player bilinear zero-sum games BID4 and Consensus Optimisation (CO) in two-player zerosum games BID15 ). An important body of work including BID10; BID16 has also appeared for the specific case of GANs. Working towards bridging this gap, some of the authors recently proposed Symplectic Gradient Adjustment (SGA), see BID0. This algorithm is provably'attracted' to stable fixed points while'repelled' from unstable ones in all differentiable games (n-player, non-convex). Nonetheless, these are weaker than strict convergence guarantees. Moreover, SGA agents may act against their own self-interest by prioritising stability over individual loss. SGA was also discovered independently by BID8, drawing on variational inequalities. In a different direction, Learning with Opponent-Learning Awareness (LOLA) modifies the learning objective by predicting and differentiating through opponent learning steps. This is intuitively appealing and experimentally successful, encouraging cooperation in settings like the Iterated Prisoner's Dilemma (IPD) where more stable algorithms like SGA defect. However, LOLA has no guarantees of converging or even preserving fixed points of the game. Contribution. We begin by constructing the first explicit tandem game where LOLA agents adopt'arrogant' behaviour and converge to non-fixed points. We pinpoint the cause of failure and show that a natural variant named LookAhead (LA), discovered before LOLA by BID24, successfully preserves fixed points. We then prove that LookAhead locally converges and avoids strict saddles in all differentiable games, filling a theoretical gap in multi-agent learning. This is enabled through a unified approach based on fixed-point iterations and dynamical systems. These techniques apply equally well to algorithms like CO and SGA, though this is not our present focus. While LookAhead is theoretically robust, the shaping component endowing LOLA with a capacity to exploit opponent dynamics is lost. We solve this dilemma with an algorithm named Stable Opponent Shaping (SOS), trading between stability and exploitation by interpolating between LookAhead and LOLA. Using an intuitive and theoretically grounded criterion for this interpolation parameter, SOS inherits both strong convergence guarantees from LA and opponent shaping from LOLA.On the experimental side, we show that SOS plays tit-for-tat in the IPD on par with LOLA, while all other methods mostly defect. We display the practical consequences of our theoretical guarantees in the tandem game, where SOS always outperforms LOLA. Finally we implement a more involved GAN setup, testing for mode collapse and mode hopping when learning Gaussian mixture distributions. SOS successfully spreads mass across all Gaussians, at least matching dedicated algorithms like CO, while LA is significantly slower and simultaneous gradient descent fails entirely. We frame the problem of multi-agent learning as a game. Adapted from BID0, the following definition insists only on differentiability for gradient-based methods to apply. This concept is strictly more general than stochastic games, whose parameters are usually restricted to action-state transition probabilities or functional approximations thereof. Definition 1. A differentiable game is a set of n players with parameters θ = (θ 1, . . ., θ n) ∈ R d and twice continuously differentiable losses DISPLAYFORM0 Crucially, note that each loss is a function of all parameters. From the viewpoint of player i, parameters can be written as θ = (θ i, θ −i) where θ −i contains all other players' parameters. We do not make the common assumption that each L i is convex as a function of θ i alone, for any fixed opponent parameters θ −i, nor do we restrict θ to the probability simplex -though this restriction can be recovered via projection or sigmoid functions σ: R →. If n = 1, the'game' is simply to minimise a given loss function. In this case one can reach local minima by (possibly stochastic) gradient descent (GD). For arbitrary n, the standard solution concept is that of Nash equilibria. Definition 2. A pointθ ∈ R d is a (local) Nash equilibrium if for each i, there are neighbourhoods DISPLAYFORM1 In other words, each player's strategy is a local best response to current opponent strategies. We write DISPLAYFORM2 Define the simultaneous gradient of the game as the concatenation of each player's gradient, DISPLAYFORM3 The ith component of ξ is the direction of greatest increase in L i with respect to θ i. If each agent minimises their loss independently from others, they perform GD on their component ∇ i L i with learning rate α i. Hence, the parameter update for all agents is given by θ ← θ − α ξ, where α = (α 1, . . ., α n) and is element-wise multiplication. This is also called naive learning (NL), reducing to θ ← θ − αξ if agents have the same learning rate. This is assumed for notational simplicity, though irrelevant to our . The following example shows that NL can fail to converge. Example 1. Consider L 1/2 = ±xy, where players control the x and y parameters respectively. The origin is a (global and unique) Nash equilibrium. The simultaneous gradient is ξ = (y, −x) and cycles around the origin. Explicitly, a gradient step from (x, y) yields (x, y) ← (x, y) − α(y, −x) = (x − αy, y + αx) which has distance from the origin (1 + α 2)(x 2 + y 2) > (x 2 + y 2) for any α > 0 and (x, y) = 0. It follows that agents diverge away from the origin for any α > 0. The cause of failure is that ξ is not the gradient of a single function, implying that each agent's loss is inherently dependent on others. This in a contradiction between the non-stationarity of each agent, and the optimisation of each loss independently from others. Failure of convergence in this simple two-player zero-sum game shows that gradient descent does not generalise well to differentiable games. We consider an alternative solution concept to Nash equilibria before introducing LOLA. Consider the game given by L 1 = L 2 = xy where players control the x and y parameters respectively. The optimal solution is (x, y) DISPLAYFORM0 However the origin is a global Nash equilibrium, while also a saddle point of xy. It is highly undesirable to converge to the origin in this game, since infinitely better losses can be reached in the anti-diagonal direction. In this light, Nash equilibria cannot be the right solution concept to aim for in multi-agent learning. To define stable fixed points, first introduce the'Hessian' of the game as the block matrix DISPLAYFORM1 This can equivalently be viewed as the Jacobian of the vector field ξ. Importantly, note that H is not symmetric in general unless n = 1, in which case we recover the usual Hessian H = ∇ 2 L.Definition 3. A pointθ is a fixed point if ξ(θ) = 0. It is stable if H(θ) 0, unstable if H(θ) ≺ 0 and a strict saddle if H(θ) has an eigenvalue with negative real part. The name'fixed point' is coherent with GD, since ξ(θ) = 0 implies a fixed updateθ ←θ − αξ(θ) = θ. Though Nash equilibria were shown to be inadequate above, it is not obvious that stable fixed points (SFPs) are a better solution concept. In Appendix A we provide intuition for why SFPs are both closer to local minima in the context of multi-loss optimisation, and more tractable for convergence proofs. Moreover, this definition is an improved variant on that in BID0, assuming positive semi-definiteness only atθ instead of holding in a neighbourhood. This makes the class of SFPs as large as possible, while sufficient for all our theoretical . Assuming invertibility of H(θ) at SFPs is crucial to all convergence in this paper. The same assumption is present in related work including BID15, and cannot be avoided. Even for single losses, a fixed point with singular Hessian can be a local minimum, maximum, or saddle point. Invertibility is thus necessary to ensure that SFPs really are'local minima'. This is omitted from now on. Finally note that unstable fixed points are a subset of strict saddles, making Theorem 6 both stronger and more general than for SGA by BID0. Accounting for nonstationarity, Learning with Opponent-Learning Awareness (LOLA) modifies the learning objective by predicting and differentiating through opponent learning steps. For simplicity, if n = 2 then agent 1 optimises L 1 (θ 1, θ 2 + ∆θ 2) with respect to θ 1, where ∆θ 2 is the predicted learning step for agent 2. assume that opponents are naive learners, namely ∆θ 2 = −α 2 ∇ 2 L 2. After first-order Taylor expansion, the loss is approximately given by L 1 + ∇ 2 L 1 · ∆θ 2. By minimising this quantity, agent 1 learns parameters that align the opponent learning step ∆θ 2 with the direction of greatest decrease in L 1, exploiting opponent dynamics to further reduce one's losses. Differentiating with respect to θ 1, the adjustment is DISPLAYFORM0 By explicitly differentiating through ∆θ 2 in the rightmost term, LOLA agents actively shape opponent learning. This has proven effective in reaching cooperative equilibria in multi-agent learning, finding success in a number of games including tit-for-tat in the IPD. The middle term above was originally dropped by the authors because "LOLA focuses on this shaping of the learning direction of the opponent". We choose not to eliminate this term, as also inherent in LOLA-DiCE. Preserving both terms will in fact be key to developing stable opponent shaping. First we formulate n-player LOLA in vectorial form. Let H d and H o be the matrices of diagonal and anti-diagonal blocks of H, so that DISPLAYFORM1 While experimentally successful, LOLA fails to preserve fixed pointsθ of the game since DISPLAYFORM2 in general. Even ifθ is a Nash equilibrium, the updateθ ←θ − αLOLA =θ can push them away despite parameters being optimal. This may worsen the losses for all agents, as in the game below. Example 2 (Tandem). Imagine a tandem controlled by agents facing opposite directions, who feed x and y force into their pedals respectively. Negative numbers correspond to pedalling backwards. Moving coherently requires x ≈ −y, embodied by a quadratic loss (x+y) 2. However it is easier for agents to pedal forwards, translated by linear losses −2x and −2y. The game is thus given by L 1 (x, y) = (x + y) 2 − 2x and L 2 (x, y) = (x + y) 2 − 2y. These sub-goals are incompatible, so agents cannot simply accelerate forwards. The SFPs are given by {x + y = 1}. Computing χ (x, 1 − x) = = 0, none of these are preserved by LOLA. Instead, we show in Appendix C that LOLA can only converge to sub-optimal scenarios with worse losses for both agents, for any α. Intuitively, the root of failure is that LOLA agents try to shape opponent learning and enforce compliance by accelerating forwards, assuming a dynamic response from their opponent. The other agent does the same, so they become'arrogant' and suffer by pushing strongly in opposite directions. The shaping term χ prevents LOLA from preserving fixed points. Consider removing this component entirely, giving (I − αH o)ξ. This variant preserves fixed points, but what does it mean from the perspective of each agent? Note that LOLA optimises L 1 (θ 1, θ 2 + ∆θ 2) with respect to θ 1, while ∆θ 2 is a function of θ 1. In other words, we assume that our opponent's learning step depends on our current optimisation with respect to θ 1. This is inaccurate, since opponents cannot see our updated parameters until the next step. Instead, assume we optimise DISPLAYFORM0 are the current parameters. After Taylor expansion, the gradient with respect to θ 1 is given by DISPLAYFORM1 does not depend on θ 1. In vectorial form, we recover the variant (I −αH o)ξ since the shaping term corresponds precisely to differentiating through ∆θ 2. We name this LookAhead, which was discovered before LOLA by BID24 though not explicitly named. Using the stop-gradient operator ⊥ 1, this can be reformulated as optimising L 1 (θ 1, θ 2 + ⊥∆θ 2) where ⊥ prevents gradient flowing from ∆θ 2 upon differentiation. The main of BID24 is that LookAhead converges to Nash equilibria in the small class of two-player, two-action bimatrix games. We will prove local convergence to SFP and non-convergence to strict saddles in all differentiable games. On the other hand, by discarding the problematic shaping term, we also eliminated LOLA's capacity to exploit opponent dynamics and encourage cooperation. This will be witnessed in the IPD, where LookAhead agents mostly defect. We propose Stable Opponent Shaping (SOS), an algorithm preserving both advantages at once. Define the partial stop-gradient operator ⊥ p:= p⊥ + (1 − p)I, where I is the identity and p stands for partial. A p-LOLA agent optimises the modified objective DISPLAYFORM0 collapsing to LookAhead at p = 0 and LOLA at p = 1. The ing gradient is given by DISPLAYFORM1 We obtain an algorithm trading between shaping and stability as a function of p. Note however that preservation of fixed points only holds if p is infinitesimal, in which case p-LOLA is almost identical to LookAhead -losing the very purpose of interpolation. Instead we propose a two-part criterion for p at each learning step, through which all guarantees descend. First choose p such that ξ p points in the same direction as LookAhead. This will not be enough to prove convergence itself, but prevents arrogant behaviour by ensuring convergence only to fixed points. Formally, the first criterion is given by ξ p, ξ 0 ≥ 0. If −α χ, ξ 0 ≥ 0 then ξ p, ξ 0 ≥ 0 automatically, so we choose p = 1 for maximal shaping. Otherwise choose DISPLAYFORM2 −α χ, ξ 0 with any hyperparameter 0 < a < 1. This guarantees a positive inner product DISPLAYFORM3 We complement this with a second criterion ensuring local convergence. The idea is to scale p by a function of ξ if ξ is small enough, which certainly holds in neighbourhoods of fixed points. Let 0 < b < 1 be a hyperparameter and take p = ξ 2 if ξ < b, otherwise p = 1. Choosing p 1 and p 2 according to these criteria, the two-part criterion is p = min{p 1, p 2}. SOS is obtained by combining p-LOLA with this criterion, as summarised in Algorithm 1. Crucially, all theoretical in the next section are independent from the choice of hyperparameters a and b. Algorithm 1: Stable Opponent Shaping 1 Initialise θ randomly and fix hyperparameters a, b ∈. 2 while not done do DISPLAYFORM4 Our central theoretical contribution is that LookAhead and SOS converge locally to SFP and avoid strict saddles in all differentiable games. Since the learning gradients involve second-order Hessian terms, our assume thrice continuously differentiable losses (omitted hereafter). Losses which are C 2 but not C 3 are very degenerate, so this is a mild assumption. Statements made about SOS crucially hold for any hyperparameters a, b ∈. See Appendices D and E for detailed proofs. Convergence is proved using Ostrowski's Theorem. This reduces convergence of a gradient adjustment g to positive stability (eigenvalues with positive real part) of ∇g at stable fixed points. Theorem 2. Let H 0 be invertible with symmetric diagonal blocks. Then there exists > 0 such that (I − αH o)H is positive stable for all 0 < α <.This type of would usually be proved either by analytical means showing positive definiteness and hence positive stability, or direct eigenvalue analysis. We show in Appendix D that (I − αH o)H is not necessarily positive definite, while there is no necessary relationship between eigenpairs of H and H o. This makes our theorem all the more interesting and non-trivial. We use a similarity transformation trick to circumvent the dual obstacle, allowing for analysis of positive definiteness with respect to a new inner product. We obtain positive stability by invariance under change of basis. Corollary 3. LookAhead converges locally to stable fixed points for α > 0 sufficiently small. Using the second criterion for p, we prove local convergence of SOS in all differentiable games despite the presence of a shaping term (unlike LOLA). Theorem 4. SOS converges locally to stable fixed points for α > 0 sufficiently small. Using the first criterion for p, we prove that SOS only converges to fixed points (unlike LOLA). Proposition 5. If SOS converges toθ and α > 0 is small thenθ is a fixed point of the game. Now assume that θ is initialised randomly (or with arbitrarily small noise), as is standard in ML. Let F (θ) = θ − αξ p (θ) be the SOS iteration. Using both the second criterion and the Stable Manifold Theorem from dynamical systems, we can prove that every strict saddleθ has a neighbourhood U such that {θ ∈ U | F n (θ) →θ as n → ∞} has measure zero for α > 0 sufficiently small. Since θ is initialised randomly, we obtain the following . Theorem 6. SOS locally avoids strict saddles almost surely, for α > 0 sufficiently small. This also holds for LookAhead, and could be strenghtened to global initialisations provided a strong boundedness assumption on H 2. This is trickier for SOS since p(θ) is not globally continuous. Altogether, our for LookAhead and the correct criterion for p-LOLA lead to some of the strongest theoretical guarantees in multi-agent learning. Furthermore, SOS retains all of LOLA's opponent shaping capacity while LookAhead does not, as shown experimentally in the next section. We evaluate the performance of SOS in three differentiable games. We first showcase opponent shaping and superiority over LA/CO/SGA/NL in the Iterated Prisoner's Dilemma (IPD). This leaves SOS and LOLA, which have differed only in theory up to now. We bridge this gap by showing that SOS always outperforms LOLA in the tandem game, avoiding arrogant behaviour by decaying p while LOLA overshoots. Finally we test SOS on a more involved GAN learning task, with similar to dedicated methods like Consensus Optimisation. IPD: This game is an infinite sequence of the well-known Prisoner's Dilemma, where the payoff is discounted by a factor γ ∈ at each iteration. Agents are endowed with a memory of actions at the previous state. Hence there are 5 parameters for each agent i: the probability P i (C | state) of cooperating at start state s 0 = ∅ or state s t = (a 1 t−1, a 2 t−1) for t > 0. One Nash equilibrium is to always defect (DD), with a normalised loss of 2. A better equilibrium with loss 1 is named tit-for-tat (TFT), where each player begins by cooperating and then mimicks the opponent's previous action. We run 300 training episodes for SOS, LA, CO, SGA and NL. The parameters are initialised following a normal distribution around 1/2 probability of cooperation, with unit variance. We fix α = 1 and γ = 0.96, following. We choose a = 0.5 and b = 0.1 for SOS. The first is a robust and arbitrary middle ground, while the latter is intentionally small to avoid poor SFP. Tandem: Though local convergence is guaranteed for SOS, it is possible that SOS diverges from poor initialisations. This turns out to be impossible in the tandem game since the Hessian is globally positive semi-definite. We show this explicitly by running 300 training episodes for SOS and LOLA. Parameters are initialised following a normal distribution around the origin. We found performance to be robust to hyperparameters a, b. Here we fix a = b = 0.5 and α = 0.1. We reproduce a setup from BID0. The game is to learn a Gaussian mixture distribution using GANs. Data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training -see ground truth in Appendix F. The generator and discriminator networks each have 6 ReLU layers of 384 neurons, with 2 and 1 output neurons respectively. Learning rates are chosen by grid search at iteration 8k, with a = 0.5 and b = 0.1 for SOS, following the same reasoning as the IPD. IPD: Results are given in FIG1. Parameters in part (A) are the end-run probabilities of cooperating for each memory state, encoded in different colours. Only 50 runs are shown for visibility. Losses at each step are displayed in part (B), averaged across 300 episodes with shaded deviations. SOS and LOLA mostly succeed in playing tit-for-tat, displayed by the accumulation of points in the correct corners of (A) plots. For instance, CC and CD points are mostly in the top right and left corners so agent 2 responds to cooperation with cooperation. Agents also cooperate at the start state, represented by ∅ points all hidden in the top right corner. Tit-for-tat strategy is further indicated by the losses close to 1 in part (B). On the other hand, most points for LA/CO/SGA/NL are accumulated at the bottom left, so agents mostly defect. This in poor losses, demonstrating the limited effectiveness of recent proposals like SGA and CO. Finally note that trained parameters and losses for SOS are almost identical to those for LOLA, displaying equal capacity in opponent shaping while also inheriting convergence guarantees and outperforming LOLA in the next experiment. Tandem: Results are given in Figure 3. SOS always succeeds in decreasing p to reach the correct equilibria, with losses averaging at 0. LOLA fails to preserve fixed points, overshooting with losses averaging at 4/9. The criterion for SOS is shown in action in part (B), decaying p to avoid overshooting. This illustrates that purely theoretical guarantees descend into practical outperfor- mance. Note that SOS even gets away from the LOLA fixed points if initialised there (not shown), converging to improved losses using the alignment criterion with LookAhead. The generator distribution and KL divergence are given at {2k, 4k, 6k, 8k} iterations for NL, CO and SOS in Figure 4. Results for SGA, LOLA and LA are in Appendix F. SOS achieves convincing by spreading mass across all Gaussians, as do CO/SGA/LOLA. LookAhead is significantly slower, while NL fails through mode collapse and hopping. Only visual inspection was used for comparison by BID0, while KL divergence gives stronger numerical evidence here. SOS and CO are slightly superior to others with reference to this metric. However CO is aimed specifically toward two-player zero-sum GAN optimisation, while SOS is widely applicable with strong theoretical guarantees in all differentiable games. Theoretical in machine learning have significantly helped understand the causes of success and failure in applications, from optimisation to architecture. While gradient descent on single losses has been studied extensively, algorithms dealing with interacting goals are proliferating, with little grasp of the underlying dynamics. The analysis behind CO and SGA has been helpful in this respect, though lacking either in generality or convergence guarantees. The first contribution of this paper is to provide a unified framework and fill this theoretical gap with robust convergence for LookAhead in all differentiable games. Capturing stable fixed points as the correct solution concept was essential for these techniques to apply. Furthermore, we showed that opponent shaping is both a powerful approach leading to experimental success and cooperative behaviour -while at the same time preventing LOLA from preserving fixed points in general. This conundrum is solved through a robust interpolation between LookAhead and LOLA, giving birth to SOS through a robust criterion. This was partially enabled by choosing to preserve the'middle' term in LOLA, and using it to inherit stability from LookAhead. This in convergence guarantees stronger than all previous algorithms, but also in practical superiority over LOLA in the tandem game. Moreover, SOS fully preserves opponent shaping and outperforms SGA, CO, LA and NL in the IPD by encouraging tit-for-tat policy instead of defecting. Finally, SOS convincingly learns Gaussian mixtures on par with the dedicated CO algorithm. In the main text we showed that Nash equilibria are inadequate in multi-agent learning, exemplified by the simple game given by L 1 = L 2 = xy, where the origin is a global Nash equilibrium but a saddle point of the losses. It is not however obvious that SFP are a better solution concept. We begin by pointing out that for single losses, invertibility and symmetry of the Hessian imply positive definiteness at SFP. These are exactly local minima of L detected by the second partial derivative test, namely those points provably attainable by gradient descent. To emphasise this, note that gradient descent does not converge locally to all local minima. This can be seen by considering the example L(x, y) = y 2 and the local (global) minimum. There is no neighbourhood for which gradient descent converges to, since initialising at (x 0, y 0) will always converge to (x 0, 0) for appropriate learning rates, with x 0 = 0 almost surely. This occurs precisely because the Hessian is singular at. Though a degenerate example, this suggests an important difference to make between the ideal solution concept (local minima) and that for which local convergence claims are possible to attain (local minima with invertible H 0). Accordingly, the definition of SFP is the immediate generalisation of'fixed points with positive semi-definite Hessian', or in other words,'second-order-tractable local minima'. It is important to impose only positive semi-definiteness to keep the class as large as possible, despite strict positive definiteness holding for single losses due to symmetry. Imposing strict positivity would for instance exclude the origin in the cyclic game L 1 = xy = −L 2, a point certainly worthy of convergence. Note also that imposing a weaker condition than H 0 would be incorrect. Invertibility aside, local convergence of gradient descent on single functions cannot be guaranteed if H 0, since such points are strict saddles. These are almost always avoided by gradient descent, as proven by BID12 and. It is thus necessary to impose H 0 as a minimal requirement in optimisation methods attempting to generalise gradient descent. Remark A.1. A matrix H is positive semi-definite iff the same holds for its symmetric part S = (H + H)/2, so SFP could equivalently be defined as S(θ) 0. This is the original formulation given by part of the authors BID0, who also imposed the extra requirement S(θ) 0 in a neighbourhood ofθ. After discussion we decided to drop this assumption, pointing out that it is 1) more restrictive, 2) superficial to all theoretical and 3) weakens the analogy with tractable local minima. The only thing gained by imposing semi-positivity in a neighbourhood is that SFP become a subset of Nash equilibria. Regarding unstable fixed points and strict saddles, note that H(θ) 0 implies H(θ) 0 in a neighbourhood, hence being equivalent to the definition in BID0. It follows also that unstable points are a subset of strict saddles: if H(θ) ≺ 0 then all eigenvalues are negative since any eigenpair (v, λ) satisfies DISPLAYFORM0 We introduced strict saddles in this paper as a generalisation of unstable FP, which are more difficult to handle but nonetheless tractable using dynamical systems. The name is chosen by analogy to the definition in BID12 for single losses. Proposition B.1. The LOLA gradient adjustment is DISPLAYFORM0 in the usual assumption of equal learning rates. Proof. Recall the modified objective DISPLAYFORM1 for agent 1, and so on for each agent. First-order Taylor expansion yields DISPLAYFORM2 and similarly for each agent. Differentiating with respect to θ i, the adjustment for player i is DISPLAYFORM3 and thus DISPLAYFORM4 C TANDEM GAME We provide a more detailed exposition of the tandem game in this section, including computation of fixed points for NL/LOLA and corresponding losses. Recall that the game is given by DISPLAYFORM5 Intuitively, agents wants to have x ≈ −y since (x + y) 2 is the leading loss, but would also prefer to have positive x and y. These are incompatible, so the agents must not be'arrogant' and instead make concessions. The fixed points are given by ξ = 2(x + y − 1) 1 1 = 0, namely any pair (x, 1 − x). The corresponding losses are L 1 = 1 − 2x = −L 2, summing to 0 for any x. We have DISPLAYFORM6 everywhere, so all fixed points are SFP. LOLA fails to preserve these, since DISPLAYFORM7 which is non-zero for any SFP (x, 1 − x). Instead, LOLA can only converge to points such that DISPLAYFORM8 We solve this explicitly as follows: DISPLAYFORM9 The fixed points for LOLA are thus pairs (x, y) such that DISPLAYFORM10 This leads to worse losses DISPLAYFORM11 for agent 1 and similarly for agent 2. In particular, losses always sum to something greater than 0. This becomes negligible as the learning rate becomes smaller, but is always positive nonetheless Taking α arbitrarily small is not a viable solution since convergence will in turn be arbitrarily slow. LOLA is thus not a strong algorithm candidate for all differentiable games. We use Ostrowski's theorem as a unified framework for proving local convergence of gradient-based methods. This is a standard on fixed-point iterations, adapted from (, 10.1.3). We also invoke and prove a topological of our own, Lemma D.9, at the end of this section. This is useful in deducing local convergence, though not central to intuition. DISPLAYFORM0, and assumex ∈ Ω is a fixed point. If all eigenvalues of ∇F (x) are strictly in the unit circle of C, then there is an open neighbourhood U ofx such that for all x 0 ∈ U, the sequence F (k) (x 0) converges tox. Moreover, the rate of convergence is at least linear in k. Recall the simultaneous gradient ξ and the Hessian H defined for differentiable games. Let X be any matrix with continuously differentiable entries. Corollary D.3. Assumex is a fixed point of a differentiable game such that XH(x) is positive stable. Then the iterative procedure DISPLAYFORM1 converges locally tox for α > 0 sufficiently small. Proof. By definition of fixed points, ξ(x) = 0 and so DISPLAYFORM2 is positive stable by assumption, namely has eigenvalues a k + ib k with a k > 0. It follows that DISPLAYFORM3 has eigenvalues 1 − αa k − iαb k, which are in the unit circle for small α. More precisely, DISPLAYFORM4 which is always possible for a k > 0. Hence ∇F (x) has eigenvalues in the unit circle for 0 DISPLAYFORM5, and we are done by Ostrowski's Theorem sincex is a fixed point of F.We apply this corollary to LookAhead, which is given by DISPLAYFORM6 where X = (I − αH o). It is thus sufficient to prove the following . Theorem D.4. Let H 0 invertible with symmetric diagonal blocks. Then there exists > 0 such that (I − αH o)H is positive stable for all 0 < α <.Remark D.5. Note that (I − αH o)H may fail to be positive definite, though true in the case of 2 × 2 matrices. This no longer holds in higher dimensions, exemplified by the Hessian DISPLAYFORM7 By direct computation (symbolic in α), one can show that G = (I − αH o)H always has positive eigenvalues for small α > 0, whereas its symmetric part S always has a negative eigenvalue with magnitude in the order of α. This implies that S and in turn G is not positive definite. As such, an analytical proof of the theorem involving bounds on the corresponding bilinear form will fail. This makes the all the more interesting, but more involved. Central to the proof is a similarity transformation proving positive definiteness with respect to a different inner product, a novel technique we have not found in the multi-agent learning literature. Proof. We cannot study the eigenvalues of G directly, since there is no necessary relationship between eigenpairs of H and H o. In the aim of using analytical tools, the trick is to find a positive definite matrix which is similar to G, thus sharing the same positive eigenvalues. First define DISPLAYFORM8 where H d is the sub-matrix of diagonal blocks,and rewrite DISPLAYFORM9 Note that H d is block diagonal with symmetric blocks ∇ ii L i 0, so (I + αH d) is symmetric and positive definite for all α ≥ 0. In particular its principal square root DISPLAYFORM10 is unique and invertible. Now note that DISPLAYFORM11 which is positive semi-definite since DISPLAYFORM12 for all non-zero u. In particular M provides a similarity transformation which eliminates H d from G 1 while simultaneously delivering positive semi-definiteness. We can now prove that First note that a Taylor expansion of M in α yields DISPLAYFORM13 DISPLAYFORM14 and DISPLAYFORM15 There are two cases to distinguish. If u Hu > 0 then DISPLAYFORM0 for α sufficiently small. Otherwise, u Hu = 0 and consider decomposing H into symmetric and antisymmetric parts S = (H + H)/2 and A = (H − H)/2, so that H = S + A. By antisymmetry of A we have u Au = 0 and hence u Hu = 0 = u Su. Now H 0 implies S 0, so by Cholesky decomposition of S there exists a matrix T such that S = T T. In particular 0 = u Su = T u 2 implies T u = 0, and in turn Su = 0. Since H is invertible and u = 0, we have 0 = Hu = Au and so Au 2 > 0. It follows in particular that DISPLAYFORM1 Using positive semi-definiteness of DISPLAYFORM2 for α > 0 small enough. We conclude that for any u ∈ S m there is u > 0 such that DISPLAYFORM3 compact. By Lemma D.9, this can be extended uniformly with some > 0 such that DISPLAYFORM4 for all u ∈ S m and 0 < α <. It follows that M −1 GM is positive definite for all 0 < α < and thus G is positive stable for α in the same range, by similarity. Corollary D.6. LookAhead converges locally to stable fixed points for α > 0 sufficiently small. 0 invertible by definition, with diagonal blocks ∇ ii L i symmetric by twice continuous differentiability. We are done by the above and Corollary D.3.We now prove that local convergence descend to SOS. The following lemma establishes the crucial claim that our criterion for p is C 1 in neighbourhoods of fixed points. This is necessary to invoke analytical arguments including Ostrowski's Theorem, and would be untrue globally. Lemma D.7. Ifθ is a fixed point and α is sufficiently small then p = ξ 2 in a neighbourhood ofθ. Proof. First note that ξ(θ) = 0, so there is a (bounded) neighbourhood V ofθ such that ξ(θ) < b for all θ ∈ V, for any choice of hyperparameter b ∈. In particular p 2 (θ) = ξ(θ) 2 by definition of the second criterion. We want to show that p(θ) = p 2 (θ) nearθ, or equivalently DISPLAYFORM0 in some neighbourhood U ⊆ V ofθ, for any choice of hyperparameter a ∈. Now by boundedness of V and continuity of χ, there exists c > 0 such that −α χ (θ) = α 2 χ (θ) < c for all θ ∈ V and bounded α. It follows by Cauchy-Schwartz that DISPLAYFORM1 in V, for some d > 0 and α sufficiently small, by boundedness of V and continuity of H o. Finally there is a sub-neighbourhood U ⊂ V such that ξ(θ) < ad/c for all θ ∈ U, so that ad ξ /c > ξ(θ) 2 and hence DISPLAYFORM2 2 for all θ ∈ U, as required. Theorem D.8. SOS converges locally to stable fixed points for α > 0 sufficiently small. Proof. Though the criterion for p is dual, we will only use the second part. More precisely, DISPLAYFORM3 The aim is to show that ifθ is an SFP then ∇ξ p (θ) is positive stable for small α, using Ostrowski to conclude as usual. The first problem we face is that ∇ξ p does not exist everywhere, since p(θ) is not a continuous function. However we know by Lemma D.7 that p = ξ 2 in a neighbourhood U ofθ, so ξ p is continuously differentiable in U. Moreover, p(θ) = ξ(θ) 2 = 0 with gradient DISPLAYFORM4 by definition of fixed points. It follows that DISPLAYFORM5 which is identical to LookAhead. This is positive stable for all 0 < α <, andθ is a fixed point of the iteration since DISPLAYFORM6 We conclude by Corollary D.3 that SOS converges locally to SFP for any a, b ∈ and α sufficiently small. Lemma D.9. Let g: R + × Y → Z continuous with Y compact and Z ⊆ R. Assume that for any u ∈ Y there is u > 0 such that g(α, u) > 0 for all 0 < α < u. Then there exists > 0 such that g(α, u) > 0 for all 0 < α < and u ∈ Y.Proof. For any u ∈ Y there is u > 0 such that DISPLAYFORM7 We would like to extend this uniformly in u, namely prove that DISPLAYFORM8 is open by continuity of g, so each (0, u) × {u} has a neighbourhood X u contained in g −1 (0, ∞). Open sets in a product topology are unions of open products, so DISPLAYFORM9 In particular (0, u) ⊆ x U x and at least one V x contains u, so we can take the open neighbourhood to be DISPLAYFORM10 for some neighbourhood V u of u. In particular Y ⊆ u∈Y V u, and by compactness there is a finite DISPLAYFORM11 Lemma E.1. Let a k and b k be sequences of real numbers, and define DISPLAYFORM0 for all k ≥ M, k ≥ N. Expanding the absolute value, this implies DISPLAYFORM1 which implies the contradiction L < L + δ.Proposition E.2. If SOS converges toθ and α > 0 is small thenθ is a fixed point of the game. Proof. The iterative procedure is given by DISPLAYFORM2 If θ k →θ as k → ∞ then taking limits on both sides of the iteration yields DISPLAYFORM3 and so lim k ξ p (θ k) = 0, omitting k → ∞ for convenience. It follows by continuity that DISPLAYFORM4 noting that p(θ) is not a globally continuous function. Assume for contradiction that ξ 0 (θ) = 0. There are two cases to distinguish for clarity.(i) First assume −α χ, ξ 0 (θ) ≥ 0. Note that lim k p(θ k) ≥ 0 since p(θ) ≥ 0 for all θ, and so DISPLAYFORM5 In both cases a contradiction is obtained, hence ξ 0 (θ) = 0 = (I − αH o)ξ(θ). Now note that (I − αH o)(θ) is singular iff H o (θ) has an eigenvalue 1/α, which is impossible for α sufficiently small. Hence (I − αH o)ξ(θ) = 0 implies ξ(θ) = 0, as required. Now assume that θ is initialised randomly (or with arbitrarily small noise around a point), as is standard in ML. We prove that SOS locally avoids strict saddles using the Stable Manifold Theorem, inspired from BID13. Theorem E.3 (Stable Manifold Theorem). Letx be a fixed point for the C 1 local diffeomorphism F: U → R d, where U is a neighbourhood ofx in R d. Let E s ⊕ E u be the generalised eigenspaces of ∇F (x) corresponding to eigenvalues with |λ| ≤ 1 and |λ| > 1 respectively. Then there exists a local stable center manifold W with tangent space E s atx and a neighbourhood B ofx such that DISPLAYFORM6 In particular, if ∇F (x) has at least one eigenvalue |λ| > 1 then E u has dimension at least 1. Since W has tangent space E s atx, with codimension at least one, it follows that W has measure zero in R d. This is central in proving that the set of initial points in a neighbourhood which converge through SOS to a given strict saddleθ has measure zero. Theorem E.4. SOS locally avoids strict saddles almost surely, for α > 0 sufficiently small. Proof. Letθ a strict saddle and recall that SOS is given by DISPLAYFORM7 Recall by Lemma D.7 that p(θ) = ξ(θ) 2 for all θ in a neighbourhood U ofθ. Restricting F to U, all terms involved are continuously differentiable and DISPLAYFORM8 by assumption that ξ(θ) = 0. Since all terms except I are of order at least α, ∇F (θ) is invertible for all α sufficiently small. By the inverse function theorem, there exists a neighbourhood V ofθ such that F is has a continuously differentiable inverse on V. Hence F restricted to U ∩ V is a C 1 diffeomorphism with fixed pointθ. By definition of strict saddles, H(θ) has a negative eigenvalue. It follows by continuity that (I − αH o)H(θ) also has a negative eigenvalue a + ib with a < 0 for α sufficiently small. Finally, DISPLAYFORM9 has an eigenvalue λ = 1 − αa − iαb with |λ| = 1 − 2αa + α 2 (a 2 + b 2) ≥ 1 − 2αa > 1.It follows that E s has codimension at least one, implying in turn that the local stable set W has measure zero. We can now prove that DISPLAYFORM10 has measure zero, or in other words, that local convergence toθ occurs with zero probability. Let B the neighbourhood guaranteed by the Stable Manifold Theorem, and take any θ ∈ Z. By definition of convergence there exists N ∈ N such that F N +n (θ) ∈ B for all n ≥ 0, so that DISPLAYFORM11 by the Stable Manifold Theorem. This implies that θ ∈ F −N (W), and finally θ ∈ ∪ n∈N F −n (W). Since θ was arbitrary, we obtain the inclusion Z ⊆ ∪ n∈N F −n (W). Now F −1 is C 1, hence locally Lipschitz and thus preserves sets of measure zero, so that F −n (W) has measure zero for each n. Countable unions of measure zero sets are still measure zero, so we conclude that Z also has measure zero. In other words, SOS converges toθ with zero probability upon random initialisation of θ in U. In the Gaussian mixture experiment, data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training, given in FIG5.The generator distribution and KL divergence are given at {2k, 4k, 6k, 8k} iterations for LA, LOLA and SGA in Figure 6. LOLA and SGA successfully spread mass across all Gaussians. LookAhead displays mode collapse and hopping in early stages, but begins to incorporate further mixtures near 8k iterations. We ran further iterations and discovered that LookAhead eventually spreads mass across all mixtures, though very slowly. Comparing with for NL/CO/SOS in the main text, we see that CO/SOS/LOLA/SGA are equally successful in qualitative terms. Note that SOS/CO are slightly superior with respect to KL divergence after 6-8k iterations, though LOLA is initially faster. This may be due only to random sampling. We also noticed experimentally that LOLA often moves away from the correct distribution after 8-10k iterations (not shown), while SOS stays stable in the long run. This may occur thanks to the two-part criterion encouraging convergence, while LOLA continually attempts to exploit opponent learning. Finally we plot ξ at all iterations up to 12k for SOS, LA and NL in Figure 7 (other algorithms are omitted for visibility). This gives further evidence of SOS converging quite rapidly to the correct distribution, while NL perpetually suffers from mode hopping and LA lags behind significantly.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Opponent shaping is a powerful approach to multi-agent learning but can prevent convergence; our SOS algorithm fixes this with strong guarantees in all differentiable games.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: The rate at which medical questions are asked online significantly exceeds the capacity of qualified people to answer them, leaving many questions unanswered or inadequately answered. Many of these questions are not unique, and reliable identification of similar questions would enable more efficient and effective question answering schema. While many research efforts have focused on the problem of general question similarity, these approaches do not generalize well to the medical domain, where medical expertise is often required to determine semantic similarity. In this paper, we show how a semi-supervised approach of pre-training a neural network on medical question-answer pairs is a particularly useful intermediate task for the ultimate goal of determining medical question similarity. While other pre-training tasks yield an accuracy below 78.7% on this task, our model achieves an accuracy of 82.6% with the same number of training examples, an accuracy of 80.0% with a much smaller training set, and an accuracy of 84.5% when the full corpus of medical question-answer data is used. With the ubiquity of the Internet and the emergence of medical question-answering websites such as ADAM (www.adam.com), WebMD (www.webmd.com), and HealthTap (www.healthtap. com), people are increasingly searching online for answers to their medical questions. However, the number of people asking medical questions online far exceeds the number of qualified experts -i.e doctors -answering them. One way to address this imbalance is to build a system that can automatically match unanswered questions with semantically similar answered questions, or mark them as priority if no similar answered questions exist. This approach uses doctor time more efficiently, reducing the number of unanswered questions and lowering the cost of providing online care. Many of the individuals seeking medical advice online are otherwise reluctant to seek medical help due to cost, convenience, or embarrassment. For these patients, an accurate online system is critical because it may be the only medical advice they receive. Of course, some medical problems require in-person care, and an online system must indicate that. Other patients use the internet in addition to in-person care either to determine when an appointment is needed or to follow up after visits when they have lingering questions. For this second group, if the answers they see online do not match those given to them by their doctors, they are less likely to follow the advice of their doctors , which can have serious consequences. Coming up with an accurate algorithm for finding similar medical questions, however, is difficult. Simple heuristics such as word-overlap are ineffective because Can a menstrual blood clot travel to your heart or lungs like other blood clots can? and Can clots from my period cause a stroke or embolism? are similar questions with low overlap, but Is candida retested after treatment and Is Chlamydia retested after treatment? are critically different and only one word apart. Machine learning is a good candidate for such complex tasks, but requires labeled training data. As no widely available data for this particular task exists, we generate and release our own dataset of medical question pairs such as the ones shown in Table 1. Given the recent success of pre-trained bi-directional transformer networks for natural language processing (NLP) outside the medical field (; ; Radford et al.; ;), most research efforts in medical NLP have tried to apply general. However, these models are not trained on medical information, and make errors that reflect this. In this work, we augment the features in these general language models using the depth of information that is stored within a medical question-answer pair to embed medical knowledge into the model. Our models pre-trained on this task outperform models pre-trained on out-of-domain question similarity with high statistical significance, and the show promise of generalizing to other domains as well. The task of question-answer matching was specifically chosen because it is closely related to that of question similarity; one component of whether or not two questions are semantically similar is whether or not the answer to one also answers the other. We show that the performance gains achieved by this particular task are not realized by other in-domain tasks, such as medical questioncategorization and medical answer completion. The main contributions of this paper are: • We release a dataset of medical question pairs generated and labeled by doctors that is based upon real, patient-asked questions • We prove that, particularly for medical NLP, domain matters: pre-training on a different task in the same domain outperforms pre-training on the same task in a different domain • We show that the task of question-answer matching embeds relevant medical information for question similarity that is not captured by other in-domain tasks 2 RELATED WORK 2.1 PRE-TRAINED NETWORKS FOR GENERAL LANGUAGE UNDERSTANDING NLP has undergone a transfer learning revolution in the past year, with several large pre-trained models earning state-of-the-art scores across many linguistic tasks. Two such models that we use in our own experiments are BERT and XLNet . These models have been trained on semi-supervised tasks such as predicting a word that has been masked out from a random position in a sentence, and predicting whether or not one sentence is likely to follow another. The corpus used to train BERT was exceptionally large (3.3 billion words), but all of the data came from BooksCorpus and Wikipedia. recently found that BERT generalizes better to other datasets drawn from Wikipedia than to tasks using other web snippets. This is consistent with our finding that pre-training domain makes a big difference. To address the need for pre-trained models in particular domains, some researchers have recently re-trained BERT on different text corpora such as scientific papers , doctor's medical notes and biomedical journal articles . However, retraining BERT on the masked-language and next-sentence prediction tasks for every new domain is unwieldy and time-consuming. We investigate whether the benefits of re-training on a new domain can also be realized by fine-tuning BERT on other in-domain tasks. see a boost with other tasks across less dramatic domain changes, where a different text corpus is used for the final task but not an entirely different technical vocabulary or domain. Previous work on question similarity has investigated the importance of in-domain word embeddings, but the methods of determining question similarity have varied widely. use a CNN that minimizes the mean squared error between two questions' vector representations. use an encoder and a question-pairs metric of cosine similarity. look at relative'best' match out of a set instead of absolute similarity, as we are interested in. Abacha & Demner-Fushman (2016; very clearly describe the utility of medical question similarity as we have framed it; rather than training a model to answer every conceivable medical question correctly, we can train a model to determine if any existing questions in an FAQ mean the same thing as a new question and, if so, use the existing answer for the new question. If there were a large corpus of labeled similar medical questions, training on that would likely produce the best . However, labeled training data is still one of the largest barriers to supervised learning, particularly in the medical field where it is expensive to get doctor time for hand-labeling data. Previous work has tried to overcome this using augmentation rules to generate similar question pairs automatically , but this leads to an overly simplistic dataset in which negative question-pairs contain no overlapping keywords and positive question-pairs follow similar lexical structures. Another technique for generating training data is weak supervision , but due to the nuances of determining medical similarity, generating labeling functions for this task is difficult. Another way to address a dearth of training data is to use transfer learning from a different but related task. This is the path we choose. Several large datasets exist that are relevant to our final task of medical question similarity. Quora Question Pairs (QQP) is a labeled corpus of 363,871 question pairs from Quora, an online question-answer forum . These question pairs cover a broad range of topics, most of which are not related to medicine. However, it is a well-known dataset containing labeled pairs of similar and dissimilar questions. HealthTap is a medical question-answering website in which patients can have their questions answered by doctors. We use a publicly available crawl (durakkerem, 2018) with 1.6 million medical questions. Each question has corresponding long and short answers, doctor meta-data, category labels, and lists of related topics. We reduce this dataset to match the size of QQP for direct performance comparisons, but also run one experiment leveraging the full corpus. WebMD is an online publisher of medical information including articles, videos, and frequently asked questions (FAQ). For a second medical question-answer dataset, we use a publicly available crawl over the FAQ of WebMD with 46,872 question-answer pairs. We decrease the size of QQP and HealthTap to match this number before making direct performance comparisons. Most of our pre-training tasks come from restructuring the HealthTap and WebMD data. Question Answer Pairs (QA) In order to correctly determine whether or not two questions are semantically similar, as is our ultimate goal, a network must be able to interpret the nuances of each question. Another task that requires such nuanced understanding is that of pairing questions with their correct answers. We isolate each true question-answer pair from the medical questionanswering websites and label these as positive examples. We then take each question and pair it with a random answer from the same main category or tag and label these as negative examples. Finally, we train a classifier to label question-answer pairs as either positive or negative. Answer Completion (AA) One task that has been known to generalize well is that of next-sentence prediction, which is one of two tasks used to train the BERT model. To mimic this task, we take each answer from HealthTap and split it into two parts: the first two sentences (start), and the remaining sentences (end). We then take each answer start and end that came from the same original question and label these pairs as positives. We also pair each answer start with a different end from the same main category and label these as negatives. This is therefore a binary classification task in which the model tries to predict whether an answer start is completed by the given answer end. Question Categorization (QC) We take the questions from HealthTap, pair them up with their main-category labels and call these positive examples. We then pair each question with a random other category and call this a negative example. There are 227 main categories represented, such as abdominal pain, acid reflux, acne, adhd, alcohol, etc. The model is trained to classify category matches and mismatches, rather than predict to which of the classes each example belongs. A small number of questions from HealthTap are used in the question pairs dataset and thus withheld from the above tasks to reduce bias. There is no existing dataset that we know of for medical question similarity. Therefore, one contribution of this paper is that we have generated such a dataset and are releasing it. Although our task is related to that of recognizing question entailment (RQE), for which there is a small dataset available , it is different in two key ways. First, our metric of similarity is symmetric, but in question entailment, it is possible to have asymmetric entailment, if one question is more specific than the other. Second, our questions are all patient-asked, which means they use less technical language, include more misspellings, and span a different range of topics than doctor-asked questions. Because of these differences we decide to generate a dataset that is specific to our needs. We have doctors hand-generate 3,000 medical question pairs. We explicitly choose doctors for this task because determining whether or not two medical questions are the same requires medical training that crowd-sourced workers rarely have. We present doctors with a list of patient-asked questions from HealthTap, and for each provided question, ask them to: 1. Rewrite the original question in a different way while maintaining the same intent. Restructure the syntax as much as possible and change medical details that would not impact your response (ex.'I'm a 22-y-o female' could become ' My 26 year old daughter'). 2. Come up with a related but dissimilar question for which the answer to the original question would be WRONG OR IRRELEVANT. Use similar key words. The first instruction generates a positive question pair (match) and the second generates a negative question pair (mismatch). With the above instructions, we intentionally frame the task such that positive question pairs can look very different by superficial metrics, and negative question pairs can conversely look very similar. This ensures that the task is not trivial. What specific exercises would help bursitis of the suprapatellar? Hey doc! My doctor diagnosed me with suprapatellar bursitis. Are there any exercises that I can do at home? Can I take any medication for pain due to suprapatellar bursitis? Unable to exercise.:(We anticipate that each doctor interprets these instructions slightly differently, so no doctor providing data in the train set generates any data in the test set. This should reduce bias. To obtain an oracle score, we have doctors hand-label question pairs that a different doctor generated. The accuracy of the second doctor with respect to the labels intended by the first is used as an oracle and is 87.6% in our test set of 836 question pairs. See Table 2 for example questions from our curated data. Our ultimate goal is to be able to determine whether two medical questions mean the same thing. Our hypothesis is that by taking an existing language model with complex word embeddings and training it on a large corpus for a similar medical task, we can embed medical knowledge into an otherwise generic language model. Our approach uses transfer learning from a bi-directional transformer network to get the most out of our small medical question pairs dataset. We start with the architecture and weights from BERT and perform a double finetune; we first finetune on an intermediate task and then we finetune on the final task of medical question pairs. We do this for four different intermediate tasks: quora question similarity (QQP), medical question answering (QA), medical answer completion (AA), and medical question classification (QC) (Figure 1). For a baseline we skip the intermediate finetune and directly train BERT on our small medical question-pairs dataset. For each intermediate task, we train the network for 5 epochs with 364 thousand training examples to ensure that differences in performance are not due to different dataset sizes. We then finetune each of these intermediate-task-models on a small number of labeled, medicalquestion pairs until convergence. A maximum sentence length of 200 tokens, learning rate of 2e-5, and batch size of 16 is used for all models. Each model is trained on two parallel NVIDIA Tesla V100 GPUs. All experiments are done with 5 different random train/validation splits to generate error bars representing one standard deviation in accuracy. We use accuracy of each model as our quantitative metric for comparison and a paired t-test to measure statistical significance. To compare against previous state-of-the-art (SOTA) models in the medical field, we also finetune the BioBERT , SciBERT , and ClinicalBERT models on our final task, three BERT models that have been finetuned once already on the original BERT tasks but with different text corpora. We also perform an ablation over pre-trained model architecture and reproduce our starting with the XLNet model instead of BERT. To get a better qualitative understanding of performance, we perform error analysis. We define a consistent error as one that is made by at least 4 of the 5 models trained on different train/validation splits. Similarly, we consider a model as getting an example consistently correct if it does so on at least 4 of the 5 models trained on different train/validation splits. By investigating the question pairs that a model-type gets consistently wrong, we can form hypotheses about why the model may have failed on that specific example. Then, by making small changes to the input until the models label those examples correctly, we can validate or disprove these hypotheses. Here we investigate whether domain of the training corpus matters more than task-similarity when choosing an intermediate training step for the medical question similarity task. Accuracy on the final task (medical question similarity) is our quantitative proxy for performance. We finetune BERT on the intermediate tasks of Quora question pairs (QQP) and HealthTap question answer pairs (QA) before finetuning on the final task to compare performance. We find that the QA model performs better than the QQP model by 2.4% to 4.5%, depending on size of the final training set (Figure 2). Conducting a paired t-test over the 5 data splits used for each experiment, the p-value is always less than 0.0006, so this difference is very statistically significant. We thus see with high confidence that models trained on a related in-domain task (medical question-answering) outperform models trained on the same question-similarity task but an out-of-domain corpus (quora question pairs). Furthermore, when the full corpus of questionanswer pairs from HealthTap is used, the performance climbs all the way to 84.5% ±0.7%. Results hold across models The same trends hold when the BERT base model is replaced with XLNet, with a p-value of 0.0001 (Table 3). Results hold across datasets We repeat our experiments with a question-answer dataset from WebMD and restrict the HealthTap and QQP dataset sizes for fair comparison. We find that the QA model again outperforms the QQP model by a statistically significant margin (p-value 0.049) and that the WebMD model even outperforms the HealthTap model with the same amount of data (Table 3). Our findings therefore hold across multiple in-domain datasets. We investigate further the extent to which task matters for an in-domain corpus in two different ways. We start by using the same HealthTap data and forming different tasks from the questions therein, and then we compare our models against intermediate models trained by other researchers. To test the extent to which any in-domain task would boost the performance of an out-of-domain model, we design two additional tasks using the HealthTap data: answer completion (AA) and question categorization (QC). As before, we use accuracy on the final question-similarity task as our proxy for performance and keep the test set constant across all models. We follow the same protocol as above, finetuning BERT on the intermediate task before finetuning further on the final task. We find that both of these tasks actually perform worse than the baseline BERT model, making the word embeddings less useful for understanding the subtler differences between two questions (Figure 2). We conclude that, while domain does matter a lot, many tasks are not well-suited to encoding the proper domain information from the in-domain corpus. Comparison to Medical SOTA Models To benchmark ourselves against existing medical models, we compare our finetuned models to BioBERT, SciBERT, and ClinicalBERT. Each of these models has finetuned the original BERT weights on a medically relevant corpus using the original BERT tasks. We take each of these off-the-shelf models and finetune them on our final task dataset as we do with our own intermediate-task models. Only BioBERT outperforms the original BERT model, and the differences in performance are not statistically significant. We hypothesize that this is because technical literature and doctor notes each have their own vocabularies that, while more medical in nature that Wikipedia articles, are still quite distinct from those of medical question-answer forums. From looking at the question pairs that our models get wrong, we can form hypotheses about why each example is mislabeled. We can then augment each question pair to add or remove one challenging aspect at a time and observe whether or not those changes in a different label. With this method, we can prove or disprove our hypotheses. The augmented questions are not added to our test set and do not contribute to our quantitative performance metrics; they are only created for the sake of probing and understanding the network. Consider the example in Table 4. In order to label this example correctly as it is written in row 1, the model has to understand the syntax of the question and know that 4'8" in this context represents poor growth. Changing the second question to what is written in row 2 prompts the QA model to label it correctly, indicating that one thing the QA model was misunderstanding was the question's word order. Additionally, changing the phrase I am 4'8" with I have not grown as shown in row 3 is enough to help the out-of-domain models label it correctly. So, while numerical reasoning was the difficult part of that question pair for the other models, the question answer model was actually able to identify 4'8" as a short height. This supports the claim that pre-training on the medical task of Misspellings, capitalization We find that differences in spelling and capitalization do not cause a significant number of errors in any model, although they are present in many questions. To understand the broader applicability of our findings, we apply our approach to a non-medical domain: the AskUbuntu question-answer pairs from. As before, we avoid making the pre-training task artificially easy by creating negatives from related questions. This time, since there are no category labels, we index all of the data with Elasticsearch 1. For the question similarity task, the authors have released a candidate set of pairs that were human labeled as similar or dissimilar. Without any pre-training (baseline), we observe an accuracy of 65.3% ± 1.2% on the question similarity task. Pre-training on QQP leads to a significant reduction in accuracy to 62.3% ± 2.1% indicating that an out-of-domain pretraining task can actually hurt performance. When the QA task is used for intermediate pre-training, the improve to 66.6% ± 0.9%. While this improvement may not be statistically significant, it is consistent with the main premise of our work that related tasks in the same domain can help performance. We believe that the low accuracy on this task, as well as the small inter-model performance gains, may be due to the exceptionally long question lengths, some of which are truncated by the models during tokenization. In the future, we would explore ways to reduce the length of these questions before feeding them into the model. In this work, we release a medical question-pairs dataset and show that the semi-supervised approach of pre-training on in-domain question-answer matching (QA) is particularly useful for the difficult task of duplicate question recognition. Although the QA model outperforms the out-of-domain same-task QQP model, there are a few examples where the QQP model seems to have learned information that is missing from the QA model (see Appendix A). In the future, we can further explore whether these two models learned independently useful information from their pre-training tasks. If they did, then we hope to be able to combine these features into one model with multitask learning. An additional benefit of the error analysis is that we have a better understanding of the types of mistakes that even our best model is making. It is therefore now easier to use weak supervision and augmentation rules to supplement our datasets to increase the number of training examples in those difficult regions of the data. With both of these changes, we expect to be able to bump up accuracy on this task by several more percentage points.

Represent the short summary for finding the corresponding scientific article (long)

We show that question-answer matching is a particularly good pre-training task for question-similarity and release a dataset for medical question similarity

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.