abstract: We compute the electromagnetic radiative corrections to all leading annihilation processes which may occur in the Galactic dark matter halo, for dark matter in the framework of supersymmetric extensions of the Standard Model (MSSM and mSUGRA), and present the results of scans over the parameter space that is consistent with present observational bounds on the dark matter density of the Universe. Although these processes have previously been considered in some special cases by various authors, our new general analysis shows novel interesting results with large corrections that may be of importance, e.g., for searches at the soon to be launched GLAST gamma-ray space telescope. In particular, it is pointed out that regions of parameter space where there is a near degeneracy between the dark matter neutralino and the tau sleptons, radiative corrections may boost the gamma-ray yield by up to three or four orders of magnitude, even for neutralino masses considerably below the TeV scale, and will enhance the very characteristic signature of dark matter annihilations, namely a sharp step at the mass of the dark matter particle. Since this is a particularly interesting region for more constrained mSUGRA models of supersymmetry, we use an extensive scan over this parameter space to verify the significance of our findings. We also re-visit the direct annihilation of neutralinos into photons and point out that, for a considerable part of the parameter space, internal bremsstrahlung is more important for indirect dark matter searches than line signals.
author: Torsten Bringmann; Lars Bergström; Joakim Edsjö
date: January 25, 2008
title: New Gamma-Ray Contributions to Supersymmetric Dark Matter Annihilation

# Introduction

During the last few years a strong consensus has emerged about the existence of a sizeable dark matter contribution to the total cosmological energy density. The identification of experimental signatures that eventually may determine the nature of the cosmological dark matter is thus becoming ever more important. The present estimates give the fraction of the critical density of cold dark matter particles as $\Omega_{CDM}h^2\sim 0.105 \pm 0.013$, where the Hubble parameter (scaled in units of 100 km/s Mpc$^{-1}$) is $h\sim 0.70\pm 0.02$. Also, on the scales of galaxies and smaller, a number of methods including measurements of rotation curves as well as gravitational lensing agree well with the predictions from N-body calculations of gravitational clustering in cold dark matter cosmologies (see e.g. ).

The methods of detection of dark matter (for reviews, see ) can be divided into *accelerator* production and detection of missing energy (especially at the LHC at CERN, which will start operating some time in 2008), *direct detection* (of dark matter particles impinging on a terrestrial detector, with recent impressive upper limits reported by ), or *indirect detection* of particles generated by the annihilation of dark matter particles in the Galactic halo or in the Sun/Earth. All these methods are indeed complementary – it is probable that a signal from more than one type of experiment will be needed to fully identify the particle making up the dark matter. The field is just entering very interesting times, with the LHC soon starting and new detectors of liquid noble gases being developed for direct detection. For indirect detection, the satellite PAMELA was launched a year ago and will soon reveal its first sets of data for positron and antiproton yields in the cosmic rays . AMANDA at the South Pole that has searched for detection of neutrinos from the centre of the Earth or the Sun , will soon give way to the much larger detector IceCUBE , and for gamma-rays coming from annihilations of dark matter particles in the halo the space satellite GLAST , to be launched in 2008, will open up a new window to the high-energy universe, for energies from below a GeV to about 300 GeV.

One problem with all these discovery methods is that the signal searched for may be quite weak, with much larger backgrounds in many cases. For indirect detection through gamma-rays, the situation may in principle be better, due to (i) the direct propagation from the region of production, without significant absorption or scattering; (ii) the dependence of the annihilation rate on the square of the dark matter density which may give "hot spots" near density concentrations as those predicted by N-body simulations; (iii) possible characteristic features like gamma-ray lines or steps, given by the fact that no more energy than $m_{\chi}$ per particle can be released in the annihilation of two non-relativistic dark matter particles (we denote the dark matter particle by $\chi$).

As an example, it was recently shown that in models of an extended Higgs sector, the line signal from the two-body final states $\gamma\gamma$ and $Z\gamma$ could give a spectacular signature in the gamma-ray spectrum between 40 and 80 GeV. On the other hand, in models of universal extra dimensions (UED) or in the theoretically perhaps most favoured, supersymmetric, models of dark matter the line feature is in general not very prominent, except in some particular regions of the large parameter space. However, it was early realised that there could be other important spectral features , and recently it has been shown that internal bremsstrahlung (IB) from produced charged particles in the annihilations could yield a detectable "bump" near the highest energy for heavy gauginos or Higgsinos annihilating into $W$ boson pairs, such as expected in split supersymmetry models . In , it was furthermore pointed out that IB often can be estimated by simple, universal formulas and often gives rise to a very prominent step in the spectrum at photon energies of $E_\gamma=m_\chi$ (such as in UED models ).

Encouraged by these partial results, we have performed a detailed analysis of the importance of IB in the minimal supersymmetric extension to the standard model (MSSM). We have therefore calculated the IB contributions for all two-particle charged final states from general neutralino annihilations. Besides confirming the mentioned partial results for the universal radiative corrections, in particular those relating to soft and collinear bremsstrahlung, we also point out interesting cases of model-dependent "virtual" bremsstrahlung (i.e. photons emitted from charged virtual particles), see Fig. 1. We confirm the suspicion expressed already in that this type of emission may circumvent the chiral suppression, i.e., the annihilation rate being proportional to $m_f^2$ for annihilation into a fermion pair from an $S$-wave initial state, as is the case in lowest order for non-relativistic dark matter Majorana particles in the Galactic halo (see also ). Since this enhancement mechanism is most prominent in cases where the neutralino is close to degenerate with charged sleptons, it is of special importance in the so-called stau coannihilation region in models of minimal supergravity (mSUGRA, as implemented in ). We therefore run through an extensive scan over these models (based on ) and find, indeed, remarkable cases of enhancement of the gamma-ray rate in the stau coannihilation region, near the maximal possible photon energy $E_\gamma=m_\chi$.

Let us stress that the radiative corrections to the main annihilation channels, here computed systematically for the first time, may turn out to be of utmost importance when fitting gamma-ray data, e.g. from GLAST, to supersymmetric dark matter templates. Over much of the parameter space we have scanned, these corrections give a large factor of enhancement over the commonly adopted estimates, especially at the observationally most interesting, highest energies. More importantly, they add a feature, the very sharp step at the dark matter mass, that would distinguish this signal from all other astrophysical background (or foreground) processes.

# Supersymmetric extensions to the standard model

Although our calculated electromagnetic radiative corrections will be applicable to the annihilation of any Majorana dark matter WIMP (weakly interacting massive particle), we will present results for the arguably most plausible dark matter candidate; the lightest neutralino in the MSSM, which is a linear combination of the superpartners of the gauge and Higgs fields: $$\chi\equiv\tilde\chi^0_1= N_{11}\tilde B+N_{12}\tilde W^3 +N_{13}\tilde H_1^0+N_{14}\tilde H_2^0\,.$$ We perform all numerical calculations using the DarkSUSY code (see for our sign conventions and other details).

The parameter $\mu$ is as usual the Higgsino mass parameter, $\tan\beta$ is the ratio of vacuum expectation values of the two Higgs doublets, $M_1$, $M_2$ and $M_3$ are the gaugino mass parameters, $m_A$ is the neutral pseudoscalar Higgs mass. We are also using parameters $m_0$, $A_t$ and $A_b$, defined through the simplifying ansatz: ${\bf M}_Q = {\bf M}_U = {\bf M}_D = {\bf M}_E = {\bf M}_L = m_0{\bf 1}$, ${\bf A}_U = {\rm diag}(0,0,A_t)$, ${\bf A}_D = {\rm diag}(0,0,A_b)$, ${\bf A}_E = {\bf 0}$. Here ${\bf A}$ are soft trilinear couplings and ${\bf M}$ soft sfermion masses which in general are $3\times3$ matrices in generation space, but are thus simplified through our ansatz and encoded in $m_0$, $A_t$ and$A_b$. We do not allow for CP-violating phases other than the CKM phase of the standard model. As a natural further simplification, the grand unification condition for the gauge couplings, leading to $M_1={5\over 3}\tan^2\theta_wM_2\approx {1\over 2}M_2$ is used. For the MSSM scans, we use FeynHiggsFast for the Higgs boson masses and decay widths. For each model, we will denote with $m_\chi$ the mass of the lightest neutralino and with $Z_g\equiv\left|N_{11}\right|^2+\left|N_{12}\right|^2$ the gaugino fraction.

As a more restricted, but in some sense more natural, set of parameters we use those that stem from demanding that the electroweak symmetry be spontaneously broken by electroweak radiative effects such as appears in minimal supergravity (mSUGRA) models . Also here we use the implementation in DarkSUSY , which relies on the public code Isajet for the solution of the renormalization group equations (RGE) and for the mass spectra. In these models, parameters are given at the grand unification scale, and are then (using the RGEs) let to run down to the electroweak scale. The range of models with correct symmetry breaking are usually displayed in the $m_0$-$m_{1/2}$ plane, where $m_0$ and $m_{1/2}$ are the universal scalar and gaugino masses, respectively, at the grand unification scale. The additional parameters of mSUGRA models are $\tan \beta$ (as for the MSSM), $A_0$ (which is the common trilinear term at the grand unified scale) and the sign of $\mu$ ($|\mu|$ is determined from the other parameters).

Let us briefly mention those regions in the $m_0$-$m_{1/2}$ plane that are important from a cosmological point of view, as they correspond to models with a relic density in accordance with the WMAP value: The *bulk region* which has low $m_0$ and $m_{1/2}$; the *funnel region* $m_A\approx 2m_\chi$, where annihilations in the early universe are enhanced by the presence of the near-resonant pseudoscalar Higgs boson; the hyperbolic branch or *focus point region* where $m_0 \gg m_{1/2}$; the *stau coannihilation region* where $m_{\chi}\approx m_{\tilde\tau}$; and finally the *stop coannihilation region* (arising when $A_0 \ne 0$) where $m_\chi \approx m_{\tilde{t}}$. The stau coannihilation region has recently been noticed to have favourable properties for indirect detection rates in antiprotons and gamma-rays . In this paper we will show that, in addition, there is a great enhancement of the high energy gamma-ray signature in this region.

# Internal Bremsstrahlung from WIMP annihilations

## The general case

Whenever WIMPs annihilate into pairs of charged particles $X\bar X$, this process will with a finite probability automatically be accompanied by internal bremsstrahlung (IB), i.e. the emission of an additional photon in the final state (note that in contrast to ordinary, or external, bremsstrahlung no external electromagnetic field is required for the emission of the photon). As visualized in Fig. , one may distinguish between photons directly radiated from the external legs (*final state radiation*, FSR) and photons radiated from virtual charged particles (which we will refer to as *virtual internal bremsstrahlung*, VIB). So, to be more specific, *the IB photons will be the total contribution from both FSR and VIB photons*.

If the charged final states are relativistic, FSR diagrams are always dominated by photons emitted *collinearly* with $X$ or $\bar X$. This is a purely kinematical effect and related to the fact that the propagator of the corresponding outgoing particle, $$D(p)\propto\left((k+p)^2-m_X^2\right)^{-1},$$ diverges in this situation. Here, $k$ and $p$ denote the momenta of the photon and the outgoing particle, respectively. The resulting photon spectrum turns out to be of a universal form, almost independent of the underlying particle physics model . Defining the *photon multiplicity* as $$\frac{dN_{X\bar X}}{dx}\equiv \frac{1}{\sigma_{\chi\chi\rightarrow X\bar X}}\frac{d\sigma_{\chi\chi\rightarrow X\bar X\gamma}}{dx}\,,$$ where $x\equiv2E_\gamma/\sqrt{s}=E_\gamma/m_\chi$ and $s$ is the center-of-mass energy, it is given by : $$\label{fsr}
  \frac{dN_{X\bar X}}{dx}\approx\frac{\alpha Q_X^2}{\pi}\mathcal{F}_X(x)\log \left(\frac{s(1-x)}{m_X^2}\right)\,.$$ Here, $Q_X$ and $m_X$ are the electric charge and mass of $X$; the splitting function $\mathcal{F}(x)$ depends only on the spin of the final state particles and takes the form $$\mathcal{F}_\mathrm{fermion}(x)=\frac{1+(1-x)^2}{x}$$ for fermions and $$\mathcal{F}_\mathrm{boson}(x)=\frac{1-x}{x}$$ for bosons. Due to the logarithmic enhancement that becomes apparent in Eq. (), FSR photons are often the main source for IB (note that very near the kinematical endpoint, $x\sim1-m_X^2/s$, it is not sufficient anymore to only keep leading logarithms and one can thus no longer expect Eq. () to be a good approximation for the actual spectrum). A prominent example where FSR in this universal form not only dominates IB but in fact the total gamma-ray spectrum from WIMP annihilations, is the case of Kaluza-Klein dark matter .

In general, one can single out two situations where photons emitted from virtual charged particles may give an even more important contribution to the total IB spectrum than FSR: i) the three-body final state $X\bar X\gamma$ satisfies a symmetry of the initial state that cannot be satisfied by the two-body final state $X\bar X$ or ii) $X$ is a boson and the annihilation into $X\bar X$ is dominated by $t$-channel diagrams. To understand that the first case only leads to an enhancement of VIB, and not of FSR, we recall that the latter is dominated by collinear photons, i.e. the (virtual) final state particles are almost on mass-shell; the two- and three-body final states are thus bound to the same symmetry constraints. The enhancement of the annihilation rate in the second case follows from a closer inspection of the $t$-channel propagator. For non-relativistic WIMPs, it takes the form $$\begin{aligned}
  D_t(p)&\propto&\left((l-p)^2-m_{\tilde X}^2\right)^{-1}\nonumber\\
   &\approx&\left(m_\chi^2-m_{\widetilde X}^2+m_X^2+2m_\chi E_X\right)^{-1}\,,
\end{aligned}$$ where $l$ is the momentum of one of the ingoing WIMPs and $\widetilde X$ denotes the particle that is exchanged in the $t$-channel. If $\chi$ and $\widetilde X$ are almost degenerate in mass, one thus finds an enhancement for small $E_X$ which – for kinematical reasons – corresponds to large photon energies $E_\gamma$. Note that this enhancement is less efficient for fermions since the infrared external spinor leg would result in a further suppression factor. In both cases, and in contrast to the situation for FSR, the resulting spectrum is highly model-dependent. Prominent examples where VIB dominates over FSR as a consequence of these two special situations are neutralino annihilations into leptons and charged gauge bosons , respectively, to which we will shortly return.

## The neutralino case

The relevant final states of neutralino annihilations are $W^+W^-$, $W^\pm H^\mp$, $H^+H^-$ and $f\bar f$; analytical expressions for the corresponding rates can be found, e.g., in . The inclusion of an additional photon in the final state is straight-forward, though tedious. Here, we are interested in the annihilation of neutralinos today, so we work in the limit of vanishing neutralino velocity – which greatly simplifies the calculation and the form of the analytical expressions. Our analytical results for arbitrary neutralino compositions agree, in the corresponding limits, with the special situations considered earlier, i.e. pure Higgsino or Wino annihilation into $W^+W^-$ and photino annihilation into light leptons . Let us stress that, although we state in the following simplified expressions for the photon multiplicities in some special cases, we always use the full expressions in our numerical calculations.

Let us first note that for neutralino annihilations, in contrast to the situation for Kaluza-Klein dark matter, we cannot in general expect very large contributions from FSR (from external legs). This is because the lightest charged final states, for which the logarithmic enhancement shown in Eq. () would be most effective, are fermionic – but the annihilation rate of neutralinos into light fermions is strongly suppressed by a factor $m_f^2/m_\chi^2$ due to the helicity properties of a highly non-relativistic pair of Majorana fermions . On the other hand, as we will demonstrate now, all the possible final states for neutralino annihilations have the potential of showing considerable VIB contributions.

For large neutralino masses $m_\chi\gg m_W$ and charginos almost degenerate with the neutralino, e.g., $W^+W^-$ and $W^\pm H^\mp$ final states fall into the second of the categories discussed in the previous subsection. If the neutralino is an almost pure Higgsino (or Wino), we then find that the photon multiplicity for $W^+W^-$ final states can be well approximated by (see also Ref.  for a similar result): $$\begin{aligned}
 \frac{\mathrm{d}N_{W^+W^-}} {\mathrm{d}x} &\approx& \frac{\alpha_\mathrm{em}}{\pi}\frac{4(1-x+x^2)^2}{(1-x+\epsilon/2)x} \\
  &&\times\left[\log\left(2\frac{1-x+\epsilon/2}{\epsilon}\right)-1/2+x-x^3\right]\,,\nonumber
\end{aligned}$$ where $\epsilon \equiv m_W/m_\chi$. As demonstrated in the next section, the most significant IB contributions from final states with charged gauge bosons to the total annihilation spectrum are, in fact, often of this form. The full analytical expressions for ${\mathrm{d}N_{W^+W^-}}/{\mathrm{d}x}$ and ${\mathrm{d}N_{W^\pm H^\mp}}/{\mathrm{d}x}$ are rather lengthy and we will therefore not explicitly state them here. Let us stress again, however, that we of course do use them in our actual calculations.

As mentioned before, the annihilation into light fermions is helicity suppressed; for large photon energies, however, fermion final states containing an additional photon, $f\bar f\gamma$, are not subject to such a suppression. While our full analytical expressions are again rather lengthy, they simplify considerably in the limit of $m_f\rightarrow0$. In this case, and assuming that both sfermions have the same mass, the photon multiplicity is given by

where $\mu\equiv m_{{\tilde f}_R}^2/m_\chi^2=m_{{\tilde f}_L}^2/m_\chi^2$ and $\tilde g_RP_L$ ($\tilde g_LP_R$) is the coupling between neutralino, fermion and right-handed (left-handed) sfermion. This confirms the result found in for photino annihilation (while states a result that is an overall factor of $2$ larger). Note the large enhancement factor $m_\chi^2/m_f^2$ due to the lifted helicity suppression (from ${\langle\sigma v\rangle}_{\chi\chi\rightarrow f\bar f}\propto m_f^2m\chi^{-4}$), and another large enhancement that appears at high photon energies for sfermions degenerate with the neutralino.

Charged Higgs pairs $H^+H^-$, finally, provide yet another interesting example of the second category discussed in the previous subsection: In the limit $v\rightarrow0$, this final state is not allowed because of $CP$ conservation. The annihilation into $H^+H^-\gamma$, on the other hand, *is* possible. However, since charged Higgs bosons in most models have considerably larger masses than gauge bosons the enhancement mechanism described in the previous subsection is not as efficient as in the latter case. These final states are thus expected to be of less importance in our context.

In the last part of this section, let us briefly describe how we implemented IB from the various possible final states of neutralino annihilations in DarkSUSY. The total gamma-ray spectrum is given by $$\frac{dN^{\gamma,\mathrm{tot}}}{dx}=\sum_f B_f\left(\frac{dN_f^{\gamma,\mathrm{sec}}}{dx}+\frac{dN_f^{\gamma,\mathrm{IB}}}{dx} + \frac{dN_f^{\gamma,\mathrm{line}}}{dx}\right)\,,$$ where $B_f$ denotes the branching ratio into the annihilation channel $f$. The last term in the above equation gives the contribution from the direct annihilation into photons, $\gamma\gamma$ or $Z\gamma$, which result in a sharp line feature . The first term encodes the contribution from secondary photons, produced in the further decay and fragmentation of the annihilation products, mainly through the decay of neutral pions. This "standard" part of the total gamma-ray yield from dark matter annihilations shows a feature-less spectrum with a rather soft cutoff at $E_\gamma=m_\chi$. In DarkSUSY, these contributions are included by using the Monte Carlo code PYTHIA  to simulate the decay of a hypothetical particle with mass $2m\chi$ and user-specified branching ratios $B_f$. In this way, also FSR associated to this decay is automatically included (the main contribution here comes from photons directly radiated off the external legs, but also photons radiated from other particles in the decay cascade are taken into account). On the other hand, IB from the decay of such a hypothetical particle cannot in general be expected to show the same characteristics as IB from the actual annihilation of two neutralinos. In particular, and as discussed in length at the beginning of this Section, we expect important VIB contributions in the latter case – while in the first case there are simply no virtual particles that could radiate photons. We therefore calculate analytically the IB associated to the decay (i.e. FSR from the final legs) and subtract it from $dN_f^{\gamma,\mathrm{sec}}/dx$ as obtained with PYTHIA; for $dN_f^{\gamma,\mathrm{IB}}/dx$, we then take the full IB contribution from the actual annihilation process as described before. Hence, this procedure leaves us with corrected PYTHIA results without FSR on the external legs and our analytical calculation of IB (including FSR and VIB) that we add to this. [^1]

```latex
\begin{table*}[t]{
    \begin{tabular}{|l||c|c|c|c|c||c|c|c|c|c|c|c|c|}
        \hline
        & $m_0$ & $m_{1/2}$ & $\tan\beta$ & $A_0$ & $\mathrm{sgn}$ & $m_\chi$ & $Z_g/$ & $\Omega h^2$ & $t$-channel & $\cal S$ & IB/ & IB/ \\
    & [GeV] & [GeV] & & [GeV] &$(\mu)$& [GeV] &$(1-Z_g)$& & & & sec.& lines\\
        \hline \hline
       $\!$BM1 & $3700$ & $3060$ & $5.65$ & $\!\!-1.39\!\cdot\!10^4\!$ & $-1$ & $1396$ & $3.0\!\cdot\!10^4$ & $0.082$ & $\tilde t(1406)$ & $8\!\cdot\!10^{-5}\!$ & $19.2$ & $4.5$ \\
       $\!$BM2 & $801$ & $1046$ & $30.2$ & $\!\!-3.04\!\cdot\!10^3\!$ & $-1$ & $446.9$ & $1611$ & $0.110$ & $\tilde\tau(447.5)$ & $0.044$ &  $10.6$ & $8.5$ \\
       $\!$BM3 & $107.5$ & $\!576.4\!\!$ & $3.90$ & $28.3$ & $+1$ & $233.3$ & $220$ & $0.084$ & $\tilde\tau(238.9)$ & $1.19$ & $\!2.3\!\cdot\!10^3\!$ & $5.0$  \\
       $\!$BM4 & $\!2.2\!\cdot\!10^4\!$ & $7792$ & $24.1$ & $17.7$ & $+1$ & $1926$ & $\!\!1.2\!\cdot\!10^{-4}\!\!\!\!$ & $0.11$ & $\tilde \chi^+_1(1996)\!$ & $0.012$ & $10.8$ & $2.1$  \\
        \hline
   \end{tabular}}
  \caption{Benchmark models that represent typical regions in the supersymmetric parameter space where IB becomes important. The ``$t$-channel'' entry indicates the main contributing $t$-channel diagram, with the corresponding sparticle and its mass (all masses are given in GeV). ${\cal S}\equiv N_\gamma\frac{\langle\sigma v\rangle}{10^{-29}\mathrm{cm}^3\mathrm{s}^{-1}}\left(\frac{m_\chi}{100\mathrm{GeV}}\right)^{-2}$ is the rescaled flux from IB alone and the last two columns give the ratio of the integrated flux, all above $0.6\,m\chi$, between the new IB contribution and secondary photons as well as the line signals. The main difference between BM2 and BM3 is that in the former neutralinos mainly annihilate into $\tau$ leptons, while in the latter mainly into $t$ quarks. Also, for  BM2 only the $\tau$ final states give an important contribution, while in the second case, even the other leptonic final states contribute considerably (due to near-degenerate slepton masses). For the BM4 model, the IB contribution from the $W^+W^-$ state dominates.
\label{benchmark}}
\end{table*}
```

Let us conclude this section by showing in Fig.  four typical examples of mSUGRA models with particularly pronounced IB features. From top to bottom, they show situations in which IB from $t\bar t$, $\tau^+\tau^-$, all lepton and $W^+W^-$ final states, respectively, dominates the total gamma-ray spectrum from DM annihilations.

# Numerical results

We are now ready to apply the results in the previous section to a set of supersymmetric MSSM and mSUGRA models. For MSSM, we use a set of scans of the 7 parameters mentioned in Sec. . These scans are fairly general and partly use the method of Markov Chain Monte Carlo models to focus on models that have the relic density in the WMAP preferred range. In total we have about 200 000 MSSM models that pass the WMAP relic density constraint and all accelerator constraints (checking experimental bounds on various masses and branching ratios, such as that of $b\to s\gamma$). For mSUGRA, we use the very large set of scans performed in Ref.  (also made with a Markov Chain Monte Carlo). This data set contains about 550 000 models that pass all the constraints.

Since the IB photons are most distinguished at higher energies (as seen in Fig. ), we will integrate the flux above $0.6m_\chi$ and compare the flux in this energy range from the flux obtained from secondary photons (arising mainly from $\pi^0$ decays in quark jets). We will also compare it with the monochromatic gamma ray lines that appear at loop-level.

In Fig.  we show our combined result for both the MSSM and mSUGRA models. In the left panel we show how IB compares with the regular secondary photons from quark jets. We plot this in the $Z_g/(1-Z_g)$ vs $m_\chi$ plane where gauginos (binos and winos) are at the top, Higgsinos are at the bottom and mixed neutralinos are in the middle. In the right panel, we show how IB photons compare with the monochromatic lines. It can be clearly seen that in large parts of this plane, IB photons are *the dominating component*, outnumbering both secondary and monochromatic photons. Hence, this effect is very important to include when searching for gamma ray signatures from dark matter.

In Fig.  we looked at the ratio of IB to other gamma ray contributions, let's now turn to the absolute fluxes. We can write the flux observed at earth as $$\begin{aligned}
  \label{eq:flux}
  &&\Phi_\gamma (E,\Delta \Omega,\, \psi)
    = \\
   &&\qquad
   9.35 \cdot 10^{-14}
  \, \frac{d{\cal S}}{dE}  \nonumber\times  \langle\,J\left(\psi \right)\,\rangle\,(\Delta\Omega)
  \;\;\rm{cm}^{-2}\;\rm{s}^{-1}\;\rm{sr}^{-1}\,.
\end{aligned}$$ where we have put all the particle physics into the quantity $\cal S$ and the astrophysics into $J$. $\cal S$ is given by $$\left(\frac{d{\cal S}}{dE}\right)_{\!{\rm IB}\,\gamma}
  \!\!\!\!=\!\!\!\!\!\!\!  \quad \left( \frac{100\,\rm{GeV}}{m_\chi}\right)^2 \sum_f \left( \frac{v\sigma_f}{10^{-29}\ {\rm cm}^3\;
   {\rm s}^{-1}}\right)\,\frac{dN_{f}^{\gamma, IB}}{dE}\,,$$ where the sum is over all final states $f$. The astrophysical part (that depends on the chosen halo profile) is given by $$\begin{aligned}
&&\langle\,J\left(\psi\right)\,\rangle\,(\Delta\Omega) =\\
&&\qquad\frac{1} {8.5\, \rm{kpc}}\,\frac{1} {\Delta\Omega}
\int_{\Delta\Omega}\!\!\!d\Omega'\, \mathop{\int_{line}}_{of\,sight}\!\!\!dL\,
\left(\frac{\rho(L,\,\psi')}
{0.3\,{\rm GeV}/{\rm cm}^3}\right)^2\,.\nonumber
\label{eq:jpsi}
\end{aligned}$$ For example, for a standard NFW halo profile $\langle J \rangle \simeq 21$ averaged over $\Delta \Omega = 1$ sr towards the galactic centre. This value can be orders of magnitude larger though both for steeper profiles and/or for more concentrated observations towards the galactic centre (i.e. smaller $\Delta \Omega$ – as, e.g., for Air Cherenkov Telescopes). The effect of specific halo profiles and a more detailed analysis of the absolute fluxes with respect to the astrophysical background will be left for future work . We will here focus on the particle physics factor $\cal S$.

In Fig.  we show the quantity $\cal S$, which is $d {\cal S}/dE$ integrated above $0.6\,m_\chi$. In the left panel, we show the yields ${\cal S}$ for the IB contribution, in the middle for monochromatic $\gamma \gamma$ and on the right for $Z \gamma$. In the regions where the IB contribution was the largest in Fig. , we typically have lower absolute yields. However, there are very pronounced regions, especially at small and intermediate masses, where the IB yields are very high even in absolute terms. We also note that, for neutralino masses in the TeV range, we expect a sizeable increase of the annihilation rate due to non-perturbative effects related to long-distance forces between the annihilating particles . These effects have not been taken into account here and would result in a considerable enhancement (by a similar factor) of the quantity $\cal S$ for both line signals *and* IB.

In Fig.  we focus on the mSUGRA case and show the contribution relative to the secondary yield of gamma rays for various final states separately. In the left panel, we show the IB yield from the $W^+W^-$ channel, in the middle from the $\tau^+ \tau^-$ channel and in the right from the $t \bar{t}$ channel. Large IB contributions for the $W^+ W^-$ channel occur when a chargino is almost degenerate with the neutralino, as is the case for the focus point region. Note that due to the grand unification condition, $M_1\approx \frac{1}{2}M_2$, a large gaugino fraction $Z_g$ always means that the neutralino is a *Bino*, with vanishing annihilation rates to $W^+W^-$ or $W^+W^-\gamma$ final states. The large yields from the $\ell^+ \ell^-$ and $t \bar{t}$ channels, on the other hand, occur when there is a strong degeneracy with the lightest $\tilde{\ell}$ and $\tilde{t}$ respectively. These latter cases occur in the phenomenologically important $\tilde{\tau}$ and $\tilde{t}$ coannihilation regions: in these regimes, coannihilations with $\tilde{\tau}$ and $\tilde{t}$, respectively are needed to push the relic density down into the WMAP preferred region. Hence, we have a strong mass degeneracy between $\tilde{\chi}$ and $\tilde{\tau}$/$\tilde{t}$ which forces the IB contribution to the gamma yields to be strong.

As for the other possible final states, we note that the corresponding IB contributions never exceed $10\%$ of the secondary photon flux; these channels are subdominant also for the MSSM models contained in our scan. In fact, from our discussion in the previous section, this is somewhat expected: Charged Higgs bosons, for example, are always heavier than charged gauged bosons, so multi-TeV neutralino masses would be needed for sufficiently large annihilation rates into $W^\pm H^\mp\gamma$ or $H^+H^-\gamma$ (recalling that the annihilation rate in these cases is enhanced for relativistic final states). IB from light quarks is suppressed by the mass difference between the neutralino and the corresponding squark (as compared to the small mass difference that can be achieved in the stop coannihilation region); down-type quarks, finally, receive a further suppression due to their smaller electric charge.

The main results of our paper may be more easily grasped by looking at the effect of IB on a small number of benchmarks models. Of course, for the mSUGRA case, it is known that the exact location in parameter space of such benchmarks depends very sensitively on details of the calculation (see e.g. ). We therefore define our own set in Table , which is very similar to that used by except that we also include one point in the focus point region (BM4). This set of benchmark models is calculated with ISAJET 7.69 together with DarkSUSY (see for details). Point BM1 is a model where $A_0$ has been chosen large and negative to make the stop almost degenerate with the neutralino. BM2 is a model where the stau is almost degenerate with the neutralino and in BM3 also the selectron and the smuon are degenerate with the neutralino. BM4, finally, is in the focus point region, i.e. where the lightest chargino is almost degenerate with the lightest neutralino. The main IB characteristics of these benchmark models are summarized in Table .

# Summary and Conclusions

As can be seen already from our benchmark points in Table , and in more detail from the scatter plots in Fig. , the internal bremsstrahlung effects computed in this work can be very significant, changing sometimes by more than an order of magnitude the lowest-order prediction for the high-energy gamma-ray signal from neutralino dark matter annihilation. Although some of these enhancements have been found before , this is the first time the first-order radiative corrections have been computed systematically, for all relevant final states in supersymmetric dark matter models. The resulting enhancements of the expected fluxes are surprisingly large over significant regions in the parameter space of the MSSM, including the more constrained mSUGRA models. Despite the fact that some large corrections apply to absolute rates that are too small to be of practical interest, Fig. 4 shows that the quantity ${\cal S}$, which is directly proportional to the expected signal in gamma-ray detection experiments, also is significant for the internal bremsstrahlung contribution in large regions of parameter space. For $m_\chi < 300$ GeV, for example, values of ${\cal S}_{IB}$ greater than 0.1 are generic, and for masses below 100 GeV, values of 1 or higher are common, which in very many cases is higher than the corresponding values for the line signals $\gamma\gamma$ and $Z\gamma$. One should also bear in mind that the sensitivity of Air Cherenkov Telescopes increases significantly with energy; detectional prospects for a $m_\chi\sim1~$TeV neutralino with ${\cal S}\sim0.01$, e.g., correspond very roughly to those for a $m_\chi\sim100~$GeV neutralino with ${\cal S}\sim0.5$ (see, e.g., ). In this light, the situation becomes very interesting even for TeV scale Higgsinos, where IB generically contributes more than 10 times as much as secondary photons.

We note that (as anticipated in ) helicity suppression and also CP selection rules of certain final states may be circumvented by emitting a photon; this is for example the origin of the very substantial enhancements of the signal obtained in the stau annihilation region in mSUGRA models. In this situation, the probability of emitting gamma rays vanishes at zero photon energy but increases rapidly at high energy (see Fig. b and c), which gives a photon "bump" at $E_\gamma\approx m_\chi$. We also note that in less constrained versions of the MSSM than considered here, we expect even more situations where large enhancements of the annihilation signal due to internal bremsstrahlung can be found. An example is heavy Wino dark matter , which becomes possible when relaxing the condition $M_1\approx \frac{1}{2}M_2$ (as realized, e.g., in anomaly mediated supersymmetry breaking scenarios ).

Of course, the line signals, in particular $\gamma\gamma$, have the virtue of being at the highest possible energy, so in order to make a more accurate comparison between these and the IB signal computed here, one would have to model also the expected spectral shape of possible astrophysical gamma-ray backgrounds and the energy resolution of the detector. This is left for future work . We note, however, that in general also the new contributions have a characteristic signature, (*cf.* Fig. 2) which can hardly be mimicked by any known astrophysical gamma-ray source. In fact, in some cases these spectra could even be used by future experiments to distinguish between different dark matter candidates (note that, e.g., the distinction between Kaluza-Klein dark matter and a neutralino in the focus point region like our BM4 point would be possible already with the energy resolution of present Air Cherenkov Telescopes ).

To conclude, we have shown that the commonly neglected first-order radiative corrections to neutralino dark matter annihilation should definitely be taken into account when predicting rates for gamma-ray telescopes. In particular, the soon to be launched GLAST space telescope will have an enhanced possibility over what has previously been assumed to detect radiation from supersymmetric dark matter annihilation. The routines needed to compute these new processes will be included in the next release of the DarkSUSY package .

We wish to thank Piero Ullio, Maxim Perelstein, Michael Gustafsson and Martin Eriksson for useful discussions. L.B. and J.E. acknowledge support from the Swedish Research Council (VR). We are grateful to E.A. Baltz for letting us use his extensive data base of mSUGRA models.

[^1]: We would like to stress that this prescription is fully consistent since both the original and the corrected IB versions are gauge-invariant separately. Strictly speaking, however, we have only corrected for photons originating directly from the external states and not for those radiated from particles that appear later in the decay cascade. On the other hand, one would of course expect that modifying the energy distribution of the charged particles corresponding to these external legs also affects the further decay cascade. Note, however, that the resulting change in the photon spectrum is a second order effect; more important, for kinematical reasons it does not affect photons at energies close to $m_\chi$ – which, as we shall see, are the most relevant. Finally, we observe that our subtraction procedure has only a minor effect on the photon spectrum obtained by PYTHIA and no practical consequences for the quantities we calculate in the next Section in order to quantify the importance of IB for neutralino annihilations.


abstract: Here we explore a new algorithmic framework for multi-agent reinforcement learning, called Malthusian reinforcement learning, which extends self-play to include fitness-linked population size dynamics that drive ongoing innovation. In Malthusian RL, increases in a subpopulation's average return drive subsequent increases in its size, just as Thomas Malthus argued in 1798 was the relationship between preindustrial income levels and population growth . Malthusian reinforcement learning harnesses the competitive pressures arising from growing and shrinking population size to drive agents to explore regions of state and policy spaces that they could not otherwise reach. Furthermore, in environments where there are potential gains from specialization and division of labor, we show that Malthusian reinforcement learning is better positioned to take advantage of such synergies than algorithms based on self-play.
author: Joel Z. Leibo; Julien Perolat; Edward Hughes; Steven Wheelwright; Adam H. Marblestone; Edgar Duéñez-Guzmán; Peter Sunehag; Iain Dunning; Thore Graepel
bibliography: main.bib
title: Malthusian Reinforcement Learning

# Introduction

Reinforcement learning algorithms have considerable difficulty avoiding local optima and continually exploring large state and policy spaces. This is known as the problem of exploration. In single-agent reinforcement learning, the main approach is to rely on intrinsic motivations, e.g., for individual curiosity , empowerment , or social influence .

However, critical adaptation events in human history are difficult to explain with intrinsic motivations. Consider the dispersal of homo sapiens out of Africa, where they first evolved between $200,000$ and $150,000$ years ago, throughout the globe, eventually occupying essentially all terrestrial climatic conditions and habitats by $14,500$ years before the present . This example is relevant to AI research because intelligence is often defined as an ability to adapt to a diverse set of environments . Essentially no other process on earth has led so quickly to so much adaptive diversity as did the expansion of human foragers throughout the globe[^1]. Human foraging communities were capable both of discovering water-finding strategies suitable for the arid Australian desert as well as how to hunt seals hiding beneath ice sheets and keep warm in the Arctic . From this perspective, the great dispersal of human foragers can be seen as some of the best evidence for an "existence proof" that intelligence, by this definition, is even possible. Yet there's no evidence that intrinsic motivations like curiosity played a role in it. Rather, considerable evidence points to a variety of extrinsic motivation mechanisms as the main drivers of human migration including climate change and demographic expansion .

One perspective on the problem of exploration is that the difficulty comes from the sparseness of extrinsic rewards. If extrinsic rewards are very sparse, then it is hard to estimate state-value functions and policy gradients since returns will have very high variance. Thus intrinsic motivation methods produce frequent intermediate (dense) rewards in hopes of bridging the long gaps between extrinsic rewards . Multi-agent reinforcement learning offers an alternative. Algorithms based on self-play like AlphaGo are aimed at an essentially single-agent objective, e.g., defeat a specific human grandmaster. Directly training by single-agent reinforcement learning to accomplish that objective is a lost cause. Since the untrained agent would never win a game, it would never get any reward signal to learn from. Self-play, on the other hand, provides an alternative incentive for agents to explore deeply through the strategy space. In two-player zero-sum, it escapes local optima by learning to exploit them, diminishing the returns available from such strategies, and thereby extrinsically motivating new exploration. Of course, life is not a zero-sum game. Nevertheless, real-life feats of exploration like the dispersal of homo sapiens out of Africa really were motivated in part by a pressure to out-compete rivals in a struggle for scarce resources that became increasingly difficult over time due to demographic expansion. Moreover, local competition between individuals is a universal feature of natural habitats, and underlies the evolution of dispersal . In this paper we investigate whether such population size dynamics can be exploited as an algorithmic mechanism in multi-agent reinforcement learning.

As an algorithmic mechanism, augmenting multi-agent reinforcement learning with population size dynamics appears to have the requisite property needed to evade local optima and traverse large state spaces. Strategies that work well at low densities do not necessarily translate well to high densities, but success at any density ensures density will increase further in the future. The rules of the game therefore naturally shift over time in a manner that depends on past outcomes. This ensures that species cannot remain too long in comfortable local optima. When resources are scarce, rising populations eventually dissipate gains from learning, forcing agents to innovate just to maintain existing reward levels [^2].

So far we've motivated introducing population dynamics to multi-agent reinforcement learning by appealing to a competitive struggle for existence against a well-matched foe, i.e., the same argument underlying the performance of self-play in two-player zero-sum games. However, there is more to life than competition. As an algorithmic mechanism to promote learning in general-sum multi-agent environments, population dynamics may also be more suitable than self-play-based approaches. In particular, we will consider whether this approach provides greater scope for adapting to synergies between specialists, making it easier to discover joint policies involving significant division of labor.

In this work we introduce a new algorithm for multi-agent reinforcement learning based on these principles of population dynamics. It is called Malthusian reinforcement learning because improvements in returns for any subpopulation translate directly into increases in the size of that subpopulation in subsequent episodes. Thus it may be evaluated on either the individual or the group level. In this work we are interested in two specific questions:

1.  Is Malthusian reinforcement learning better at avoiding becoming stuck in bad local optima in *individual* policy space than competing algorithms based on intrinsic motivation?

2.  Is it easier to evolve joint policies to implement heterogeneous mutualism behaviors with Malthusian reinforcement learning than with alternative approaches based on self-play?

# Model

## Characteristics of the Malthusian reinforcement learning framework

Malthusian reinforcement learning differs from standard multi-agent reinforcement paradigms in a number of ways.

1.  Malthusian reinforcement learning may be seen as an algorithm for "community coevolution". It produces a set of communities, called *islands* in our terminology. Each island has a set of agents implementing policies that should, if the training was successful, function well together.

2.  Each individual is a member of a *species*. All individuals of the same species share a policy neural network.

3.  The algorithm unfolds on two timescales corresponding to (A) the population dynamics (ecological) time, and (B) policy execution (behavioral) time.

4.  The population dynamics are linked to individual reinforcement learning returns. If individuals of a given species perform well on a particular island then their population will increase there in the future.

5.  During each episode all individuals of a given species generate experience to train a common neural network via v-trace . Experiences generated by individuals of a particular species are used only to update their own species neural network. After each episode the distribution over islands of each species is updated by a policy gradient-like update rule.

6.  *Conservation of compute:* Biologically realistic population dynamics all contain at least the possibility of exponential growth. In practice, they are limited by carrying capacities, i.e., by environment properties. Since Malthusian reinforcement learning is mainly intended for multi-agent machine learning applications rather than for ecological simulations we cannot rely on resource constraints in the environment to limit growth. Thus, to ensure it can be executed with bounded compute resources, the population dynamic works by maintaining probability distributions for each species over the set of all islands. The probability assigned to any given island may grow or shrink based on the individual returns achieved there, but it is always constrained to be a valid probability distribution. Compute remains bounded because a fixed number of samples are used to assign individuals to islands. The total population varies on any given island from episode to episode, but across the entire *archipelago*, the number of individuals is always constant.

## Archipelagos, islands, and species

#### Island and Archipelago

an *island* is a multi-agent environment where a variable number of agents can interact. An *archipelago* is a set $I$ of islands. Furthermore we will write $N_I = |I|$ the number of islands in an archipelago.

#### Species

A species $\Psi^l$ is a set of individuals sharing the same policy network parameterization. There are $L$ species indexed by $l$. Each species is composed of a policy network $\pi^l$ with parameters $\theta^l$ which encodes the behavior of each individual of the species. The distribution of agents of a given species $l$ over the islands is $\mu^l \in \Delta_{N_I}$ (where $\Delta_{N_I}$ is the set of distributions over islands. $\mu^l$ is defined as a softmax over weights $w^l$. The total number of individuals is $K$ and the number of individuals per species is $M = \frac{K}{L}$. We will denote each individual of a species $l$ by $k^l$.

The learning process unfolds over two timescales, a slow *ecological* scale which adapts the distribution of species over islands $\mu^l$ and a fast *behavioral* scale over which individuals execute their policies. Species adapt $\pi^l$ to behave in the presence of others at the level of the island. The ecological scale timesteps are indexed by $e$, and the behavioral scale timesteps are indexed by $t$. The ecological scale ticks at the level of single episodes for the behavioral scale.

## Population dynamics

The population dynamics govern how individuals of each species are assigned to the different islands over the ecological time scale. At a fixed ecological timestep $e$, individuals of each species are assigned to islands by sampling $M$ times from the distributions $\mu^l$. For each island, this yields an allocation $\Psi^l_{i,e}$, the set of individuals from species $l$ playing on island $i$ at ecological timestep $e$.

Each island has its own environment, in general the islands could have different environments from one another, though in this work we only consider the case where they are all the same.

Over the course of ecological time, the population evolves according to a gradient-based dynamic. At each timestep $e$, each individual $k^l$ of each species $l$ receives a fitness $\phi_{k^l, e}$, which is exactly its cumulative reward over the behavioral scale timesteps that have elapsed during one step of the ecological timescale. The per-island fitness for each species is then calculated as

$$\phi_{i, e}^l = \left(\sum \limits_{k^l \in \Psi^l_{i, e}} \phi_{k^l, e} \right) / |\Psi^l_{i, e}| \textrm{  and  } 0 \textrm{  if  } \Psi^l_{i, e} = \emptyset \, .$$

The distribution over islands for each species, $\mu^l(i) = e^{w^l_i} / \sum_j e^{w^l_j}$, is updated according to policy gradient with entropy regularization. Explicitly the distribution weights for species $l$ over all islands change according to a policy-gradient update $$w^l_{e+1} = w^l_e + \alpha \left[  \sum \limits_{i \in \{1, \dots, N_I\}} \nabla_{w^l} \mu^l(i) ( \phi_{i,e}^l  - \eta  \log\mu^l(i))\right] \,.$$

The goal of the entropy regularization term is to enforce that some minimal population of each species remains on sub-optimal islands. Thus, the population distributions adapt over ecological time so as to minimize the following loss:

$$\left[  \sum \limits_{i \in \{1, \dots, N_I\}} \mu^l(i) ( \phi_{i,e}^l  - \eta  \log\mu^l(i))\right].$$

## Multiagent Reinforcement Learning

A Partially Observable Markov Game (POMG) is sequential decision model of a multiagent environment in which $N$ individuals interact. At each state $s \in S$ of a POMG, each agent selects an action $a^i \in A^i$ based on the observation $o^i$ of the state of the game they have. The observation of player $i$ is defined here as a function of the state $o^i = \psi^i(s)$. Then the state changes to $s' \sim p(.|s,a^1,\dots,a^N)$ and the individuals receive reward $r^i(s,a^1,\dots,a^N)$. Each species learns a policy $\pi^l(a^i|o^i)$ given the experience of each of its individuals. At each step, all individuals of a species collect trajectories of the experience gathered in the island they have been assigned to. The reinforcement learning algorithm produces gradient updates of the parameters for each individual of the species. The gradient updates are then averaged over all individuals of the species to update the parameters $\{\theta_l\}_{\{1,\dots,L\}}$. The V-trace algorithm is used to update the parameters as described in with truncation levels set to $1$. Note that experience (observations, actions and rewards) from all individuals of a species contribute equally, but that the individuals may be spread non-uniformly over islands. This means that the species parameter update may be disproportionately affected by the performance of the species on particular islands.

#### Function approximation:

The neural network architecture was similar to that of . It consists of a convnet with $16$ channels, kernel size of $3$, and stride of $1$. The output of the convnet is passed to a a 1-layer MLP of size $32$, followed by a recurrent module (an LSTM ) of size $64$. The recurrent module's output is then linearly transformed into the the policy and value. All nonlinearities between layers were rectified linear units.

#### Distributed computing:

The island simulation and the species neural network updates were implemented as separate processes, potentially running on different machines. Islands produce trajectories and send them to a circular queue on the species update process. The species update process waits until it can dequeue a complete batch of $32$ trajectories, at which point it computes the v-trace update.

#### Environments:

The games studied in this work are all partially observable in that individuals can only observe via a 15$\times$`<!-- -->`{=html}15 RGB window, centered on their current location. The action space consists of moving left, right, up, and down, rotating left and right. Each species was assigned a unique color, shared by all conspecifics and preserved across all islands.

# Results

## Exploration experiments

Given an unrefined and infrequently emitted behavior, reinforcement learning algorithms are very good at estimating its value with respect to alternatives and refining it into a well-honed strategy for achieving rewards. However, a central problem in reinforcement learning concerns the initial origin of such behaviors, especially in cases where the state space is too large for exhaustive search, and there are many local optima where the policy's reward gradient becomes zero.

This section explores how population dynamics may drive innovation in individual behavior. To study this, we introduce a new game that taxes individual exploration skills. It can be seen as a multi-agent analog of the well-known Montezuma's Revenge single player game that has often been used for studies of intrinsic motivations for single-agent exploration . We hold to the game theory tradition of introducing each game with a facetious (but hopefully memorable) story, and offer the following:

In the *Clamity* game, agents begin in the trochophore stage of the bivalve mollusk lifecycle. They can freely swim around the map, a partially observed grid-world (map size = $36 \times 60$, window size = $15 \times 15$. Then whenever they are ready, they can perform the \*settle\* action. This action causes the agent to metamorphose into the adult clam stage of their lifecycle at their current location and removes their ability to swim. After settling, their shell grows around them, up to a maximum size. Shell growth is also restricted by the presence of adjacent shells from other clams. Each adult clam filters invisible food particles from the ocean at a rate proportional to the size of its shell, receiving reward for each food particle filtered. However, clam shells that are adjacent to the shell of another individual become unhealthy and do not filter any food. There are also nutrient patches located a considerable distance away from the starting location (more than $10$ steps away, see maps in Fig. -A). Individuals that settle near a nutrient patch so that it is either partially or fully engulfed in their shell absorb additional nutrients from it. Episodes terminate after $T = 250$ steps. Settling immediately on the first action is a very attractive local optimum. The global optimum solution is to swim quickly out to a nutrient patch and settle there instead[^3].

Single-agent reinforcement learning algorithms become stuck in the local optimum[^4] and fail to ever discover the nutrient patches. To see why, consider the number of consecutive seemingly suboptimal actions that an agent would have to take in order to discover a nutrient patch. The settle action can be taken at any time, it always provides some level of rewards, and once taken, prevents movement for the rest of the episode. Thus any reasonable reinforcement learning algorithm that follows the initial gradient of its experience will reach the local optimum. If it starts out settling on step $t_s > 1$, it will receive an expected return of $T - t_s \times \text{reward rate}$. But if it were to settle earlier instead, e.g., on step $t_s - 1$, it would receive a larger expected return. Thus there is a strong gradient from any policy initialization to the local optimum of settling on step $t_s = 1$. Furthermore. since the environment is partially observable, a single agent would need to choose to move in the same direction for several steps despite registering no change at all in its observation during that time.

On the other hand, Clamity can also be played by multiple agents simultaneously. All the trochophores begin each episode nearby one another in the center of the map. Since intersecting shells become unhealthy and provide no reward, individuals are penalized for settling too close to one another[^5]. This provides a gradient that incentivizes agents to swim away from the starting location to avoid competing with one another for shell space. If the population size is large enough then this competition-motivated spreading eventually leads individuals to discover the nutrient patches[^6].

### Experimental procedure

To make like-for-like comparisons between single-agent and multi-agent training regimes, we adopt the following protocol. In parallel with the archipelago ($N_I$ islands), we run $L$ (the number of species) additional *solitary islands*. On the $l$-th solitary island, a single individual of species $l$ plays each episode alone. All the experience generated on islands where species $\Psi_l$ appears, even its solitary island, is used to update its policy $\pi^l$. However, the amount of each species's total experience derived from the solitary island is comparatively small since in this experiment, $M = 960$, the number of individuals of species $l$ appearing across all islands of the archipelago. The final results are reported only from the solitary islands but reflect the policy learning accumulated in the competitive archipelago setting.

Our single agent training protocol simply sets the number of islands in the archipelago $N_I$ to $0$ and replicates each solitary island $32$ times. Since there is only a single species ($L = 1$), and all solitary island replicas are the same as one another (though with different random environment seeds), the result is exactly equivalent to the A3C training regime .

This protocol also makes it easy to compare the proposed training regime where population sizes are dynamic and variable from episode to episode to the case of a "standard self-play" training regime, where population sizes are fixed. In this case, the archipelago contains just one island inhabited by a fixed number of individuals. As before, most of the experience is generated from the island where multiple individuals play. Results are reported only from the (single) solitary island, just as it is in the dynamic case.

### Results

Individuals of species trained by Malthusian reinforcement learning find the globally optimum single-player solution, despite most of their experience coming from multi-player islands. Individuals trained by two baseline single-agent reinforcement learning algorithms completely fail to escape the local optimum. The first baseline we tried had all the exact same hyperparameters as in the Malthusian case, but all of its experience was in solitary islands ($32$ of them in parallel).[^7] The second single-agent reinforcement learning algorithm baseline we tried was an implementation of the current state-of-the-art in curiosity-driven reinforcement learning, the intrinsic curiosity module provides a pseudo-reward to the agents based on its prediction error in predicting the next timestep in the evolution of a compressed encoding of its observations . In this case, augmenting the agent with the intrinsic curiosity module is still insufficient to get it it to consistently discover the nutrient patches. It does stumble upon them from time to time, especially early on in training (Fig. , but does not even do so consistently enough to register in a smoothed plot of rewards versus time with a $100$ step smoothing window (Fig. ). In contrast, individual members of species trained by Malthusian reinforcement learning with dynamic population sizes consistently implement globally optimal policies once they have discovered them (Fig. ).

Next we asked whether dynamic population sizes were specifically important or whether the key was just the simultaneous training in multi-agent islands with a given, sufficiently large, population size. We noticed that most runs with dynamical population sizes converged to an island population size around $32$ in the best performing islands. Thus we ran several experiments where agents trained in fixed population islands, evaluated on solitary islands as before. We found that individuals that trained in a fixed population size of $32$ were able to discover the global optimum, but apparently less consistently than in the case with dynamic population size (red curve above navy curve in Fig. ), and apparently with greater vulnerability to forgetting (the navy colored curve eventually declines back to the local optimum).

## Mutualism and specialization experiments

Solution concepts for general-sum games may involve mutualistic interactions between synergistic strategies. Successful cooperative joint strategies may be either homogeneous, as in facultative mutualism, or heterogeneous. In nature, partners in mutually profitable associations are often very different from one another so that they can provide complementary capabilities to the partnership. In fact, most known mutualisms involve partners from different kingdoms, e.g., corals and their algae symbionts, vascular plants and mycorrhizal fungi, mammals and their gut bacteria, etc . Moreover, division of labor and the subsequent efficiency gains from specialization are thought to be key components of complex human society .

However, it may be difficult to learn such mutually profitable partnerships of widely divergent strategies with self-play. All partners would need to represent all specializations, wasting valuable representation capacity. In addition, a policy learned by self-play requires a switching mechanism to break the symmetry and determine which sub-policy to emit in any given situation. For example, an agent could learn to become a blacksmith if standing on the left and a farmer if standing on the right. The complexity of the switching policy is itself related to the extent of partial observability in the environment. In some cases it may be very difficult to determine the right proportion of individuals needed to perform each part of the partnership at any given time, e.g. if the others' strategies cannot easily be observed. It would be easier to learn a heterogeneous set of policies, each one implementing only its own part of the partnership. But then, it would seem that the number of copies of each would have to be known in advance, thus adding many new difficult-to-tune hyperparameters, one for each species.

In this section we explore whether Malthusian reinforcement learning can find mutualistic partnerships more easily than other multi-agent reinforcement learning methods, especially when there is a potential for gains from heterogeneous populations containing multiple specialized members. To test this, we created another partially observed Markov game environment. Again continuing the game-theoretic tradition of accompanying each game with a facetious and memorable story, we offer the following.

The Allelopathy game has two main rules. (1) shrubs grow in random positions on an open field. Shrubs allelopathically suppress one another's growth. That is, the probability that a seed of a given type grows into a shrub in any given timestep is inversely proportional to the number of nearby shrubs of other types. (2) Agents in Allelopathy are herbivorous animals that can eat many different types of shrub. However, switching frequently between digesting different shrub types imposes a metabolic cost since different enzymes must be synthesized for each. Thus, agents benefit from specialization in eating only a single type of shrub. Agents receive increasing rewards for repeatedly harvesting the same type of shrub (up to a maximum of $r = 250$). Rewards drop back down to their lowest level, $(r = 1)$, when the agent harvests a different type of shrub (since that entails their switching into a different metabolic regime). Thus an agent that randomly harvests any shrub they come across is likely to receive low rewards. An agent that only harvests a particular type of shrub while ignoring others will obtain significantly greater rewards. The combined effect of these two rules is to make it so that a specialist in any one shrub type benefits from the presence of others who specialize in different shrub types since their foraging increases the growth rate of all the shrubs they do not consume.

We studied two variants of the Allelopathy game. The first variant, unbiased Allelopathy, has two shrub types $A$ and $B$ that appear with equal probability. In the second variant, biased Allelopathy, the two shrub types do not appear with the same frequency. Type $A$ is significantly more common than type $B$. In addition, each shrub of type $A$ consumed provides a maximum reward of $8$ when at least $8$ in a row are consumed. Whereas type $B$ shrubs yield a maximum reward of $250$ for any agent that manages to consume that many consecutively. Biased Allelopathy is a social dilemma since specialists in type $B$ are clearly better off than specialists in type $A$, but both do better when the other is around.

### Results

Here the critical comparison is between homogeneous $(L = 1)$ and heterogeneous $(L > 1)$ population dynamics. Therefore the object of study is the performance of the *islands* rather than specific individuals. The Allelopathy environment contains two niches, corresponding to specialization in consuming either shrub type $A$ or $B$. In the heterogeneous case, mutualistic partnerships may develop from initial conditions where species in proximity to one another randomly fill either role. This situation features a gradient that guides each species in different directions. Whichever species begins with a propensity toward role $A$ ends up specializing in role $A$. Likewise, the other species evolves to specialize in role $B$, to the mutual benefit of both partners. On the other hand, in the homogeneous case, it is still possible for mutualistic interactions to develop, but it is more difficulty since (1) both specialized parts of the joint policy must be represented in the same network, and (2) the policy must include a switching mechanism that breaks the symmetry, determining which sub-policy to implement in each situation. Heterogeneous species avoid the need for this symmetry breaking, and the relative proportions assigned to each role are handled naturally by the adapting relative population sizes (Fig. ).

The total number of individuals in each experiment was $K = 960$. Thus in the homogeneous $L = 1$ case, $M = 960$, and in the heterogeneous $L = 4$ case, $M = 240$. The number of islands $N_I$ was $60$ in both dynamic population size conditions. In the fixed population size conditions the number of islands was chosen so that the total number of individual instances would still be $960$, e.g., for fixed population size $32$, this required $N_I = 960 / 32 = 30$.

Results were similar for both the biased and unbiased Allelopathy games. Heterogeneous $(L = 4)$ population dynamics achieved higher returns, both per capita, and in aggregate, than the other tested methods including the homogeneous population $(L = 1)$ with size dynamics (Fig. ). Interestingly both heterogeneous and homogeneous runs converged to the same population size, but the heterogeneous case increased more slowly to that point, and did so while maintaining a higher per capita rate of return.

# Discussion

This paper introduces Malthusian reinforcement learning, a multi-agent reinforcement learning algorithm that motivates individual exploration and takes advantage of possibilities for synergy to evolve heterogeneous mutualisms. If populations rise when returns improve then the problem itself shifts over time so no local optimum need ever be reached. This gives rise to a strategy that we may term *exploration by exploitation*. Individuals can always follow the gradient of their experience, they need never depart from their current estimate of the best policy just to explore the state space. They will naturally explore it, just by following the gradient in a changing world. Opportunities for heterogeneous mutualism may also be detected by gradient following. Initially weak specialization in one agent incentivizes its soon-to-be partner to specialize in a complementary direction, which in turn catalyzes more specialization, and so on.

How does this paper's proposed population dynamic relate to dynamics studied in evolutionary theory? Our requirement of conservation of compute, that the number of individuals of a given species on a given island may vary from episode to episode, but the total number of individuals of each species in the archipelago is always the same fixed value, implies that for populations to increase on one island they much decrease elsewhere. Thus the population dynamic introduced here may be understood as an evolutionary model of migration. Moreover, since fitnesses are computed globally, i.e. relative to the entire archipelago, it is more similar to *hard* selection models in evolutionary theory where populations are regulated globally than *soft* selection, where population regulation occurs locally within each island .

Other possible relationships between population size and innovation have appeared in the evolutionary anthropology literature. For instance, it is possible that—especially in preliterate societies—larger populations provide for more protection from forgetting of useful cultural elements since more elders, functioning as repositories of cultural knowledge, will be alive at any given time . Or alternatively, larger social networks may provide more opportunities for recombination of disparate cultural elements that originated in farther and farther away contexts . As hypotheses for the origin of innovative behaviors in biology, these possibilities appear to be at odds with the mechanism implied by our algorithm, since each could explain, for instance, the same correlations between brain size and group size across the primate order as well as innovation and (cultural) group size in humans . However, they are not mutually exclusive. In fact, all three mechanisms may even operate synergistically with one another. More research is needed in order to tease apart the precise mechanisms in biology. In computer science, we think this line of thought opens up a goldmine of new algorithmic ideas concerning the combination of population dynamics with social learning and imitation.

[^1]: The rapid dispersal of homo sapiens across the globe and their adaptation to the full range of diverse terrestrial habitats can be regarded as a great feat of intelligence. We may even assess its universal intelligence $\Upsilon$ by adapting the following definition from , $\Upsilon(\pi) = \sum_{h \in \mathcal{H}} 2^{-K(h)} V_h^\pi$, where $\pi$ is a policy, $H$ is the set of terrestrial habitats, $V_h^\pi$ is the expected value of the sum of rewards from inhabiting the habitat $h$, and $K(h)$ is a measure of the complexity of $h$. The complexity measure could be defined as any sensible ecological distance metric measuring the distance from the ancestrally adapted environment to $h$.

[^2]:  argues that preindustrial human populations generally oscillated around a fixed, and only very slowly increasing, carrying capacity until the industrial revolution. Similar oscillations in subpopulation sizes were recently observed in a large-scale multi-agent learning simulation by . As those authors pointed out, its possible for population dynamics to endlessly oscillate rather than increasing over time. The same is true for the strategies used by learning algorithms based on self-play. One fix that was used in AlphaGo and elsewhere is to require agents to learn to defeat all previous versions of themselves, not just the most recent . This prevents self-play-based agents from endlessly learning and forgetting the same exploit and defense.

[^3]: A video of the single-agent global optimum policy can be viewed here: .

[^4]: A video of the single-agent local optimum policy can be viewed here: .

[^5]: A video of such a multiplayer bad outcome can be viewed here: .

[^6]: A video of a group of agents implementing a multiplayer global optimum joint policy can be viewed here: <https://youtu.be/TnxMnSClBHY>.

[^7]: These hyperparameters were not tuned for the Malthusian case—they were prespecified before the runs of both methods, and not subsequently changed.