abstract: This paper demonstrates the use of genetic algorithms for evolving a grandmaster-level evaluation function for a chess program. This is achieved by combining supervised and unsupervised learning. In the supervised learning phase the organisms are evolved to mimic the behavior of human grandmasters, and in the unsupervised learning phase these evolved organisms are further improved upon by means of coevolution.
  .
  While past attempts succeeded in creating a grandmaster-level program by mimicking the behavior of existing computer chess programs, this paper presents the first successful attempt at evolving a state-of-the-art evaluation function by learning only from databases of games played by humans. Our results demonstrate that the evolved program outperforms a two-time World Computer Chess Champion.
author:
Eli (Omid) David

H. Jaap van den Herik

Moshe Koppel

;
Nathan S. Netanyahu

title: Simulating Human Grandmasters: Evolution
  and Coevolution of Evaluation Functions

\[Parameter learning\]

# Introduction

Despite the many advances in Machine Learning and Artificial Intelligence, there are still areas where learning mechanisms have not yielded performance comparable to humans. Computer chess is a prime example of the difficulties in such fields.

It is well-known that computer games have served as an important testbed for spawning various innovative AI techniques in domains and applications such as search, automated theorem proving, planning, and learning. In addition, the annual World Computer Chess Championship (WCCC) is arguably the longest ongoing performance evaluation of programs in computer science, which has inspired other well-known competitions in robotics, planning, and natural language understanding.

Computer chess, while being one of the most researched fields within AI, has not lent itself to the successful application of conventional learning methods, due to its enormous complexity. Hence, top chess programs still resort to manual tuning of the parameters of their evaluation function, typically through years of trial and error. The evaluation function assigns a score to a given chess position and is thus the most critical component of any chess program.

Currently, the only successful attempt reported on automatic learning of the parameter values of an evaluation function is based on "mentor-assisted" evolution . This approach evolves the parameter values by mimicking the evaluation function of an available chess program that serves as a "mentor". It essentially attempts to reverse engineer the evaluation function of this program by observing the scores it issues for a given set of positions. The approach relies heavily on the availability of the numeric evaluation score of each position, which is provided by the reference program.

In this paper, we deal successfully with a significantly more difficult problem, namely that of evolving the parameter values of an evaluation function by relying solely on the information available from games of human grandmasters, i.e., the moves played. Lacking any numeric information provided typically by a standard chess program, we combine supervised and unsupervised learning. The organisms are first evolved to mimic the behavior of these human grandmasters by observing their games, and the best evolved organisms are then further evolved through coevolution. The results show that our combined approach efficiently evolves the parameters of interest from randomly initialized values to highly tuned ones, yielding a program that outperforms a two-time World Computer Chess Champion.

In Section 2 we review past attempts at applying evolutionary techniques in computer chess. We also compare alternative learning methods to evolutionary methods, and argue why the latter are more appropriate for the task in question. Section 3 presents our new approach, including a detailed description of the framework of the GA as applied to the current problem. Section 4 provides our experimental results, and Section 5 contains concluding remarks and suggestions for future research.

# Learning in Computer Chess

While the first chess programs could not pose a challenge to even a novice player, the current advanced chess programs are on par with the strongest human chess players, as the recent man vs. machine matches clearly indicate. This improvement is largely a result of deep searches that are possible nowadays, thanks to both hardware speed and improved search techniques. While the search depth of early chess programs was limited to only a few plies, nowadays tournament-playing programs easily search more than a dozen plies in middlegame, and tens of plies in late endgame.

Despite their groundbreaking achievements, a glaring deficiency of today's top chess programs is their severe lack of a learning capability (except in most negligible ways, e.g., "learning" not to play an opening that resulted in a loss, etc.). In other words, despite their seemingly intelligent behavior, those top chess programs are mere brute-force (albeit efficient) searchers.

## Conventional vs. Evolutionary Learning in Computer Chess

During more than fifty years of research in the area of computer games, many learning methods such as reinforcement learning have been employed in simpler games. Temporal difference learning has been successfully applied in backgammon and checkers . Although temporal difference learning has also been applied to chess , the results showed that after three days of learning, the playing strength of the program was only 2150 Elo (see Appendix B for a description of the Elo rating system), which is a very low rating for a chess program. Wiering provided formal arguments for the failure of these methods in more complicated games such as chess.

The issue of learning in computer chess can be seen as an optimization problem. Each program plays by conducting a search, where the root of the search tree is the current position, and the leaf nodes (at some predefined depth of the tree) are evaluated by some static evaluation function. In other words, sophisticated as the search algorithms may be, most of the knowledge of the program lies in its evaluation function. Even though automatic tuning methods, that are based mostly on reinforcement learning, have been successfully applied to simpler games such as checkers, they have had almost no impact on state-of-the-art chess engines. Currently, all top tournament-playing chess programs use hand-tuned evaluation functions, since conventional learning methods cannot cope with the enormous complexity of the problem. This is underscored by the following four points.

\(1\) The space to be searched is huge. It is estimated that there are about $10^{46}$ possible positions that can arise in chess . As a result, any method based on exhaustive search of the problem space is infeasible.

\(2\) The search space is not smooth and unimodal. The evaluation function's parameters of any top chess program are highly co-dependent. For example, in many cases increasing the values of three parameters will result in a worse performance, but if a fourth parameter is also increased, then an improved overall performance would be obtained. Since the search space is not unimodal, i.e., it does not consist of a single smooth "hill", any gradient-ascent algorithm such as hill climbing will perform poorly. In contrast, genetic algorithms are known to perform well in large search spaces which are not unimodal.

\(3\) The problem is not well understood. As will be discussed in detail in the next section, even though all top programs are hand-tuned by their programmers, finding the best value for each parameter is based mostly on educated guessing and intuition. (The fact that all top programs continue to operate in this manner attests to the lack of practical alternatives.) Had the problem been well understood, a domain-specific heuristic would have outperformed a general-purpose method such as GA.

\(4\) We do not require a global optimum to be found. Our goal in tuning an evaluation function is to adjust its parameters so that the overall performance of the program is enhanced.

In view of the above points it seems appropriate to employ GA for automatic tuning of the parameters of an evaluation function. Indeed, at first glance this appears like an optimization task, well suited for GA. The many parameters of the evaluation function (bonuses and penalties for each property of the position) can be encoded as a bit-string. We can randomly initialize many such "chromosomes", each representing one evaluation function. Thereafter, one needs to evolve the population until highly tuned "fit" evaluation functions emerge.

However, there is one major obstacle that hinders the above application of GA, namely the fitness function. Given a set of parameters of an evaluation (encoded as a chromosome), how should the fitness value be calculated? For many years, it seemed that the solution was to let the individuals, at each generation, play against each other a series of games, and subsequently record the score of each individual as its fitness value. (Each "individual" is a chess program with an appropriate evaluation function.)

The main drawback of this approach is the unacceptably large amount of time needed to evolve each generation. As a result, severe limitations were imposed on the length of the games played after each generation, and also on the size of the population involved. With a population size of 100 and a limit of 10 seconds per game, and assuming that each individual plays each other individual once in every generation, it would take 825 minutes for each generation to evolve. Specifically, reaching the 100th generation would take up to 57 days. As we see in the next section, past attempts at applying this process resulted in weak programs, which were far inferior to state-of-the-art programs.

In Section 3 we present our GA-based approach for using GA in evolving state-of-the-art chess evaluation functions. Before that, we briefly review previous work of applying evolutionary methods in computer chess.

## Previous Evolutionary Methods Applied to Chess

Despite the abovementioned problems, there have been some successful applications of evolutionary techniques in computer chess, subject to some restrictions. Genetic programming was successfully employed by Hauptman and Sipper for evolving programs that can solve Mate-in-N problems and play chess endgames.

Kendall and Whitwell used evolutionary algorithms for tuning the parameters of an evaluation function. Their approach had limited success, due to the very large number of games required (as previously discussed), and the small number of parameters used in their evaluation function. Their evolved program managed to compete with strong programs only if their search depth (lookahead) was severely limited.

Similarly, Aksenov employed genetic algorithms for evolving the parameters of an evaluation function, using games between the organisms for determining their fitness. Again, since this method required a very large amount of games, it evolved only a few parameters of the evaluation function with limited success. Tunstall-Pedoe also suggested a similar approach, without providing an implementation.

Gross *et al.*  combined genetic programming and evolution strategies to improve the efficiency of a given search algorithm using a distributed computing environment on the Internet.

David, Koppel, and Netanyahu used "mentor-assisted" evolution for reverse engineering the evaluation function of a reference chess program (the "mentor"), thereby evolving a new comparable evaluation function. Their approach takes advantage of the evaluation score of each position considered (during the training phase), that is provided by the reference program. In fact, this numeric information is key to simulating the program's evaluation function. In other words, notwithstanding the high-level performance of the evolved program, the learning process is heavily dependent on the availability of the above information.

In this paper, we combine supervised evolution and unsupervised coevolution for evolving the parameter values of the evaluation function to simulate the moves of a human grandmaster, without relying on the availability of evaluation scores of some computer chess program. As will be demonstrated, the evolved program is on par with today's strongest chess programs.

# Evolution and Coevolution of Evaluation Functions

Encoding the parameters of an evaluation function as a chromosome is a straightforward task, and the main impediment for evolving evaluation functions is the difficulty of applying a fitness function (a numerical value representing how well the organism performs). However, as previously noted, establishing the fitness evaluation by means of playing numerous games between the organisms in each generation (i.e., single-population coevolution) is not practical.

As mentioned earlier, the fitness value in mentor-assisted evolution is issued as follows. Both an organism and a grandmaster-level chess program are run on a given set of positions; for each position the difference between the evaluation score computed by the organism and that computed by the reference program is recorded. The fitness value is taken to be inversely proportional to this difference.

In contrast, no evaluation scores of any chess program are assumed available in this paper, and we only make use of (widely available) databases of games of human grandmasters. The task of evolution, in this case, is thus significantly more difficult than that based on an existing chess program, as the only information available here consists of the actual moves played in the positions considered.

The evaluation function is evolved by learning from grandmasters according to the steps shown in Figure .

Although performing a search for each position appears to be a costly process, in fact it consumes little time. Conducting a 1-ply search amounts to less than a millisecond for a typical chess program on an average machine, and so one thousand positions can be processed in one second. This allows us to use a large set of positions for the training set.

The abovementioned process, which will be discussed below in greater detail, results in a grandmaster-level evaluation function (see next section). Due to the random initialization of the chromosomes, each time the above process is applied, a different "best evolved organism" is obtained. Comparing the best evolved organisms from different runs, we observe that even though they are of similar playing strength, their evolved parameter values differ, and so does their playing style.

After running the supervised evolution process a number of times, we obtain several evolved organisms. Each organism is the best evolved organism from one complete run of the evolutionary process. We next use a coevolution phase for further improving upon the obtained organisms. During this single-population coevolution phase the evolved organisms play against each other, and the fitness function applied is based on their relative performance. Completing this phase for a predetermined number of generations, the best evolved organism is selected as the best overall organism. According to the results in the next section, this "best of best" organism improves upon the organisms evolved from the supervised phase. As noted before, previous attempts at applying coevolution have failed to produce grandmaster-level evaluation functions. The difference here is that the population size is small (we used 10), and the initial organisms are already well tuned (in contrast to randomly initialized).

In the following subsections, we describe in detail the chess program, the implementation of the supervised and unsupervised evolution, and the GA parameters used.

## The Chess Program and the Evaluation Function

Our chess program uses NegaScout/PVS search, in conjunction with standard enhancements such as null-move pruning , internal iterative deepening , dynamic move ordering (history + killer heuristic) , multi-cut pruning , selective extensions (consisting of check, one-reply, mate-threat, recapture, and passed pawn extensions), transposition table , and futility pruning near leaf nodes .

The evaluation function of the program (which we are interested in tuning automatically) consists of 35 parameters. Even though this is a small number of parameters in comparison to other top programs, the set of parameters used does cover all important aspects of a position, e.g., material, piece mobility and centricity, pawn structure, and king safety.

The parameters of the evaluation function are represented as a binary bit-string (chromosome size: 224 bits), initialized randomly. The value of a pawn is set to a fixed value of 100, which serves as a reference for all other parameter values. Except for the four parameters representing the material values of the pieces, all the other parameters are assigned a fixed length of 6 bits per parameter. Obviously, there are many parameters for which 3 or 4 bits suffice. However, allocating a fixed length of 6 bits to all parameters ensures that *a priori* knowledge does not bias the algorithm in any way.

Note that the program's evaluation function is randomly initialized, i.e., other than knowing the rules of the game, the program has essentially no game skills at all at this point.

## Supervised Evolution using Human Grandmaster Games

As indicated, our goal is to evolve the parameters of a program's evaluation function, so as to simulate the moves played by grandmasters for a given set of positions.

For our experiments, we use a database of 10,000 games by grandmasters of rating above 2600 Elo, and randomly pick one position from each game. We pick winning positions only, i.e., positions where the side to move ultimately won the game (e.g., if it is white's turn to move, the game was won eventually by white). Of these 10,000 positions, we select 5,000 positions for training and 5,000 for testing.

In each generation, for each organism we translate its chromosome bit-string to a corresponding evaluation function. For each of the $N$ test positions (in our case, $N=5,000$), the program performs a 1-ply search using the decoded evaluation function, and the best move returned from the search is compared to that of the grandmaster in the actual game. The move is deemed "correct" if it is the same as the move played by the grandmaster, and "incorrect" otherwise. The fitness of the organism is calculated as the square of the total number of correct moves.

Note, unlike the mentor-assisted approach for mimicking an existing chess program, which provides numeric values for each position, here we only have 1-bit of information for each processed position (correct/incorrect). This underscores why relying on human games is much more difficult than using computers as mentors.

Other than the special fitness function described above, we use a standard GA implementation with Gray coded chromosomes, fitness-proportional selection, uniform crossover, and elitism (the best organism is copied to the next generation). The following parameters are used:

population size = 100
crossover rate = 0.75
mutation rate = 0.005
number of generations = 200

## Coevolution of the Best Evolved Organisms

Rerunning the supervised evolution ten times, we obtain ten "best organisms" corresponding to the various runs. The evaluation functions of these evolved organisms do not have the same evolved parameter values, since each run produces different results (due to the random initialization). Although the ten programs are of similar playing strength, their playing style is somewhat different. At any rate, the above ten best organisms are used for the coevolution phase described below. Note that selecting, instead, the top ten evolved organisms from one of the supervised runs is not desirable, as it could result in "inbreeding", in the sense that the parameter values of these organisms tend to be fairly similar.

Consider, on the other hand, generating multiple evolved organisms using different training sets for each run. Specifically, for each run we might pick games of a specific grandmaster, in the hope of obtaining organisms that mimic the individual styles of the various grandmasters. Preliminary tests suggest, however, that this variant provides no additional insight or improvement. Apparently, the 1-ply searches enable mimicking only a "generic" grandmaster style, rather than the style of a specific player.

In the coevolution phase, the ten best organisms selected serve as the initial population, which is then coevolved over 50 generations. In each generation, each organism plays four games against each other organism (to obtain a more reliable result). At the end of each generation, rank-based selection is applied for selecting the organisms for breeding. Elitism is used here as well, which ensures that the best organism survives for the next generation. This is especially critical in light of the small population size. Other GA parameters remain unchanged, that is, uniform crossover with crossover rate of 0.75 and mutation rate of 0.005.

In the following section we present our experimental results, both in terms of the learning efficiency and the performance gain of the best evolved individual.

# Experimental Results

We now present the results of running the evolutionary process described in the previous section. We also provide the results of several experiments that measure the strength of the evolved program in comparison to Crafty, a former two-time World Computer Chess Champion that is commonly used as a baseline for testing chess programs.

## Results of Supervised Evolution

Running the evolution for 200 generations, the average number of solved positions (i.e., the number of correct moves found) increases until stabilizing at around 1,500 (out of 5,000), which corresponds to 30% of the positions. The best organism at generation 200 solves 1,621 positions, which corresponds to 32.4% of the positions. Due to the use of elitism, the number of solved positions for the best organism is monotonically increasing, since the best organism is preserved. The entire 200-generation evolution took approximately 2 hours on our machine (see Appendix A).

At first glance, a solution rate of 32% might not seem too high. However, considering that the evolved organism selects successfully the "correct" move in one out of three cases, by applying merely a 1-ply search (as opposed to the careful analysis of a position by the grandmaster), this is quite satisfactory.

With the completion of the learning phase, we used the additional 5,000 positions set aside for testing. We let our best evolved organism perform a 1-ply search on each of these positions. The number of correctly solved positions was 1538 (30.7%). This indicates that the first 5,000 positions used for training cover most types of positions that can arise, as the success rate for the testing set is close to the success rate for the training set.

To measure the performance of the best evolved organism after the supervised evolution phase (we call this program Evol\*), we conducted a series of matches against the chess program Crafty . Crafty has successfully participated in numerous World Computer Chess Championships (WCCC), and is a direct descendent of Cray Blitz, the WCCC winner of 1983 and 1986. It is frequently used in the literature as a standard reference.

Table  provides the results of 500 games between Evol\* and Crafty. The results show that the evolved organism (Evol\*) is on par with Crafty, clearly demonstrating that the supervised evolution has succeeded in evolving a grandmaster-level evaluation function by purely mimicking grandmaster moves.

| Match | Result | W% | RD |
|:---|:--:|:--:|:--:|
| Evol\* - Crafty | 254.5 - 245.5 | 50.9% | $+6$ |

Results of the games between COEVOL\* and CRAFTY (W% is the winning percentage, and RD is the Elo rating difference (see Appendix B)). Win = 1 point, draw = 0.5 point, and loss = 0 point.

## Results of Coevolution

Repeating the supervised evolutionary process, we obtained each time a "best evolved organism" with a different set of evolved parameter values. That is, each run produced a different grandmaster-level program. Even though the performance of these independently evolved best organisms is fairly similar, our goal was to improve upon these organisms and create an enhanced "best of best" organism.

We applied single-population coevolution to enhance the performance of the program. After running the supervised evolution ten times (which ran for about 20 hours), ten different best organisms were obtained. Using these ten organisms as the starting population, we applied GA for 50 generations, where each organism played each other organism four times in every round. Each game was limited to ten seconds (5 seconds per side). In practice, this coevolution phase ran for approximately 20 hours.

We measured the performance of the best evolved organism after coevolution (we call this program Coevol\*) by conducting a series of matches against Crafty and also against Evol\*. Table  provides the results of 500 games between Coevol\* and Evol\*, and between Coevol\* and Crafty.

| Match | Result | W% | RD |
|:---|:--:|:--:|:--:|
| Coevol\* - Crafty | 304.5 - 195.5 | 60.9% | $+77$ |
| Coevol\* - Evol\* | 293.0 - 212.0 | 58.6% | $+60$ |

Results of the games of COEVOL\* against CRAFTY and EVOL\*.

The results demonstrate that the coevolution phase further improved the performance of the program, resulting in the superiority of Coevol\* to both Crafty and Evol\*.

# Concluding Remarks and Future Research

In this paper we presented a novel approach for evolving grandmaster-level evaluation functions by combining supervised and unsupervised evolution. In contrast to the previous successful attempt which focused on mimicking the evaluation function of a chess program that served as a mentor, the approach presented in this paper focuses on evolving the parameters of interest solely by observing games of human grandmasters, where the only available information to guide the evolution consists of the moves made in these games.

Learning from games of human grandmasters in the supervised phase of the evolution, we obtained several grandmaster-level evaluation functions. Specifically, running the procedure ten times, we obtained ten such evolved evaluation functions, which served as the initial population for the second coevolution phase.

While previous attempts at using coevolution have failed due to the unacceptably large amount of time needed to evolve each generation, the use of coevolution succeeded in our case because the initial population was not random, but relatively well tuned due to the first phase of supervised evolution.

According to our experiments, organisms evolved from randomly initialized chromosomes to sets of highly tuned parameters. The coevolution phase further improved the performance of the program, resulting in an evolved organism which resoundingly defeats a grandmaster-level program. Note that this performance was achieved despite the fact that the evaluation function of the evolved program consists of a considerably smaller number of parameters than that of Crafty, of which the evaluation function consists of over 100 parameters.

In summary, we demonstrated how our approach can be used for automatic tuning of an evaluation function from scratch. Furthermore, the approach can also be applied for enhancing existing highly tuned evaluation functions. Starting from several sets of tuned parameter values of the evaluation function, the coevolution phase can be applied to refine these values, so as to further improve the evaluation function.

Running the supervised evolution phase ten times, together with coevolution, took a total of about 40 hours. Both the supervised and unsupervised phases can be easily parallelized for obtaining linear scalability. During the supervised evolution each organism can be evaluated independently on a different processor, without having to share any information with the other organisms. Also, during coevolution, multiple games can be run in parallel. In this work we ran the experiments on a single processor machine (see Appendix A). Running these tests on an 8-core processor (which is readily available today) would reduce the overall running time from 40 hours to as little as 5 hours.

Finally, the results presented in this paper point to the vast potential in applying evolutionary methods for learning from human experts. We believe that the approach presented in this paper for parameter tuning could be applied to a wide array of problems for essentially "reverse engineering" the knowledge of a human expert.

# Experimental Setup

Our experimental setup consisted of the following resources:

-   Crafty 19 chess program running as a native ChessBase engine.

-   Fritz 9 interface for automatic running of matches, using Shredder opening book.

-   AMD Athlon 64 3200+ with 1 GB RAM and Windows XP operating system.

# Elo Rating System

The Elo rating system, developed by Arpad Elo, is the official system for calculating the relative skill levels of players in chess. The following statistics from the January 2009 FIDE rating list provide a general impression of the meaning of Elo ratings:

-   21079 players have a rating above 2200 Elo.

-   2886 players have a rating between 2400 and 2499, most of whom have either the title of International Master (IM) or Grandmaster (GM).

-   876 players have a rating between 2500 and 2599, most of whom have the title of GM.

-   188 players have a rating between 2600 and 2699, all of whom have the title of GM.

-   32 players have a rating above 2700.

Only four players have ever had a rating of 2800 or above. A novice player is generally associated with rating values of below 1400 Elo. Given the rating difference ($RD$) of two players, the following formula calculates the expected winning rate ($W$, between 0 and 1) of the player:

$$W = \frac{1}{10^{-RD/400} + 1}.$$

Given the winning rate of a player, as is the case in our experiments, the expected rating difference can be derived from the above formula:

$$RD = -400 \log_{10}(\frac{1}{W} - 1).$$


Supergravity and its Legacy
S. Ferrara${}^{\; a,b,c}$
${}^a$*Theoretical Physics Department, CERN
CH - 1211 Geneva 23, SWITZERLAND
*

${}^b$*INFN - Laboratori Nazionali di Frascati
Via Enrich Fermi 40, I-00044 Frascati, ITALY*

${}^c$*Department of Physics and Astronomy and Mani L. Bhaumik Institute for Theoretical Physics,
U.C.L.A., Los Angeles CA 90095-1547, USA*

Abstract

A personal recollection of events that preceded the construction of Supergravity and of some subsequent developments.

**

Ettore Majorana Medal address Contribution to the Proceedings of the International School of Subnuclear Physics, 54th Course: THE NEW PHYSICS FRONTIERS IN THE LHC-2 ERA, Erice, 14 June - 23 June 2016

-3.5ex -1ex -.2ex 2.3ex .2ex The Prelude In the early 1970s I was a staff member at the Frascati National Laboratories of CNEN (then the National Nuclear Energy Agency), and with my colleagues Aurelio Grillo and Giorgio Parisi we were investigating, under the leadership of Raoul Gatto (later Professor at the University of Geneva) the consequences of the application of "Conformal Invariance" to Quantum Field Theory (QFT), stimulated by the ongoing experiments at SLAC where an unexpected Bjorken Scaling was observed in inclusive electron-proton cross sections, which was suggesting a larger space-time symmetry in processes dominated by short distance physics.

In parallel with Alexander Polyakov, at the time in the Soviet Union, we formulated in those days Conformally Invariant Operator Product Expansions (OPE) and proposed the "Conformal Bootstrap" as a non-perturbative approach to QFT.

Conformal Invariance, OPEs and the Conformal Bootstrap have become again fashionable subjects in recent times, because of the introduction of efficient new methods to solve the "Bootstrap Equations" (Riccardo Rattazzi, Slava Rychkov, Eric Tonni and Alessandro Vichi), and mostly because of their role in the AdS/CFT correspondence.

The latter, pioneered by Juan Maldacena, Edward Witten, Steve Gubser, Igor Klebanov and Polyakov, can be regarded, to some extent, as one of the great legacies of higher dimensional Supergravity. It can be used to gain information on strongly coupled gauge theories, and affords a variety of applications to Particle Physics, String Theory and Condensed Matter Physics.

In 1973 I was awarded a Fellowship at the CERN Theory Division, which was considered the most prestigious place where a young European physicist could go. Subsequent events revealed that going to Geneva was a truly timely choice, since when I started my Fellowship, in the fall of 1973, Julius Wess and Bruno Zumino had just formulated supergauge invariant (now called supersymmetric) QFT in four dimensions, and then found that milder divergences occur in these theories with respect to standard renormalizable QFT.

The original name was inherited from fermionic strings, where a local (conformal) fermionic symmetry on the world sheet was introduced by André Neveu and John H. Schwarz, Pierre Ramond, and Jean-Loup Gervais and Benji Sakita.

During my two-year period as a CERN Fellow I wrote four papers with Zumino, two of which also with Wess and Jean Iliopoulos (at the time a visiting scientist at CERN). With Zumino we formulated four dimensional Supersymmetric Yang Mills Theories (the new name for supergauges introduced by Abdus Salam and John Strathdee), thus opening the way to consider supersymmetric extensions of the Standard Model, the ISM and alike.

The merging of supersymmetry with Yang-Mills gauge symmetry was a non-trivial step, and the potential link with the Standard Model was motivated by the so-called "Hierarchy Problem" and "Naturalness of Scales", which in these new theories were alleviated thanks to so-called "Non-Renormalization Theorems".

Many people contributed to these developments. Aside from the previously mentioned collaborators of Zumino, I can surely mention Kelly Stelle, Peter West, Warren Siegel, Jim Gates, Marc Grisaru and Martin Roček.

In retrospect, the most influential of my joint papers with Zumino, written during my period as a Fellow, was the one on the "Supercurrent multiplet", a multiplet which extended Noether's theorem to Supersymmetry and whose components include the energy momentum tensor and a vector-spinor fermionic current, as well as a vector, a scalar and a pseudo scalar. The considerations that follow were among the starting points which led Daniel Freedman, Peter van Nieuwenhuizen and myself to formulate Supergravity in four dimensions as the supersymmetric extension of General Relativity (GR).

Since in theories with local (gauge) symmetries Noether currents couple, to first order, to gauge fields, the fact that the stress tensor couples to the metric tensor in GR indicated that the spinor current should have coupled to a spin-3/2 massless gauge field, of the type introduced by William Rarita and Julian Schwinger (RS) in 1941, and promptly used by Shuichi Kusaka as a hypothetical spin-3/2 neutrino.

The structure of the super-current multiplet indicated two facts that were later exploited, namely that a supersymmetric extension of GR should have involved only two particles of helicities (2,3/2), and that the field corresponding to the latter should have been a Rarita-Schwinger field, to be coupled to GR. This seemed in contrast with a superspace formulation of a Super Riemannian Geometry, which was originally pursued by Pran Nath, Richard Arnowitt and Zumino, where many ordinary fields with different spins were needed.

In the fall of 1975, even if I had offers from the US, following a suggestion of Jacques Prentki, at the time CERN-TH Division leader, I accepted a CNRS position at the École Normale Supérieure in Paris.

There I had the chance to meet Daniel Freedman, who was also trying to work out a consistent theory for spin 3/2 coupled to GR.

Peter van Nieuwenhuizen was then his colleague at Stony Brook. He was also collaborating with us, since he was investigating the quantum properties of fields of different spins coupled to GR, and the possibility of ultraviolet cancellations similar to those encountered in super-Yang-Mills theories.

In the spring of 1976 the breakthrough was achieved. Using the Noether procedure, Fierz rearrangements and Riemann tensor identities (and a computer at the Brookhaven National Laboratory), we proved that there is a unique Lagrangian, with only spin 2 and 3/2, and a unique set of supersymmetry transformation rules, which defines pure Supergravity in four dimensions, the supersymmetric extension of GR.

General coordinate transformations, when combined with Supersymmetry, promoted the latter to a gauge symmetry as well, since spinors in curved space must necessarily depend on the space time point due to the local Lorentz symmetry.

-3.5ex -1ex -.2ex 2.3ex .2ex The Play

The construction of Supergravity as the gauge theory of local supersymmetry prompted Stanley Deser and Zumino to formulate this theory as the Einstein-Cartan version of GR minimally coupled to a (spinor valued) one-form field, the RS gravitino (word coined by Sidney Coleman).

In their work, a quartic spin-3/2 coupling with gravitational strength, required in our formulation by local supersymmetry, originated as a torsion contribution to the spin-connection when treated as an independent field. This extends the so-called first-order or Palatini formulation of GR.

A subsequent superspace formulation of Supergravity by Wess and Zumino in 1978, where an appropriate super torsion constraint was introduced to reproduce the flat super geometry of the super-Poincaré algebra, revealed that our second-order approach, rather than the Palatini-Einstein-Cartan formulation, is equivalent to the Wess-Zumino curved Superspace. In particular, it reproduces the so called "auxiliary fields", needed to have an off-shell formulation and thus a tensor calculus as in standard GR, introduced by Peter van Nieuwhenuizen and myself and Stelle and West.

In the months following the Spring of 1976 several important results were obtained, both during the Summer Institute organized by the École Normale Supérieure in Paris and during subsequent visits of some of us to Stony Brook and to CERN.

In particular, also in collaborations with Eugene Cremmer, Joel Scherk, Bernard Julia, Ferdinando Gliozzi and Peter Breitenlohner, the first matter couplings were obtained and, with Peter van Nieuwnhuizen, the first extended N=2 Supergravity formulated.

In a fundamental parallel work, Gliozzi, David Olive and Scherk discovered that the dual spinor model, after a GSO projection, becomes supersymmetric in space-time and not only on the string world-sheet. Moreover, they found that N=4 Supergravity Coupled to N=4 Yang-Mills is the low energy effective theory of the 10D dual spinor model. Meanwhile Scherk, Lars Brink and Schwarz classified super Yang-Mills theories in all dimensions.

In these milestones of Superstring Theory, Supergravity in diverse dimensions started to emerge as a low-energy limit.

The years 1977 and 1978 were the most performing for Supergravity because the discovery of off-shell formulations allowed to write general matter-coupled Lagrangians and to begin a systematic study of the SuperHiggs effect, the mass generation for the gravitino as a consequence of the spontaneous breaking of local Supersymmetry. This includes a joint work with Cremmer, Julia, Scherk, Girardello and van Nieuwenhuizen.

Moreover, the ultraviolet properties of Supergravity theories started to be investigated and pure supergravity was shown to be finite at one and two loops. This is to be contrasted with the later, fundamental work of Marc Goroff and Augusto Sagnotti, where GR was shown to diverge at two loops. Three-loop counter terms in different forms of Supergravity were however exhibited by different authors, including Renata Kallosh, Grisaru and Deser, Jason Kay and Stelle.

In a pivotal step, in those years examples of all extended supergravities in four dimensions were derived and the appearance of electric magnetic (e.m) duality symmetries was noticed, with duality groups that act in a non-linear way on the scalar fields of the theory.

The N=8 supergravity in 4D was constructed by Cremmer and Julia, and shown to possess a very large duality group, the exceptional group E7(7).

Gauged N=8 Supergravity in 4D was constructed by Bernard de Wit and Hermann Nicolai, and gauged Supergravity in diverse dimensions was studied by many people, including Michael Duff, Murat Günaydin, Chris Hull, Nicholas Warner, Paul Townsend, Leonardo Castellani, Riccardo D'Auria and Pietro Fré. E.m. dualities were later promoted to string U-dualities by Hull and Townsend.

Cremmer, Julia and Scherk constructed Supergravity in the highest possible number of dimensions, eleven, following a previous classification of Super-Poincaré algebras by Werner Nahm.

This theory led Witten to formulate, in the Mid 1990s, the remarkable M-theory hypothesis, thus identifying an elusive theory in eleven space-time dimensions via the strong coupling limit of the Type-IIA Superstring. Through webs of dualities, this step allowed to unify the different form of 10D Superstrings, thus also connecting different types of extended objects, called generically branes, which emerge as solitonic solutions from higher dimensional Supergravity.

Duff had been the first, in the 1980s, to advocate the central role of generic extended objects in an overall picture.

D-branes in string theories first appeared in the classic works of Massimo Bianchi, Gianfranco Pradisi and Sagnotti, and of Joseph Polchinski.

The advent of M theory marked the "second string revolution". The 1980s witnessed extensive investigations of Kaluza-Klein (KK) theories, largely aimed at classifying possible KK consistent reductions (spontaneous compactifications).

General Lagrangians including soft breaking terms of gravitational and/or gauge origin were derived by Cremmer, Luciano Girardello, Antoine van Proeyen and myself, and were applied to supersymmetric models of Particle Physics.

No-scale Supergravity, a theory with a naturally vanishing cosmological constant, was built by Cremmer, Costas Kounnas, Dimitri Nanopoulos and myself, and was readily applied to Physics beyond the Standard Model with John Ellis. Twenty years later these models were linked to string vacua in flux compactifications by Steve Giddings, Shamit Kachru and Polchinski.

All these developments made it possible to derive parameter spaces of masses and couplings for supersymmetric extensions of the Standard Model, which can be explored in experimental searches for new physics, as was done at LEP experiments in the past and is currently done at the LHC collider.

Supersymmetric extensions of the Standard Model were introduced and widely investigated by Pierre Fayet and by Savas Dimopoulos and Howard Georgi, relying heavily the soft-breaking terms proposed by Girardello and Grisaru and derived in Supergravity by Riccardo Barbieri, Carlos Savoy and myself, and independently by Lawrence Hall, Joseph Lykken and Steven Weinberg, and by Arnowitt, Ali Chamseddine and Nath.

"Split Supersymmetry", a mechanism advocated in order to get large splitting among MSSM super partners while keeping the nice predictions of gauge coupling unification was later introduced by James Wells, Nima Arkani-Hamed, Dimopoulos, Gian Giudice, Andrea Romanino, and was further developed by the same authors.

A new wave of activity in Supergravity was spurred in 1984, when Michael Green and Schwarz ignited the "first superstring revolution" with the discovery of their cancellation mechanism (for gauge groups E8xE8 or/and SO(32)), making Heterotic and Type-I superstrings (which can only afford the latter option) and their point limit, 10D (1,0) supergravity, free of potential gauge anomalies and also of the gravitational anomalies previously studied by Luis Alvarez-Gaumé and Witten.

The anomaly cancellation mechanism placed restrictions on compactification manifolds, uncovering enticing connections between the supergravity analysis and Algebraic Geometry.

The Calabi-Yau compactifications investigated by Philip Candelas, Gary Horowitz, Andrew Strominger and Witten resulted in a paradigmatic example of N=2 supergravity analysis in terms of Special Geometry and Mirror Symmetry.

All this machinery of compactifications was thus reconsidered, taking into account restrictions coming from the Green-Schwarz mechanism. Important implications exist for compactifications to six dimensions preserving eight supercharges, where some anomaly canceling terms modify the low energy couplings of tensor multiplets to gauge fields.

In the 1990s, Supergravity evolved into a major tool for the study of Black Holes (BH), and in particular of their BPS properties, given their emergence as supersymmetric solitons in four-dimensional N-extended Supergravity.

An interesting interplay between charged black-holes and the moduli-space geometry of the underlying Supergravity emerged through the "Attractor Mechanism", a property noticed by Kallosh, Strominger and myself when we met at an Aspen Summer Institute, in Colorado, in 1995.

In N=2 Supergravity supersymmetric black holes enjoy a universal formula for their Bekenstein-Hawking Entropy, in terms of the (square modulus) of the central charge computed at its extremum in moduli space (see also work with Gary Gibbons).

The name attractors was coined since these solitons follow trajectories in the moduli space of scalar fields which, independently of the initial conditions, reach the same extremal point, which coincides with the value of field at the BH Horizon. This explains why the moduli dependent ADM mass is a continuous parameter while the entropy is quantized in terms of the BH charges.

These results were then extended to theories with N-extended supersymmetry and in higher dimensions in work including my collaborators and friends Kallosh, Paolo Aschieri, Bianca Cerchiai, Laura Andrianopoli, Stefano Bellucci, Anna Ceresole, Chamseddine, Gianguido Dall'Agata, D'Auria, Fré, Murat Günaydin, Maria Lledó, Alessio Marrani, Mario Trigiante, Armen Yeranyan and Zumino.

Partial supersymmetry breaking in N=2 supergravity was studied in collaboration with Girardello and Massimo Porrati, with inspiration from the work of Ignatios Antoniadis, Hervé Partouche and Tomasz Taylor, and a minimal model with flat potential was explored.

In 1997 Supergravity found itself again at center stage in the work of Juan Maldacena, amusingly entitled "The Large-N Limit of Superconformal Field Theories and Supergravity", where the famous AdS/CFT duality was formulated, which was soon followed by important additions by Witten, Gubser, Klebanov and Polyakov. This might be called the "third string revolution", and would have been impossible without a particular compactification of type IIB Supergravity on AdS5 x S5 derived in 1985 by Hyiung-Jin Kim, Larry Romans and Peter van Nieuwenhuizen.

This completes my personal recollection of many past years devoted to Supergravity. The recent years 2013-2016 witnessed a revival of spontaneous breaking of local Supersymmetry, boosted by applications to Cosmology, and in particular to the inflationary Universe with a de Sitter phase, with an eye to observational results of the Planck Mission.

A prominent role in this work was played by Andrei Linde, a proponent of Inflationary Cosmology, and indeed Daniel Freedman and I collaborated independently with Kallosh and Linde. I also collaborated with a number of colleagues, including Antoniadis, Dall'Agata, Emilian Dudas, Alex Kehagias, Porrati, Sagnotti, Jesse Thaler, Van Proeyen, Timm Wrase and Fabio Zwirner.

Supergravity has also found applications in higher-spin field theories, via the AdS/CFT correspondence. Supersymmetric versions of OPEs and of the Conformal Bootstrap for N-extended superconformal field theories are currently under investigation. In the quantum domain N=8 Supergravity in four dimensions was proved to be ultraviolet finite at very high loop order, following the pioneering work of Zvi Bern, Lance Dixon and David Kosower. These properties seem to descend from some amazing relations with the rigid N=4 Yang-Mills theory, which possesses exceptional conformal and integrability properties. It is not yet excluded that N=8 Supergravity be a perturbatively finite theory of Quantum Gravity. In any case this search is going to teach us new lessons on the ultraviolet cancellations which can occur in theories including General Relativity.

The author was supported in part by the CERN Th Department and by INFN (IS CSN4-GSS-PI).


abstract: A key science goal of upcoming dark energy surveys is to seek time evolution of the dark energy. This problem is one of *model selection*, where the aim is to differentiate between cosmological models with different numbers of parameters. However, the power of these surveys is traditionally assessed by estimating their ability to constrain parameters, which is a different statistical problem. In this paper we use Bayesian model selection techniques, specifically forecasting of the Bayes factors, to compare the abilities of different proposed surveys in discovering dark energy evolution. We consider six experiments — supernova luminosity measurements by the Supernova Legacy Survey, SNAP, JEDI, and ALPACA, and baryon acoustic oscillation measurements by WFMOS and JEDI — and use Bayes factor plots to compare their statistical constraining power. The concept of Bayes factor forecasting has much broader applicability than dark energy surveys.
author: Pia Mukherjee,$^1$ David Parkinson,$^1$ Pier Stefano Corasaniti,$^2$ Andrew R. Liddle$^1$`<!-- -->`{=html}and Martin Kunz$^3$
*$^1$Astronomy Centre, University of Sussex, Brighton BN1 9QH, United Kingdom*
*$^2$ISCAP, Columbia University, New York, NY 10027, USA*
*$^3$Department of Theoretical Physics, University of Geneva, 24 quai Ernest Ansermet, CH-1211 Geneva 4, Switzerland*
title: Model selection as a science driver for dark energy surveys

# Introduction

Uncovering the nature of dark energy in the Universe is perhaps the greatest challenge facing cosmologists in coming years. In recent months many proposed experiments to probe dark energy have been defined, especially in response to a call for white papers by the Dark Energy Task Force set up jointly in the US by the NSF, NASA and DOE. These propose a variety of techniques to constrain dark energy parameters, including the luminosity distance–redshift relation of Type Ia supernovae (SNe-Ia), the angular-diameter distance–redshift and expansion rate–redshift relations measured by baryon acoustic oscillations, and use of weak gravitational lensing to probe the growth rate of structures.

Following on from heritage of CMB anisotropy studies, the standard tool used to illustrate the power of a given instrument or survey is a plot of the projected parameter errors around one or more fiducial models, estimated using a Fisher information matrix approach or likelihood analysis of Monte Carlo simulated data (Knox 1995; Jungman et al. 1996; Zaldarriaga, Spergel & Seljak 1997; Bond, Efstathiou & Tegmark 1997; Efstathiou & Bond 1999). Typically, a projection of the parameter uncertainties onto a two-parameter equation-of-state model for dark energy is deployed, showing how tightly parameters are expected to be constrained around, for instance, the cosmological constant model. The implication is intended to be that if the true values lie outside those error ellipses, then the survey will be able to exclude the cosmological constant model.

However, the principal goal of such surveys is usually identified as being the discovery of dark energy evolution. This is not a parameter estimation question, but rather one of *model selection* (Jeffreys 1961; MacKay 2003; Gregory 2005), where one seeks to compare cosmological models with different numbers of variable parameters. Within the framework of Bayesian inference, the statistical machinery to make such comparisons exists, and is based around statistics known as the Bayesian evidence and the Bayes factor. The Bayes factor has the literal interpretation of measuring the change in relative probabilities of two models in light of observational data, updating the prior relative model probabilities to the posterior relative model probabilities.

In this paper we use Bayesian model selection tools to assess the power of different proposed experiments. Our method is related to the Expected Posterior Odds (ExPO) forecasting recently developed by Trotta (2005). The main difference is that he takes the present observational constraints on the extended model, and seeks to estimate the fraction of that parameter space within which that model can be distinguished from a simpler embedded model. By contrast, we take a theoretically-motivated view of the parameter space of interest, and seek the locations within that parameter space corresponding to dark energy models which are distinguishable from a cosmological constant by a given experiment. We also differ computationally, in that as well as using approximate techniques, we use the nested sampling algorithm of Skilling (2004), as implemented by Mukherjee, Parkinson & Liddle (2006), to compute the evidences accurately numerically.

The paper is organized as follows. In Section  we introduce model selection in the Bayesian framework. Section  describes the dark energy surveys we make model selection forecasts for, and Section  presents the results. We conclude in Section . We consider some additional technical details and review the standard parameter forecast procedure in two Appendices.

# Bayesian model selection

## The model selection framework

The Bayesian model selection framework has now been described in a variety of places (Jaffe 1996; MacKay 2003; Marshall, Hobson & Slosar 2003; Saini, Weller & Bridle 2004; Gregory 2005; Trotta 2005; Mukherjee et al. 2006) and we will keep our account brief.

In this context, a model is a choice of parameters to be varied to fit the data, its predictions being reflected in the prior ranges for those parameters. A model selection statistic aims to set up a tension between model complexity and goodness of fit to the observed data, ultimately providing a ranked list of models based on their probabilities in light of data. Within Bayesian inference, the appropriate statistic is the *Bayesian evidence* $E$ (also known as the marginal likelihood), which is the probability of the data given the model in question. It is given by integrating the likelihood $P(D|\theta,M)$ over the set of parameters $\theta$ of model $M$, in light of data $D$, i.e. $$E(M) \equiv P(D|M)=\int d\theta \, P(D|\theta,M)P(\theta|M)\,,
\label{evdef}$$ where the prior $P(\theta|M)$ is normalized to unity. The evidence is thus the average likelihood of the model over its prior parameter space. Rather than focusing simply on the best-fit parameters (which will always tend to favour the most complex model available), it additionally rewards models with good predictiveness.

By Bayes Theorem the evidence updates the prior model probability to the posterior model probability. The ratio of the evidences of two models, $M_0$ and $M_1$, is known as the Bayes factor (Kass & Raftery 1995): $$B_{01} \equiv \frac{E(M_0)}{E(M_1)} \,.$$ Note that the prior model probabilities are to be chosen in the Bayesian approach, and different people may have different opinions as to those. Nevertheless, everyone will agree on whether the Bayes factor led to their original belief becoming more or less tenable relative to another model in light of the data. In describing results from Bayes factors, it is common to presume that the prior model probabilities are equal, and we shall follow that practice; anyone who thinks otherwise can readily recalculate the posterior relative model probability.

The Bayesian evidence provides a ranked list of the models in terms of their probabilities, obviating the need to specify an arbitrary significance level as in frequentist chi-squared tests. Nevertheless one still has to decide how big a difference will be regarded as significant. A useful guide as to what constitutes a significant difference between models is given by the Jeffreys' scale (Jeffreys 1961); labelling as $M_0$ the model with the higher evidence, it rates $\ln B_{01} < 1$ as 'not worth more than a bare mention', $1<\ln
B_{01} < 2.5$ as 'substantial', $2.5< \ln B_{01} < 5$ 'strong' to 'very strong' and $5<\ln B_{01}$ as 'decisive'. Note that $\ln
B_{01}=5$ corresponds to odds of 1 in about 150, and $\ln B_{01}=2.5$ to odds of 1 in 13.

## Forecasts and the Bayes factor plot

In order to forecast the power of an experiment for model selection, we ask the following question: Given a well-motivated simpler model embedded in a larger parameter space, how far away does the true model have to lie in order that the experiment is able to exclude the simpler model? There are many such cases present in cosmology, for example $\Lambda$CDM in the space of evolving dark energy models, the question of whether we live in a spatially-flat universe, or whether the initial power spectrum of perturbations is exactly scale invariant, or exactly a power law, etc. Here we will use the dark energy as a worked example. The Bayesian evidence of models with dark energy has been computed from current observational datasets by several authors (Saini et al. 2004; Bassett, Corasaniti & Kunz 2004; Mukherjee et al. 2006), all finding that the simple $\Lambda$CDM model is the preferred fit to present data. Our aim here is to forecast its outcome in light of future datasets, in order to assess the power of those surveys for model selection.

Our procedure is as follows. We first select an experimental configuration. We then consider a set of 'fiducial models' characterized by parameter values $\hat{\theta}$, which we shall consider in turn to be the true model. For each choice of fiducial model in our dark energy space we generate a set of simulated data $D$ with the properties expected of that experiment. We then compute the evidences of the two models we seek to distinguish, here the $\Lambda$CDM model and the general dark energy model. For definiteness, we choose to assess a set of dark energy experiments by their ability to distinguish a $\Lambda$CDM model from a two-parameter dark energy model with equation of state given by $$w(z)=w_0+w_a(1-a),\label{lp}$$ where $w_0$ and $w_a$ are constants and $a$ is the scale factor. Although the latter is sometimes referred to as the Linder parametrization based on its use in Linder (2003), it appears to have been first introduced by Chevallier & Polarski (2001).

Here $\hat{\theta}$ refers to all the parameters of the model, but we are principally interested in the dependence of the Bayes factor on the extra parameters characterizing the extended model, here $w_0$ and $w_a$. Our main plots therefore show the difference in log evidence between the $\Lambda$CDM model and the two-parameter evolving dark energy model, plotted in the $w_0$–$w_a$ plane. This is the Bayes factor plot, which is presented in Section for different dark energy surveys, with contours showing different levels at which the two models can be distinguished by data simulated for each experiment.

In general the Bayes factor is a function of all the fiducial parameters, not just the dark energy ones. For the dark energy application this dependence turns out to be unimportant, but for completeness we discuss some issues relating to this in Appendix A.

Use of the Bayes factor plots to quantify experimental capabilities is quite distinct, both philosophically and operationally, from the use of parameter error forecasts; for readers unfamiliar with the latter we provide a short review in Appendix B. We highlight the advantages of the Bayes factor approach as follows:

1.  Most experiments, particularly dark energy experiments, are motivated principally by model selection questions, e.g. does the dark energy density evolve, and so should be quantified by their ability to answer such questions.

2.  In Bayes factor plots, the data are simulated at each point of the dark energy parameter space that is to be confronted with the simpler $\Lambda$CDM model, whereas parameter error forecasts are plotted around only selected fiducial models (often just one). In particular, in the latter case the data are usually simulated for a model that people hope to exclude, rather than the true model which would allow that exclusion.

3.  The Bayesian model selection procedure accords special status to the $\Lambda$CDM model as being a well-motivated lower-dimensional model, which in Bayesian terms is rewarded for its predictiveness in having a smaller prior volume. Parameter estimation analyses do not recognize a special status for such models, e.g. the same criterion would be used to exclude $w=-0.948$ as $w=-1$. Model selection criteria provide a more stringent condition for acceptance of new cosmological parameters than parameter estimation analyses. Model selection analyses can also accrue positive support for the simpler model, whereas parameter estimation methods can only conclude consistency of the simpler model.

4.  In parameter error studies, it is necessary that the simple model is embedded as a special case of the second model. While the models we discuss here are indeed of that type, the Bayes factor could also be used to compare non-nested models (e.g. two different types of isocurvature perturbation).

5.  Although it is not essential to do so, most parameter estimation forecasts assume a Gaussian likelihood in parameter space, while the Bayes factor plot uses the full likelihood.

Set against these advantages, the only disadvantages of the Bayes factor method are that it is computationally more demanding, and that its conceptual framework has yet to become as familiar as that of parameter estimation.

## Bayes Factor Evaluation

We use the nested sampling algorithm (Skilling 2004; Mukherjee et al. 2006), which is fast enough to enable exact evaluations of the evidence for many fiducial parameter values. For comparison we also compute results with the Savage–Dickey method outlined in Trotta (2005), using a Fisher matrix approximation to the likelihood about the true model, given as equation () in Appendix B. We discuss how the results from the two methods compare in one case, and present our main results using the more accurate nested sampling method.

### Nested Sampling Algorithm

The Bayes factor can be found by calculating the evidences of the two models independently, and then taking their ratio. This method requires integration over the extra cosmological parameters, which does not feature in the Savage–Dickey method. Here we use our implementation of the nested sampling algorithm (as described in Mukherjee et al. 2006) to perform the integration. To quickly summarize, the algorithm (Skilling 2004) recasts the problem as a one-dimensional integral in terms of the remaining 'prior mass' $X$, where $dX = P(\theta|M)d\theta$. The integral becomes $$E = \int_0^1 L(X) dX \,,$$ where $L(X)$ is the likelihood $P(D|\theta,M)$. The algorithm samples the prior a large number of times, assigning an equal prior mass to each sample. The samples are then ordered by likelihood, and the integration follows as the sum of the sequence, $$E = \sum_{j=1}^m E_j\,, \quad E_j=\frac{L_j}{2}(X_{j-1}-X_{j+1}) \,,$$ where the lowest likelihood sample goes into the sum, and is discarded to be replaced by a new sample selected under the condition that it lies above the likelihood of the discarded sample. In this way the algorithm works its way in to the highest likelihood peak.

We compute the evidences using 300 live points, averaging over six repetitions of the calculation. This requires approximately $10^4$ likelihood evaluations per evidence computation.

### Savage–Dickey Formula

Bayes factors for two nested models can be computed using the Savage–Dickey density ratio (Dickey 1971; Verdinelli & Wasserman 1995; see Trotta 2005 for an application to cosmological model selection). Assuming a Gaussian approximation to the likelihood, the Savage–Dickey formula of an extended model $M_1$ with two free model parameters ($\hat{\theta}_1$,$\hat{\theta}_2$) and flat priors ($\Delta\theta_1,\Delta\theta_2$), versus a simpler model $M_0$ with $\hat{\theta}_1=\hat{\theta}^{*}_1$ and $\hat{\theta}_2=\hat{\theta}^{*}_2$, is $$B_{01}(\hat{\theta}_1,\hat{\theta}_2)=\frac{\Delta\theta_1
\Delta\theta_2} {2\pi\sqrt{\det{F^{-1}}}}
e^{-\frac{1}{2}\sum_{\mu\nu}(\hat{\theta}_\mu-\hat{\theta}^{*}_\mu)
F_{\mu\nu} (\hat{\theta}_\nu-\hat{\theta}^{*}_\nu)}\,,\label{savage}$$ where $F_{\mu\nu}$ is the marginalized $2\times2$ Fisher matrix evaluated at $\hat{\theta}_\mu$. Our conventions are defined in Appendix B, and we have used the hat sign for the extended model parameters to emphasize that the Bayes factors directly compare the fiducial models of the parameter estimation analysis to the simpler nested model.

In our specific case, $M_1$ consists of all dark energy models parametrized by different values of $w_0$ and $w_a$, while $M_0$ is the cosmological constant model which is nested in $M_1$ with $w_0=-1$ and $w_a=0$. We use equation () to compute the Bayes factor as function of $w_0$ and $w_a$, to determine the range of dark energy models that a given experiment is able to distinguish from $\Lambda$CDM.

From equation () we can see that the Bayes factor depends on two multiplicative terms, namely an exponential factor and an overall amplitude. The former accounts for the distance in the parameter space of the model $M_1$ from $M_0$ in units of the forecasted parameter uncertainty. The latter accounts for the fraction of the accessible prior volume of the extended model $M_1$ in light of the data, and hence this factor penalizes the model $M_1$ for having a large parameter space compared to model $M_0$. As shown in Trotta (2005), this factor can be interpreted as an estimate of the informative content of the data, $$I=\log_{10}\frac{\Delta\theta_1\Delta\theta_2}{\sqrt{\det{F^{-1}}}} \,,$$ being the order of magnitude by which the prior volume of model $M_1$ will be reduced by the arrival of the forecasted data.

# Dark energy surveys

## The surveys

We have simulated observational data for two types of future dark energy experiments: luminosity distance probes made through the measurement of Type Ia supernovae, and angular-diameter distance measurements from baryonic acoustic oscillations (BAO). Some of the experiments considered have weak lensing parts too (SNAP, JEDI, ALPACA), but we do not derive dark energy constraints from simulated weak lensing measurements here.[^1] Note that all these experiments are presently undergoing optimization of their survey structure which may improve their science return.

For the SNe-Ia, we compared four different surveys. The CFHT SuperNovae Legacy Survey (SNLS) is already underway but we consider the full five-year survey, while the SuperNovae Acceleration Probe (SNAP) and the Joint Efficient Dark Energy Investigation (JEDI) satellite missions, plus the Advanced Liquid-mirror Probe for Astrophysics, Cosmology and Asteroids (ALPACA) ground-based survey, are all proposed experiments. For all experiments we assumed the same spread in magnitude $\delta_m=0.18$ of the supernovae, representing the combined effect of measurement error and intrinsic dispersion in the light-curve corrected luminosity \[the intrinsic dispersion alone was recently estimated as 0.12 mag by SNLS (Astier et al. 2006)\]. We used the number distribution of SNe-Ia with redshift for the different surveys as outlined in the literature; the total numbers used are 700, 2 000, 13 000 and 86 000 for SNLS (Pritchet et al. 2004; Astier et al. 2006), SNAP (Aldering et al. 2004), JEDI (Crotts et al. 2005) and ALPACA (Corasaniti et al. 2005) respectively. We also assumed all surveys would have support from an extra 300 nearby SNe-Ia observed by ground-based telescopes in the redshift range $0.03< z <0.08$, which also had a slightly smaller spread in magnitude ($\delta_m=0.15$). We assumed no systematic errors in any of the magnitudes, only statistical errors (except for one comparison case shown later).

For the baryonic acoustic oscillations we compared two different surveys, the ground-based Wide-Field Fibre-fed Multi-Object Spectrograph (WFMOS) and the satellite mission JEDI (JEDI will perform both a SN-Ia survey and a BAO survey). The BAO surveys measure both angular-diameter distance $D_{{\rm A}}(z)$ and the Hubble parameter $H(z)$ in a series of redshift bins. We calculated the expected errors of the measurements in each bin using the Fisher matrix approach of Seo & Eisenstein (2003), marginalizing over the physical matter density $\Omega_{{\rm m}} h^2$.

In order to obtain accurate results from experiments of these types, it is necessary that strong degeneracies with the matter density are removed by bringing in constraints from other sources. We make the assumption that by the time these surveys are operative, data compilations including Planck satellite observations will have provided a measurement of $\Omega_{{\rm m}}$ to an accuracy of $\pm
0.01$ (see for example Pogosian et al. 2005). We include such a measurement by adding an extra term to the likelihood centred around the fiducial density parameter value. In the absence of such external information, dark energy surveys would give a much poorer return. We will briefly explore the effect of varying this assumption in Section . Similarly in the BAO case we assume a 1% measurement uncertainty on $\Omega_{{\rm m}}h^2$ (see for example Tegmark et al. 2000).

## Priors

The model priors are the parameter ranges over which the evidence integral is carried out. Ordinarily in model selection these are supposed to be the wide priors seen as appropriate when the model was first considered, and not those motivated by current data. If one allows the model priors to 'follow the data' into a small region of parameter space, then model selection calculations will always be inconclusive in the long term, as this requires each new experiment to exclude a model again on its own, rather than the cumulative effect of all observations.[^2] The precise results for the Bayes factor will have some dependence on the choice of priors (see below), though the effect of the choices on model comparison or on survey comparison is diminished as the same priors are used for the common model parameters and the same priors are used for each survey being compared.

Our choices are as follows. We only consider flat Universes, so that $\Omega_\Lambda=1-\Omega_{{\rm m}}$. For the model priors, we impose the prior ranges $-2 < w_0 < -0.333$ and $-1.333 < w_a < 1.333$ on the interesting parameters, and $0<\Omega_{{\rm m}}<1$ and $0.5 < h < 0.9$ on the other parameters (the Hubble parameter is needed only for the baryon oscillation probes). The fiducial values for $\Omega_{{\rm m}}$ and $h$ are taken to be 0.27 and 0.7 respectively.

Note that for the phenomenological two-parameter evolving dark energy model, the model priors on $w_0$ and $w_a$ that we have chosen to work with are somewhat arbitrary. However if the prior space were reduced for instance by a factor of 2, that would increase the $\ln E$ of the evolving dark energy model by at most $\ln 2 \simeq 0.69$, and this would not significantly affect our contours or conclusions which are based on differences in $\ln E$ of 2.5 and 5. We make a brief investigation of some prior dependences in Section .

One should note that our conclusions also depend to some extent on our chosen dark energy parametrization being able to describe the true model. One could consider more general cases, such as the four-parameter models of Corasaniti & Copeland (2003) and Linder & Huterer (2005). For the purpose of assessing the power of an experimental proposal, it seems reasonable to presume that experiments capable of distinguishing two-parameter models are likely also to be better under other parametrizations. If the effect of dark energy were in fact a non-smooth variation in the equation of state and a non-smooth variation of the expansion history with redshift, then our results are too optimistic; the validity of reparametrizing the observables, which are the expansion history in different redshift bins as measured by the surveys, into ($w_0$,$w_a$) would need to be tested when the data arrive. Aspects of parametrization have been explored in Wang & Tegmark (2004) and Bassett et al. (2004).

# Results

## Comparison of calculational techniques

We begin by comparing our two methods of computing the Bayes factor, focussing on the SNAP mission supernova survey. The Bayes factor plots are shown in Figure . In the left panel we plot isocontours of Bayes factors in the $\hat{w}_0$–$\hat{w}_a$ plane inferred from the nested sampling method. The plot shows the generic structure expected of Bayes factor plots. In the central region, the simulated data are for models very close to $\Lambda$CDM, so that model gives a good fit and is further rewarded for its predictiveness, giving a positive Bayes factor which would support $\Lambda$CDM over the dark energy model. At the zero contour (the innermost one plotted) the models fare equally well, and then at greater distances the dark energy model becomes favoured. If the true parameters lie outside those contours, SNAP will be able to exclude $\Lambda$CDM at the probability corresponding to the contour level.

We see a strong degeneracy between the two parameters, meaning that supernova data are not good at constraining models in this particular parameter direction. This same degeneracy shows up in the usual Fisher matrix error projection method. Its precise direction depends on the redshift distribution of the supernovae.

In the right panel we plot the projected Bayes factor contours derived from the Savage–Dickey formula for the same experimental characteristics and priors assumed in the previous case. We see that this method gives generally good agreement with the nested sampling computation, indicating that our calculations are robust. Some slight differences are apparent, but this is expected as our version of the Savage–Dickey method employs a Gaussian approximation for the likelihood which may become poor at large distances from $\Lambda$CDM, with the Fisher matrix method underestimating the covariance matrix. For parameter estimation this is not a major concern, since deviations from the Gaussian approximation occur in the tail of the likelihood distribution, and quoted errors usually refer to the $68\%$ confidence intervals. However model selection calculations rely on good modelling well into the tails of the distribution.

Having verified that our methods give similar results, henceforth we will show results from the nested sampling method, since although it is computationally more intensive it does not assume a Gaussian likelihood.

## Comparison with parameter error forecasting

In this paper we are strongly advocating use of Bayes factor plots to quantify experimental capabilities, for the reasons enumerated in Section . It is useful to see explicitly what differences this gives as compared to the traditional parameter forecast approach, and so Figure  shows a plot of likelihood contours, obtained from a Markov chain Monte Carlo analysis of data simulated for the SNAP supernova survey, using precisely the same assumptions as Figure , and assuming that $\Lambda$CDM is the true model.

We see they share the same general shape, and that the same principal parameter degeneracy is picked out. Obviously the two plots are conceptually very different and so caution is needed in comparing. We see that the Bayes factor contours are significantly wider, indicating that model selection sets a more stringent condition for dark energy evolution to be supported by the data. Indeed, the 95% Fisher parameter contour lies within the $\ln B_{01}=0$ contour where model selection gives the models equal probability, hence by using the Fisher matrix plot we could rule out $\Lambda$CDM with data that actually favours it. It is fairly generic for that to be the case, indicating that 95% parameter estimation 'results' tend not to be robust under more sophisticated statistical analyses. This is a manifestation of Lindley's 'paradox' as discussed by Trotta (2005) — that parameter values rejected under a frequentist test can nevertheless be favoured by Bayesian model selection.

## Comparison of dark energy surveys

We now turn to a comparison of the six dark energy surveys described above. We stress once more that this comparison considers the statistical uncertainties alone, and several of these experiments are likely to be limited by systematics. The criteria that enable the systematics to be most effectively minimized are likely to be different from those giving experiments raw statistical power. Further we are working under the limitation of the particular $w_0$–$w_a$ parametrization; dark energy in reality could be different.

Figure  shows the six surveys, the upper four being supernova surveys and the lower two being the baryon acoustic oscillation surveys. The innermost contoured region is where the evidence of the $\Lambda$CDM model is greater than that of the evolving dark energy model ($\ln B_{01} > 0$). The outer contours show $\ln B_{01} =-2.5$ and $-5$ so that the data provides strong evidence in favour of the evolving dark energy model. As with parameter estimation contours, the smaller the contours the more powerful the experiment is.

As expected, we see a range of constraining powers depending on the scale of the experiments. We also see that they broadly share the same principal degeneracy direction, with slight rotations visible from the different probing of redshift bins. The massive size of the expected ALPACA dataset gives it the smallest contour area amongst SN-Ia experiments, with its more limited redshift range making its degeneracy more vertical.

The baryon oscillation probes share almost the same principal degeneracy as the SNe-Ia; although they use the angular-diameter distance rather than the luminosity distance these two are related by the reciprocity relation and hence follow the same degeneracy shape if the uncertainties in each redshift bin follow the same shape. A probe which partly included the growth of structure, such as weak lensing, would be expected to have a somewhat different degeneracy; this has been shown using Fisher parameter contours for the SNAP lensing survey though the rotation is still smaller than one would like.

Note that the logarithms of the Bayes factors are additive, so if more than one of these surveys happen, or if there are two independent parts to a survey, then their Bayes factor plots can be added together to give a net Bayes factor plot.

In addition to plotting Bayes factor contours, one can further compress the information on how powerful an experiment is by computing the area within a particular contour level, to give a single 'figure-of-merit'. Table  summarizes these areas, expressed in coordinate units, for the six experiments, showing the area where $\ln B_{01}$ exceeds $-2.5$. Note that this corresponds to the parameter area in which an experiment *cannot* strongly exclude $\Lambda$CDM, and hence small numbers are better. For a more extensive discussion of figures-of-merit for optimization of dark energy surveys, in a parameter estimation rather than model selection framework, see Bassett (2005) and Bassett, Parkinson & Nichol (2005).

| Experiment | Area | $\ln B_{01}(-1,0)$ |
|:-----------|:----:|:------------------:|
| SNLS       | 0.51 |        3.7         |
| SNAP       | 0.35 |        4.5         |
| JEDI SN    | 0.19 |        5.0         |
| ALPACA     | 0.08 |        6.1         |
| WFMOS      | 0.26 |        4.8         |
| JEDI BAO   | 0.04 |        6.0         |
|            |      |                    |

Two experimental figures-of-merit: the areas in the $\hat{w}_0$–$\hat{w}_a$ plane where $\ln B_{01}$ exceeds $-2.5$, and the value of $\ln B_{01}$ at $\hat{w}_0=-1$ and $\hat{w}_a=0$. The former measures the region of parameter space where the experiment would not be able to exclude the $\Lambda$CDM model, while the latter measures the strength with which the experiment would support $\Lambda$CDM were it the true model. The $\ln B_{01}$ are additive between surveys and for independent probes of dark energy within the same survey.

## Support for $\Lambda$CDM

We now consider the possibility of the experiments ruling out the dark energy model in favour of $\Lambda$CDM, rather than the opposite which we have focussed on thus far. Unlike parameter estimation methods, Bayesian model selection can offer positive support in favour of the simpler model. Because the simpler model is nested within the dark energy model, it can never fit the data better, but it can benefit from the volume effect of its smaller parameter space. All one needs to do is read off the Bayes factor for the case where the fiducial model is $\Lambda$CDM. Table  shows $\ln B_{01}$ at $\hat{w}_0=-1$ and $\hat{w}_a=0$, i.e. when $\Lambda$CDM is the true model.

We find that this value is above 2.5 for all surveys, and above 5 for several of them. Thus many of the surveys are capable of accumulating strong evidence supporting $\Lambda$CDM over evolving dark energy. This can be seen as another figure of merit quantifying the power of experiments. Note again that the $\ln B_{01}$ are additive between surveys and for independent probes of dark energy within the same survey.

Note that the absolute value of this figure of merit is more sensitive to the prior ranges chosen for the dark energy parameters, which set the volume factor. However the relative comparison of surveys is again not affected by this.

## Variation of assumptions

We end by examining the effect of varying some of the assumptions that went into the calculations, focussing on the SNAP supernova survey for definiteness. We do this in three ways, one by changing the presumed knowledge on $\Omega_{{\rm m}}$ that complements the dark energy survey, one by looking at an alternative prior in the dark energy model space, and finally by altering the assumed dispersion of supernova luminosities and allowing for a simple model of systematics.

As mentioned before, the return on dark energy surveys is quite sensitive to the availability of external constraints to remove parameter degeneracies, particularly $\Omega_{{\rm m}}$ in the case of the supernovae. Figure  shows this effect for the SNAP supernova survey, with different constraints on $\Omega_{{\rm m}}$ to be compared with our standard assumption leading to the left panel of Figure . One sees a significant worsening of the Bayes factor contours in the case of weaker knowledge on $\Omega_{{\rm m}}$.

Altering the constraint on $\Omega_{{\rm m}}$ can have a different effect on different experiments. For instance, if it were more stringent, then the difference between SNLS and SNAP or JEDI would be greater — the requirement for a more sensitive experiment would be greater. Similarly the relative comparison is dependent on the nature of dark energy itself; if a parametrization more complex than $w_0$–$w_a$ proved necessary, it would be more important to make precise high-redshift observations (e.g. SNAP or JEDI versus ALPACA).

Figure  modifies our assumptions in a different way, this time altering the prior on the dark energy parameters. It assumes a prior appropriate to quintessence models, namely that $w \ge -1$ at all redshifts. The evidence integral for the dark energy model is then carried out over a narrower region in the dark energy parameters, giving a boost to the evidence of the dark energy model relative to $\Lambda$CDM. However the effect is small; the dashed line shows where the outer contour lay with our full dark energy prior and it has shrunk in only marginally.

Caldwell & Linder (2005) classified quintessence models into freezing and thawing models and delineated areas of the $w_0$–$w_a$ space where those models typically lie. According to Figure , freezing models can only be decisively distinguished from $\Lambda$CDM by the SNAP supernova survey if $\hat{w}_0 \ga -0.9$, and thawing models if $\hat{w}_0 \ga -0.87$.

We have also investigated how changing the prior ranges on the dark energy parameters alters the areas given in Table . In this case we narrowed the priors on $w_0$ and $w_a$ by a factor of two in each direction. In cases where the posterior still lies within the priors, this shifts the evidence by $\ln4 \simeq 1.4$ in favour of the dark energy model. Unsurprisingly, we find this reduces the areas within which $\Lambda$CDM cannot be excluded, typically by 10 to 20 per cent. Importantly, however, this change preserves the rankings of the experiments.

Finally, in Figure  we examine how the outer contour would shift if a smaller magnitude dispersion were achieved (we take $\delta_m = 0.13$), and separately under a standard (but crude) modelling of possible systematics (see for example Kim et al. 2004). The systematics have been modelled as an increased redshift dependent uncertainty in magnitude of $(z/z_{\rm max}) \delta m_{\rm
sys}$ per redshift bin with $\delta m_{\rm sys} = 0.02$ mag, and added in quadrature to the (intrinsic) statistical uncertainty. For SNAP, this type of systematic has quite a small effect. There can be other types of systematics in the data, but we do not try to model them here as the ability of different experiments to detect and (internally) resolve systematics would be different and a proper study of systematics and the required marginalization over them can only be done once the data arrive.

# Conclusions

In this paper we have introduced the Bayes factor plot as a tool for assessing the power of upcoming experiments. It offers a full implementation of Bayesian model selection as a forecasting tool. As compared to the traditional parameter error forecasting technique, it offers a number of advantages enumerated in Section . Amongst those, perhaps the most important are that observational data is simulated at each point in the plane, rather than at a small number of fiducial models, and that the Bayes factor plot properly captures the experimental motivation as being one of model selection rather than parameter estimation.

As a specific example, we have used the Bayes factor plots to examine a number of proposed dark energy surveys, concentrating on their ability to distinguish between the $\Lambda$CDM model and a two-parameter dark energy model. Figure  indicates the region of parameter space outside which the true model has to lie, in order for the experiment to have sufficient statistical power to exclude $\Lambda$CDM using model selection statistics.

An important caveat is that our plots do not show the effects of systematics, which are likely to be the dominant uncertainty for many of the experiments. This drawback is shared by parameter error forecasts, and it is more or less the nature of systematic uncertainties that they cannot be usefully modelled in advance of actual observational data being obtained. In judging the true merit of an experimental proposal, it is therefore essential to judge how well structured it is for optimal removal of systematics, as well as looking at its raw statistical power.

While we have focussed on dark energy as a specific application, the concept of the Bayes factor plot has much broader applicability, and is suitable for deployment in a wide range of cosmological contexts.

# Acknowledgments

PM, DP and ARL were supported by PPARC (UK), PSC by Columbia Academic Quality Funds, and MK by the Swiss NSF. PM acknowledges Yun Wang for helpful discussions regarding the application of the Fisher matrix formalism to BAO observations. We thank Arlin Crotts for information concerning JEDI, and Roberto Trotta for helpful discussions and comments.

# Marginalizing over simulated data

In the main body of the paper, we have plotted the Bayes factor as a function of the fiducial values of the dark energy parameters, assuming particular values for the other parameters in the fiducial model. In general, however, the Bayes factor is a function of all the fiducial model parameters, not just the ones of principal interest, though a practical problem is that we cannot easily plot the evidence ratio $B_{01}$ as a function of more than two variables $\hat{\theta}$.

One solution is marginalization over the parameters that we are not interested in, as one does in parameter estimation, assuming those parameters to lie within the range motivated by present data. As the fiducial parameters $\hat{\theta}$ belong to the definition of the data, we need to marginalize the evidences, $P(D|M)$, rather than the Bayes factors. Formally, the marginalization must take place in data space, so when we wish to integrate out a 'nuisance' parameter $\hat{\theta}_\mu$ which the data is a function of, we should take into account a transformation factor $\sqrt{\sum_i (\partial
D_i/\partial\theta_\mu)^2}$, evaluated at each $\hat{\theta}$ along the integral. However, provided the evidence varies only weakly over the relevant range of the fiducial models, or if our model depends (nearly) linearly on its parameters, then this function is a constant which cancels when computing the Bayes factor. In this case we can just average the evidences. This will also conserve the relation $B_{01}=1/B_{10}$.

In practice, the main determining factor in whether particular extra parameters are justified by the data is the true values of those parameters themselves, rather than values of the other parameters. Often, then, one can choose fixed values of the uninteresting parameters, presenting results on a slice through the fiducial parameter space. Indeed, this turns out to be the case for the dark energy surveys in this paper.

# Parameter error forecasting

In this paper we are advocating the use of Bayes factors to quantify the power of upcoming experiments, in place of parameter error forecasts. For comparison, we provide a brief overview of parameter error forecasting here, and discuss some of its features and limitations.

The idea is to simulate a sample of experimental data and then infer the parameter uncertainties using standard likelihood analysis. More specifically, assuming a model $M$ specified by a set of parameters $\theta=\{\theta_\mu\}$, a sample of data $D$ with the expected experimental errors is generated for a particular fiducial model with parameter values $\hat{\theta}$. Then a likelihood $P(D|\theta,M)$ is computed and the confidence intervals on the $\theta$ parameters are inferred by computing a posterior parameter probability distribution via Bayes' rule, $$P(\theta|D,M) = \frac{P(D|\theta,M) P(\theta|M)}{P(D|M)} .$$ As a result the parameter uncertainties depend both on the experimental characteristics and the choice of the fiducial model.

A simplified way of carrying out such an analysis is to use the Fisher matrix approximation. By construction the fiducial model parameter values $\hat{\theta}$ maximize the likelihood. Hence expanding $\ln{P(D|\theta,M)}$ to quadratic order in $\delta{\theta}\equiv\theta-\hat{\theta}$ one obtains (Bond 1995; Tegmark, Taylor & Heavens 1997) $$P(D|\theta,M)\sim
\exp{\left(-\frac{1}{2}\sum_{\mu\nu}F_{\mu\nu}\delta{\theta_\mu}\delta
{\theta_\nu}\right)},\label{like}$$ which is a Gaussian approximation to the likelihood with zero mean and with variance given by the inverse of the Fisher matrix $F_{\mu\nu}$, where $$F_{\mu\nu}=\sum_{ij} C_{ij}^{-1}\frac{\partial D_i}{\partial\theta_\mu}
\frac{\partial D_j}{\partial\theta_\nu}\,.\label{fish}$$ The sum is over all measurements and the partial derivatives are evaluated at the fiducial model parameter values $\hat{\theta}_\mu$. The matrix $C_{ij}$ is the data covariance matrix; for independent measurements (e.g. different supernovae) it simplifies to $\sigma^2(D_i) \delta_{ij}$. The parameter errors are then given by the square root of the diagonal components of the covariance matrix, $\sigma(\theta_\mu)=\sqrt{(F^{-1})_{\mu\mu}}$. It is evident from equation () that more accurate data, characterized by smaller uncertainty $\sigma(D)$, provide larger Fisher matrix components, hence smaller parameter errors. It can also be noticed that for a given experiment the parameters which are better constrained are those for which the partial derivatives are larger.

Since these derivatives are computed at the fiducial model, it is natural to expect that the size of the projected errors varies for different fiducial parameter values. These contours are usually plotted with the aim of drawing a conclusion based on the true model having different parameter values from those of the fiducial model. But the dependence on the choice of fiducial model means that there is no guarantee that the conclusions based on contours around e.g. the $\Lambda$CDM model can be used to rule that model out.

As an explicit example we compute the Fisher matrix errors of dark energy parameters from SN-Ia luminosity–distance measurements. We assume experimental characteristics from the proposed SNAP mission as discussed in Kim et al. (2004). We consider two different dark energy models, one parametrized by a constant equation of state parameter $w$, and a second by the two-parameter equation of state family of equation (). We assume an independent measurement of $\Omega_{{\rm m}}$ with uncertainty $\pm 0.03$ to reduce parameter degeneracies, and compute the marginalized confidence contours around different fiducial models in $\Omega_{{\rm m}}$–$w$ and $w_0$–$w_a$ planes respectively.

In Figure  we plot the $68\%$ and $95\%$ ellipses around three models: a $\Lambda$CDM model with $w=-1$ and $\Omega_{{\rm
m}}=0.3$, a dark energy model with $w=-0.88$ and $\Omega_{{\rm
m}}=0.35$, and a phantom model with $w=-1.16$ and $\Omega_{{\rm
m}}=0.26$. The alignment of the strongest degeneracy line differs amongst the models. This is because the degeneracy in the $w-\Omega_{{\rm m}}$ plane is not a straight line, but rather a curve (see for instance Weller & Albrecht 2001). Notice also that the ellipses around the fiducial models have different sizes. As Fig.  shows, if the true model lies on say the 95% confidence limit of the $\Lambda$CDM data, one cannot necessarily presume that the $\Lambda$CDM model would lie on the 95% confidence limit of data simulated for the true model. It is possible to compute a contour indicating the locus of the fiducial models for which $\Lambda$CDM lies at their 95% confidence limit, and indeed such a locus is shown in Figure 1 of Kratochvil et al. (2004), but constructing it is a rather cumbersome procedure.

This drawback turns out to be less severe for dark energy models parametrized by equation (). In Figure  we plot the $68\%$ and $95\%$ ellipses in the $w_0$–$w_a$ plane with $\Omega_{{\rm m}}=0.3$ around a $\Lambda$CDM model, a phantom model with $w_0=-0.8$ and $w_a=-1$ lying along the degeneracy line of the $\Lambda$CDM, and a constant phantom model with $w_0=-1.4$ and $w_a=-0.2$. The dependence on the fiducial model is still present, since the ellipses become larger as the fiducial model shifts orthogonal to the principal degeneracy direction towards more negative equation of state values. Fiducial models along the same degeneracy line whose $95\%$ contours include the $\Lambda$CDM model are within the $95\%$ ellipse of the $\Lambda$CDM as well.

[^1]: Both SN-Ia and BAO are distance indicators, while weak lensing is sensitive to growth and dark energy perturbations. Complementarity of weak lensing with SN-Ia/BAO will thus be very interesting in probing dark energy more comprehensively.

[^2]: An alternative, equivalent, view more in the Bayesian spirit is that one can update the model prior ranges after new data, provided one also updates the model probabilities and keeps track of them as well. In practice, cosmological data analysis tends to re-apply a broad set of data to models with wide priors each time, which is consistent with the model selection philosophy.