author: William P. Thurston
title: Mathematical Education

This article originally appeared in the *Notices of the AMS* **37** (1990), 844–850. Copyright 1990 William P. Thurston.

Mathematics education is in an unacceptable state. Despite much popular attention to this fact, real change is slow.

Policymakers often do not comprehend the nature of mathematics or of mathematics education. The 'reforms' being implemented in different school systems are often in opposite directions. This phenomenon is a sign that what we need is a better understanding of the problems, not just the recognition that they exist and that they are important.

I am optimistic that our nation will find solutions to these problems. Problems arising from failure of understanding are curable. We do not lack for dedication, resources, or intelligence: we lack direction.

# Symptoms

There are many symptoms of the problems in mathematics education. The number of undergraduate mathematics majors is about half what it was 15 years ago. The number of U.S. graduate students is less than half what it was 15 years ago, although foreign students have taken up some of the slack. The performance of our students at all levels, as measured by standardized tests, is below that of other industrialized countries.

The typical response of American adults, on meeting a mathematician, is one of dismay. They apologetically recall the last mathematics course they took, which is usually the one where they lost their grip on the subject matter.

Many companies recognize lack of mathematical competence as a major problem in their workforce. Technology has removed the need for elementary mathematical facility in a few jobs, such as ringing up hamburger orders, but it has also eliminated many unskilled jobs, such as assembly-line work, while creating many others requiring considerable mathematical sophistication.

Students in college mathematics courses are unresponsive. They are afraid to speculate and afraid to reach into themselves for ideas. When one visits classrooms at different grade levels, one sees a dramatic decline in liveliness and spontaneity with age. One gets the impression that the natural interest and curiosity of young children in mathematics has been weeded out.

In most places, it is harder to get students to do homework or to study outside the classroom than it was twenty years ago.

Even those college students who have been successful in the high school mathematics curriculum, including calculus, have a narrow base of knowledge of mathematics.

#  Stratification and Compartmentalization

A major source of problems, and a major barrier to solutions of the problems, is the stratification and compartmentalization of the immense mathematical education enterprise, from kindergarten through graduate school. In particular, there is very little communication between high school and college teachers of mathematics. There is also less than optimal communication between the people who are involved with curricular reform at the college level and research mathematicians. This splintering partly results from the existence of several different professional associations, including the American Mathematical Society (AMS), which primarily represents mathematical research; the Mathematical Association of America (MAA), which primarily represents undergraduate mathematics; and the National Council of Teachers of Mathematics (NCTM), which primarily represents the most committed high school mathematics teachers. Additional organizations represent two-year colleges, applied mathematics, statistics, and operations research, as well as computer science.

The membership of the AMS and the MAA overlap by only about $1/3$. Most members of the AMS are not even aware of the existence of the NCTM. Conversely, few members of the NCTM are aware of the AMS. Very few academic mathematicians have any contact with school mathematics teachers except through their children. Most teachers are trained at institutions with few if any research mathematicians. A number of universities are strong both in mathematical research and teacher training, but, even at those universities, most research mathematicians are not involved in teacher training.

Even more severe is the compartmentalization due to the division of primary and secondary education into many individual school districts and schools where the real decisions are made, and the division of college education into many different academic departments where the real decisions are made. Many people are thinking about the problems and making individual efforts to solve them. Many of these local efforts are quite successful. Unfortunately, there is far too little coordination and sharing of the insights gained.

#  Mathematics is a Tall Subject

One feature of mathematics which requires special care in education is its 'height,' that is, the extent to which concepts build on previous concepts. Reasoning in mathematics can be very clear and certain, and, once a principle is established, it can be relied on. This means it is possible to build conceptual structures which are at once very tall, very reliable, and extremely powerful.

The structure is not like a tree, but more like a scaffolding, with many interconnected supports. Once the scaffolding is solidly in place, it is not hard to build it higher, but it is impossible to build a layer before previous layers are in place.

Difficulties arise because students taking a particular course are in different stages of mastery of the earlier learning. They also tend to be secretive about exactly what they know and what they don't know. For instance, many calculus students don't correctly add fractions, at least not in symbolic form: the typical mistake is that $$\frac ab + \frac cd = \frac{a + c}{b + d}$$ much simpler than $$\frac{a d + c b }{b d}$$ However, students feel guilty that they are shaky on addition of fractions and are slow to admit it and ask how and why it works, even to themselves.

Addition of fractions is a very boring topic to someone who already knows it, but it is an essential skill for algebra, which in turn is essential for calculus. It is not so hard, when talking with students individually, to find out what parts of the structure need shoring up and to deal with those parts individually. But it is quite difficult to find a level of teaching which is comprehensible and at the same time interesting to an entire class with heterogeneous background.

#  Mathematics is a Broad Subject

Mathematics is also very broad. There are many subjects that are never discussed in the mainline curriculum which culminates in calculus. The subjects that are discussed have many interesting side-branches that are never explored.

In my parents' generation (during the 1940s), the standard first college mathematics course was college algebra. Soon afterward, the standard first college course was calculus, until the early 1960s, when calculus became standard for the best high school mathematics students. By now first year calculus has largely migrated to high school in affluent school districts, so that most of the better mathematics and science students at our best universities have already taken calculus before they arrive. At Princeton, for instance, two-thirds of entering students placed out of at least one semester of calculus last year.

The acceleration of the curriculum has had its cost: there has been an accompanying trend to prune away side topics. When I was in high school, for instance, it was standard to study solid geometry and spherical geometry along with plane geometry. These topics have long been abandoned. The shape of the mathematics education of a typical student is tall and spindly. It reaches a certain height above which its base can support no more growth, and there it halts or fails.

These two trends (lengthening and narrowing of school mathematics) have been hastened by the growing reliance on standardized tests. Standardized tests are designed to cover topics on the most standard curriculum: if only half the students study some topic, it is unfair to ask about it on a standardized test. This is not so bad as long as tests are used in a disinterested way as one of several *devices for measurement*. Instead, higher test scores are often treated as the *goal*. Legislators, newspapers and parents put superintendents and school boards under pressure, superintendents and school boards put principals under pressure, principals put teachers under pressure; and teachers put students under pressure to raise scores. The sad result is that many mathematics courses are specifically designed to raise scores on some standardized test.

We don't diagnose pneumonia with only a thermometer, and we don't attempt to cure it by putting ice in a patient's mouth. We should take a similarly enlightened attitude toward testing in mathematics education.

The long-range objectives of mathematics education would be better served if the tall shape of mathematics were de-emphasized, by moving away from a standard sequence to a more diversified curriculum with more topics that start closer to the ground. There have been some trends in this direction, such as courses in finite mathematics and in probability, but there is room for much more.

#  Mathematics is Intuitive and Real

Students commonly lose touch with the reality and the intuitive nature of mathematics. From kindergarten through high school, they often have teachers who are uncomfortable with anything off the beaten path. Young children come up with many ingenious devices to work out mathematical questions, but teachers usually discourage any nonconventional approach—partly because it is not easy to understand what a child is thinking or trying to say, and the teachers don't catch on, partly because the teachers think it's not okay to use an alternative method or explanation.

By the time students are in college, they are inhibited from thinking for themselves and from admitting out loud what they are thinking. Instead, they try to figure out what routines they are supposed to learn. When there is any departure in class from the syllabus or the text, someone invariably asks whether it's going to be on the test.

Unless mathematics makes a real connection to people, they are unlikely ever to think about it or use it once they have completed a course.

#  Precocity and Competition

Along with the emphasis on tests has come an emphasis on precocity and acceleration in mathematics. It is relatively easy for a bright student to work through the mathematics curriculum far more quickly than the usual pace.

There are several problems associated with precocity. People who skip ahead in the curriculum often have gaps in their background which only show up later. At that point, the person may be too embarrassed to admit the gap and tries to fake understanding. This regularly leads to disastrous results.

Another problem is that precocious students get the idea that the reward is in being 'ahead' of others in the same age group, rather than in the quality of learning and thinking. With a lifetime to learn, this is a shortsighted attitude. By the time they are $25$ or $30$, they are judged not by precociousness but on the quality of work. It is often a big letdown to precocious students when others who are talented but not so precocious catch up, and they become one among many. The problem is compounded by parents in affluent school districts who often push their children to advance as quickly as possible through the curriculum, before they are really ready.

A third problem associated with precociousness is the social problem. Younger students are often well able to handle mathematics classes intellectually without being able to fit in socially with the group of students taking them.

Related to precociousness is the popular tendency to think of mathematics as a race or as an athletic competition. There are widespread high school math leagues: teams from regional high schools meet periodically and are given several problems, with an hour or so to solve them.

There are also state, national and international competitions. These competitions are fun, interesting, and educationally effective for the people who are successful in them. But they also have a downside. The competitions reinforce the notion that either you 'have good math genes', or you do not. They put an emphasis on being quick, at the expense of being deep and thoughtful. They emphasize questions which are puzzles with some hidden trick, rather than more realistic problems where a systematic and persistent approach is important.

This discourages many people who are not as quick or as practiced, but might be good at working through problems when they have the time to think through them. Some of the best performers on the contests do become good mathematicians, but there are also many top mathematicians who were not so good on contest math. Quickness is helpful in mathematics, but it is only one of the qualities which is helpful. For people who do not become mathematicians, the skills of contest math are probably even less relevant.

These contests are a bit like spelling bees. There is some connection between good spelling and good writing, but the winner of the state spelling bee does not necessarily have the talent to become a good writer, and some fine writers are not good spellers. If there was a popular confusion between good spelling and good writing, many potential writers would be unnecessarily discouraged.

I think the answer to these problems is to build a system which exploits the breadth of mathematics, by allowing quicker students to work through the material in greater depth and to take excursions into related topics, before racing ahead of their age group.

#  Mystery and Mastery

Mathematics is amazingly compressible: you may struggle a long time, step by step, to work through some process or idea from several approaches. But once you really understand it and have the mental perspective to see it as a whole, there is often a tremendous mental compression. You can file it away, recall it quickly and completely when you need it, and use it as just one step in some other mental process. The insight that goes with this compression is one of the real joys of mathematics.

After mastering mathematical concepts, even after great effort, it becomes very hard to put oneself back in the frame of mind of someone to whom they are mysterious.

I remember as a child, in fifth grade, coming to the amazing (to me) realization that the answer to $134$ divided by $29$ is $134/29$ (and so forth). What a tremendous labor-saving device! To me, '134 divided by 29' meant a certain tedious chore, while $134/29$ was an object with no implicit work. I went excitedly to my father to explain my major discovery. He told me that of course this is so, $a/b$ and $a$ divided by $b$ are just synonyms. To him it was just a small variation in notation.

One of my students wrote about visiting an elementary school and being asked to tutor a child in subtracting fractions. He was startled and sobered to see how much is involved in learning this skill for the first time, a skill which had condensed to a triviality in his mind.

Mathematics is full of this kind of thing, on all levels. It never stops.

The hard-earned and powerful tools which are available almost unconsciously to mathematicians, but not to students, make it hard for mathematicians to learn from their students. This puts a psychological barrier in the way of listening fully to students.

It is important in teaching mathematics to work hard to overcome this barrier and to get out of the way enough to give students the chance to work things out for themselves.

#  Competence and Intimidation

Similarly, students at more advanced levels know many things which less advanced students don't yet know. It is very intimidating to hear others casually toss around words and phrases as if any educated person should know them, when you haven't the foggiest idea what they're talking about. Less advanced students have trouble realizing that they will (or would) also learn these theories and their associated vocabulary readily when the time comes and afterwards use them casually and naturally. I remember many occasions when I felt intimidated by mathematical words and concepts before I understood them: negative, decimal, long division, infinity, algebra, variable, equation, calculus, integration, differentiation, manifold, vector, tensor, sheaf, spectrum, *etc.* It took me a long time before I caught on to the pattern and developed some immunity.

Teachers also are frequently intimidated about mathematics. High school teachers are often timid about approaching college and university teachers. They also question, with some justice, whether university professors are in touch with the problems they have to deal with. There is so little general contact between those who teach mathematics in high schools, colleges, and universities that few professors know much about the educational problems in high school or elementary school. Elementary school teachers are often quite insecure about their grasp of mathematics and timid about approaching anyone.

#  The Problems and Solutions are the Same

There is much in common about the problems—and the solutions—of mathematics education at different levels, from kindergarten through graduate school. The failure to communicate is a real loss, and the potential gain from opening of two-way communications is great.

Over the past two years, I have met many people involved in mathematical education at all levels, partly as a member of the MSEB (Mathematical Sciences Education Board), a national board for mathematics education, composed of people from diverse backgrounds in mathematics and education, including a number of teachers. I have learned a lot.

During the spring semester of 1990, John Conway, Peter Doyle, and I organized a new course ('Geometry and the Imagination') at Princeton which borrowed a good deal from the ideas I learned. We taught as a team, shunning lectures and emphasizing group discussions among students. We emphasized manipulables, cooperative learning, and problem solving. We asked students to keep journals and to write out their ideas in good and complete English. Each student did a major project for the course. These ideas are all borrowed from ideas current in K-12 mathematics education, focused in the NCTM curriculum standards.

To culminate, we held a 'geometry fair': like a science fair but with a popcorn machine and without prizes. It was great fun.

The course came alive, qualitatively more than any course we had taught before. Students learned a lot of mathematics and solved problems we wouldn't have dared ask in a conventional college class.

One topic we discussed was mathematical education. Students were given an earlier draft of this essay and wrote 70 thoughtful essays based on their own experiences. This essay has benefitted considerably from their comments.

#  Socially Excluded Groups

Why do so few women become mathematicians, and so few of the non-Asian minorities?

I am convinced that the poverty of the school mathematics teaching and curriculum has a lot to do with it. The way mathematics is taught in school does not address the real goals of a mathematical education. It is very hard to get a sense of the depth, liveliness, power, and breadth of mathematics from any ordinary experience with mathematics in school. I believe that most students who really master the subject matter, and eventually become scientists, mathematicians, computer programmers, *etc.*, are those who have some other channel for learning mathematics, outside the classroom. Sometimes it is the home, sometimes it is books, sometimes it is an unusual teacher, but often it is the 'nerd' social subgroup in school. When I was in high school, I certainly belonged to this subgroup (although the name 'nerd' was not yet current), and I appreciated it very much. However, it is a very different matter for white and Asian males to join this social subgroup than for women, Hispanics or blacks.

Since channels of learning outside the classroom are currently dominant in mathematics education for the top group of students, improvement of the general quality of mathematics within the classroom therefore should act as an equalizer, particularly helping blacks, Hispanics, and women. For people who are not already tuned in to the mathematical style of discourse, it is especially important to teach in a way that is not watered down, but that begins from a person's real experience.

Intimidation also has a lot to do with it. The emphasis on precocity, high test scores, and competition work to amplify the small differences which arise from other sources.

#  Goals and Standardized Tests

What is mathematics education good for?

##  Mathematics in life

First, mathematics is a basic tool of everyday life. When coffee comes in $13$ oz. packages and $16$ oz. cans, can you take that information in stride (walking slowly by the shelf), along with the prices and the prominent signs claiming 'contains 23% more' on the cans, to help decide which you'd prefer to buy? In buying a new car with various gimmicks in financing, rebates, and features, do you understand what is going on? If most people did, the gimmicks would be pointless.

Second, mathematical reasoning is an important part of informed citizenship. Can you understand the reasoning behind studies of health risks from various substances, and can you judge how important they are? When listening to politicians, can you and do you use your reckoning powers to help decide how important some statistic is, and what it means? Can you measure and calculate adequately for simple sewing and carpentry? Can you plan a budget? When you see graphs in newspapers and magazines, do you understand what they mean, and are you aware of the several devices frequently used either to dramatize or to play down a certain trend?

Third, mathematics is a tool needed for many jobs in the infrastructure of our increasingly complex and technological society. These uses are pervasive and varied. The dental technician, the fax repairperson, the fast food manager, the real estate agent, the computer consultant, the bookkeeper and the banker, the nurse and the lawyer, all need a certain proficiency with mathematics in their jobs.

Fourth, mathematics is intensively used (and sometimes abused) in most branches of science. Much of theoretical science really *is* mathematics. Statistics is one of the most common uses of mathematics. Many scientists use the widespread computerized statistical packages, which alleviate the need for computation. However, people who use statistical packages are often shaky in their understanding of the basic principles involved and often apply statistical tests or graphical displays inappropriately.

##  Mathematics is alive

To me, these utilitarian goals are important, but secondary. Mathematics has a remarkable beauty, power, and coherence, more than we could have ever expected. It is always changing, as we turn new corners and discover new delights and unexpected connections with old familiar grounds. The changes are rapid, because of the solidity of the kind of reasoning involved in mathematics.

Mathematics is like a flight of fancy, but one in which the fanciful turns out to be real and to have been present all along. Doing mathematics has the feel of fanciful invention, but it is really a process of sharpening our perception so that we discover patterns that are everywhere around. In his famous apology for mathematics, G.H. Hardy praised number theory for its purity, its abstraction, and the self-evident impossibility of ever putting it to practical use. Now this very subject is applied very widely, particularly for encoding and decoding communications.

My experience as a mathematician has convinced me that the aesthetic goals and the utilitarian goals for mathematics turn out, in the end, to be quite close. Our aesthetic instincts draw us to mathematics of a certain depth and connectivity. The very depth and beauty of the patterns makes them likely to be manifested, in unexpected ways, in other parts of mathematics, science, and the world.

To share in the delight and the intellectual experience of mathematics—to fly where before we walked—that is the goal of a mathematical education.

##  Testing and "accountability"

Unfortunately the goals of school mathematics have become incredibly narrow, much narrower even than the first set of utilitarian goals listed above, let alone the others. It is popular lately for politicians and the public to demand "accountability" from the school systems. This would be great, except that educational accounting is usually based on narrowly-focused multiple-choice tests.

It is as if students were considered to have mastered Shakespeare if they could pass a timed vocabulary test in Elizabethan English, or that they had learned to write when they could correctly choose the grammatical form of a sentence from four possibilities.

The state and regional boards of education these days hand out a laundry list of skills which students are supposed to know at a certain age, rather than a curriculum: horizontal addition versus vertical addition, addition of 2 digit numbers to 2 digit numbers with a 2 digit answer versus addition of 2 digit numbers to 2 digit numbers with a 3 digit answer, *etc.*

A front-page article in the New York Times Metropolitan section of July 24, 1990 contrasted the elementary schools in two similar difficult neighborhoods of Brooklyn. The first was a 'successful' school, with two reading lessons a day, the second lesson in 'test-taking skills' and practice for the standardized reading test. In this school, 80.5% scored at or above grade level. The other school was an 'unsuccessful' school where they prepare for the test for 'only' 3 months. In that school, 36.4% score at or above grade level.

The reporter cited an example of how the principal sets the tone in the 'successful' school:

> She is not satisfied with just the right answers; she wants the right steps along the way. In one fourth-grade class, she noticed that pigtailed Keanda Snagg had made a wild, though accurate, stab at a problem requiring her to average which of two stores had lower prices.
>
> "It looks like you were going to do it without doing the work," she told Keanda.
>
> As she watched Keanda go through the calculations, she stressed fundamentals beyond arithmetic, like putting numbers in clearly ordered columns.

I can't tell which school is actually more successful without seeing them for myself, but one thing I know: neither the test scores nor the cited incident are demonstrations of greater success.

#  Thinking and Rote

Narrow goals are stultifying.

People are much smarter when they can use their full intellect and when they can relate what they are learning to situations or phenomena which are real to them.

The natural reaction, when someone is having trouble understanding what you are explaining, is to break up the explanation into smaller pieces and explain the pieces one by one. This tends not to work, so you back up even further and fill in even more details.

But human minds do not work like computers: it is harder, not easier, to understand something broken down into all the precise little rules than to grasp it as a whole. It is very hard for a person to read a computer assembly language program and figure out what it is about. A computer reads and executes it in the blink of an eye. But the most powerful computer in the world is not clever enough to drive a car safely, or control a stroll along the sidewalk, or come up with an interesting mathematical discovery.

Studying mathematics one rule at a time is like studying a language by first memorizing the vocabulary and the detailed linguistic rules, then building phrases and sentences, and only afterwards learning to read, write, and converse. Native speakers of a language are not aware of the linguistic rules: they assimilate the language by focusing on a higher level, and absorbing the rules and patterns subconsciously. The rules and patterns are much harder for people to learn explicitly than is the language itself. In fact, the tremendous and so far unsuccessful attempts to teach languages to computers demonstrate that nobody can yet describe a language adequately by precise rules.

It is better not to teach a topic at all than to attempt teaching it in tiny rules and bits.

#  Answers and Questions

People appreciate and catch on to a mathematical theory much better after they have first grappled for themselves with the questions the theory is designed to answer.

There is a natural tendency, in teaching mathematics, to use the logical order and to explain all the techniques and answers before bringing up the examples and the questions, on the supposition that the students will be equipped with all the techniques necessary to answer them when they arise.

It is better to keep interesting unanswered questions and unexplained examples in the air, whether or not the students, the teachers, or anybody is yet ready to answer them. The best psychological order for a subject in mathematics is often quite different from the most efficient logical order.

As mathematicians, we know that there will never be an end to unanswered questions. In contrast, students generally perceive mathematics as something which is already cut and dried—they have just not gotten very far in digesting it.

We should present mathematics to our students in a way which is at once more interesting and more like the real situations where students will encounter it in their lives—with no guaranteed answer.

#  What Can We Do?

This depends on who we are.

In our compartmentalized system, it is hard for a single organization or individual to do very much to affect the overall system directly. But addressing the local situation will indirectly influence the global situation. I will address the question from the point of view of college and university mathematicians.

First, college and university mathematics departments should develop courses which can give students a fresh chance in mathematics. Remedial courses are widespread, but their success is limited: going over the same material one more time is tedious and boring, whether you understood it the first time or you didn't. There is a built in handicap to enthusiasm and spontaneity.

Instead, there should be more courses available to freshmen and nonmajors which exploit some of the breadth of mathematics, to permit starting near the ground level without a lot of repetition of topics that students have already heard. For instance, elementary courses in topology, number theory, symmetry and group theory, probability, finite mathematics, algebraic geometry, dynamical systems (chaos), computer graphics and linear algebra, projective geometry and perspective drawing, hyperbolic geometry, and mathematical logic can meet this criterion.

Second, we should work to create better channels of communication between the compartments of the educational system. We need to find devices so that the educational accomplishments of professors are visible within the profession, not merely within the classroom or within the department. We need to find vehicles for exchange of interesting ideas between different departments: for instance, exchange visits between directors of undergraduate studies and chairs of departments. We should visit each other's classrooms. We need more talks and special sessions related to education at our professional meetings, and more prizes for educational accomplishments.

The newsletter, *Undergraduate Mathematics Education Trends*, and the MAA newsletter, *Focus*, are two such vehicles, and there are several other publications which carry articles on undergraduate education. The annual chairman's coloquium organized by the Board on Mathematical Sciences of the National Research Council is another vehicle of communication, although it has a broader agenda than education. Still, these existing channels are very small compared with what we could establish.

Even more important and more difficult is the creation of channels for communication between the strata: most important for colleges and universities is communication between high school, college and university mathematics departments. This communication must be two-way: college and university professors can learn a lot about how to teach from school teachers. The MSEB is one such channel, along with the state mathematics coalitions that they have helped to stimulate, but what we need is a much more massive exchange.

How can the senior professors, who are at the top of a system which is clearly not doing such a great job, presume to teach their juniors how to do better? The graduate students and the junior faculty often do a better job at teaching mathematics than the senior faculty, who have sometimes become resigned to the dismal situation, settled into a routine, and given up on trying any new initiatives. Even when they do a pretty good job in their own classrooms, against the odds, they do not usually get involved in improving the overall system.

Often other professors are suspicious of the professor who does take an interest in education. They tend to assume that research is the only activity which really matters and that turning to education is a sign of failure in research. Senior professors sometimes explicitly advise junior faculty not to waste too much energy on teaching, or they will never be promoted.

We must recognize that there are many different ways that we can make important contributions to society and to our institutions. It is dumb to measure mathematicians against the single scale of research. Education is an important and challenging endeavor, which many people engage in by choice, not necessity. We should judge them by what they accomplish, not by what they might have accomplished if they spent their time and energy elsewhere.

What urgently needs to change is the system of professional rewards. We need something better than the current situation within university mathematics departments where there is lip service to the importance of teaching, but, when it comes to the crunch of hiring and tenure decisions, teaching and service count only in the marginal cases where the candidates cannot be differentiated by the quality of research.

People are socially motivated. As we discuss education with each other, we put more energy into it, and it becomes more important to us. The academic culture *can* change, and it has changed. The process of change is mostly an informal one (what you talk about at lunch), not controlled by organizational decisions. But when the time is ripe, as I believe it is now for mathematical education, a little nudging by organizations can help stimulate a huge change.

The needed reforms will take place through collegial, cooperative efforts. Good mathematical ideas spread very rapidly through informal channels in the mathematical community. As we turn more of our attention to education, good educational ideas will also spread rapidly.


Inference of ancestral recombination graphs through topological data analysis
P. G. Cámara${}^{1,2}$, A. J. Levine${}^2$ and R. Rabadán${}^1$
${}^1$Department of Systems Biology and Department of Biomedical Informatics,
Columbia University College of Physicians and Surgeons,
1130 St. Nicholas Ave., New York
${}^2$The Simons Center for Systems Biology,
Institute for Advanced Study, Princeton, NJ 08540.
**Abstract**

The recent explosion of genomic data has underscored the need for interpretable and comprehensive analyses that can capture complex phylogenetic relationships within and across species. Recombination, reassortment and horizontal gene transfer constitute examples of pervasive biological phenomena that cannot be captured by tree-like representations. Starting from hundreds of genomes, we are interested in the reconstruction of potential evolutionary histories leading to the observed data. Ancestral recombination graphs represent potential histories that explicitly accommodate recombination and mutation events across orthologous genomes. However, they are computationally costly to reconstruct, usually being infeasible for more than few tens of genomes. Recently, Topological Data Analysis (TDA) methods have been proposed as robust and scalable methods that can capture the genetic scale and frequency of recombination. We build upon previous TDA developments for detecting and quantifying recombination, and present a novel framework that can be applied to hundreds of genomes and can be interpreted in terms of minimal histories of mutation and recombination events, quantifying the scales and identifying the genomic locations of recombinations. We implement this framework in a software package, called TARGet, and apply it to several examples, including small migration between different populations, human recombination, and horizontal evolution in finches inhabiting the Galápagos Islands.

# Introduction

Since the publication of the first draft of the human genome , there has been an explosion in genomic data. The genomes of thousands of different human individuals have been sequenced , several hundreds of eukaryotic genomes have been characterized, and new viral, bacterial and archaeal species are being sequenced on an almost daily basis . Darwin provided a historical dimension to the taxonomical enterprise, proposing that closely related species in the hierarchical taxonomy share ancestors. Since then, tree-like structures have been proposed to represent the evolutionary/historical relationship between organisms. In the last few years, however, the richer and more comprehensive genomic characterization of many organisms have underscored the need of representations that are not strictly tree-like. Phenomena such as horizontal gene transfer in bacteria , the ability of viruses to borrow and lend genes across species, and hybridization in metazoa (in plants, in particular ) are exposing some of the limitations imposed by tree-like phylogenetic structures. The definition of species itself becomes cumbersome in bacteria and viruses . Within many species, including humans, genetic recombination is so pervasive that tree-like representations are useless. It is then natural to wonder what other frameworks could be used to capture phylogenetic relationships without losing the interpretability and simplicity of trees . Of particular interest are representations that reduce to trees when evolution is tree-like; that capture genetic relations between ancestors, and identify genomic regions originating from different ancestral lineages; and, more generally, that allow for an interpretation of the observed data in terms of a chronological sequence of events.

Several such frameworks have been proposed in the last two decades. The study of phylogenetic networks has been an area particularly active . Phylogenetic networks provide representations that extend trees to graphs (networks), generating loops when the data does not fit into a tree. Some of those methods can easily be applied to more than one hundred genomes providing the opportunity for large-scale representations. However, the biological interpretation of these representations is limited, as loops represent inconsistencies with trees, but it is unclear how these inconsistencies arose historically, what genomic regions were involved, or how frequently an exchange happened. Other types of representations, sometimes named *explicit networks* , do aim to provide a historical account in terms of a chronology of events. Ancestral recombination graphs (ARGs) provide potential explanations of the observed data in terms of a progression of recombination and mutation events. As in trees, mutations are represented as events along the branches. Recombinations, however, appear as the fusion of two parental branches into one offspring branch. ARGs provide simple histories that can be used in association mapping , SNP genotyping or inference of the frequency and scale of recombination . However, these applications are hindered by the computational infeasibility of constructing ARGs that explain hundreds of sequences. The construction of minimal ARGs, containing the minimum number of recombination events required to explain the sample in absence of convergent evolution and back-mutation, is an NP-hard problem . Several approximations have been developed in the last few years, including galled trees , branch and bound , heuristic and sequentially Markov coalescent approaches .

Recently, a new framework to study genomic relationships has been proposed , based on topological data analysis . Topology is the area of mathematics that aims to characterize properties of spaces up to continuous deformations, for instance the number of disconnected components, loops and holes of a space. TDA extends the concepts and tools of topology to finite metric spaces, that is, finite sets of points and distances between them. Taking the premise that a set of points has been sampled from an unknown underlying space, TDA attempts to infer the topological features of the space (Fig. A). Stability results guarantee that small fluctuations in the data only create small changes in the inferred topological features, providing robust characterizations of the data.

In a TDA framework, genomes are characterized by points in a high dimensional space where pairwise distances are genetic distances between sequences. Assuming that each genomic site mutates at most once across the evolutionary history of the sample, the genetic distance between two genomes can only increase with the acquisition of novel mutations. The only way of "closing" a loop (a close path) in this space is therefore by means of a recombination event . Hence, an approach to studying recombination in the sample of genetic sequences is to study the loops that those sequences generate when represented in the above way.

A valuable attribute of TDA methods is that they are informative about the scale or size of the inferred topological features. Given a finite set of data points, there is an infinite number of spaces that are compatible with the points. TDA structures this spectrum of possibilities by introducing a notion of scale (Fig. B): at a given scale $\epsilon$, two points are connected in the underlying space if their distance is smaller than $\epsilon$. Topological features compatible with the data can be then summarized in terms of sets of intervals, named *barcodes*  (Fig. C). Each interval in a barcode represents the range of scales across which a particular topological feature (e.g. a loop) is present in the inferred topological space. In the genomic context introduced above, barcodes of loops summarize the frequency and scale (mutational distance between recombining sequences) of recombination events, and provide a basic structure on which statistics of genomic exchange can be built .

TDA methods are particularly well suited for large datasets. In the context of molecular phylogenetics and evolution, they have been applied to the study of viral recombination and reassortment , bacterial species and point estimators in population genetics . However, these implementations of TDA have limitations, as they are not tailored for the biological problem they try to address. Specifically, traditional TDA methods only use information about genetic distances between sequences, and so they discard the full structure of segregating characters, missing numerous recombination events that are required to explain the data. Relatedly, it is unclear which specific evolutionary histories explaining the data TDA informs about, and what is the precise relation between barcodes and these histories.

Here we address these two important aspects, improving on the scalable capabilities of TDA to extract robust information on the possible evolutionary histories of a sample of genetic sequences. In particular, we show that by systematically sampling subsets of segregating sites and performing TDA, we are able to identify most of the necessary recombination events identified by bound methods , providing a significant improvement of past methods in terms of interpretation and sensitivity. Moreover, we introduce a novel type of graph (*topological ARG* or *tARG*), closely related to minimal ARGs, that captures ensembles of minimal recombination histories; and we show that TDA informs about the topological features and genetic scales of these graphs. Like minimal ARGs , tARGs can be considered as explicit, parsimonious, interpretable phylogenetic representations. The main advantage of tARGs and barcodes versus minimal ARGs is, however, the possibility of obtaining such phylogenetic information in polynomial time, which allows us to deal with hundreds of sequences. We have implemented this method in a software, called `TARGet`, and have illustrated it with several examples, including small migration between diverging populations, human recombination, and horizontal evolution of finches inhabiting the Galápagos archipelago. The software, instructions and example files used in the manuscript can be obtained from `https://github.com/RabadanLab/TARGet`.

# Results

## Topological ARGs

An ARG is an explicit phylogenetic network representing a possible evolutionary history of a sample of genetic sequences, where only mutation and recombination events are present and convergent evolution is not considered and so never occurs . ARGs are very useful constructs in population genetics and phylogenetics. However, the problem of building a minimal ARG from a set of genetic sequences is known to be NP-hard . The use of ARGs has therefore been traditionally limited to small samples, consisting of a handful of sequences.

In this section, we introduce a particular class of minimal ARGs and a set of related graphs. Then, using computational algebraic topology, in the next section we show that it is possible to extract, in polynomial time, phylogenetic information from this class of minimal ARGs, without having to explicitly construct them. Thus, by restricting to this specific class of graphs, we are able to extend the realm of ARGs to large samples of sequences.

To be specific, we consider a sample $\mathcal{S}$ consisting of $n$ distinct genetic sequences with $m$ binary segregating characters. The latter can be single nucleotide polymorphisms (SNPs), indels, gene duplications or any other genetic trait that takes one of two possible states, 0 or 1, in each sequence. An ARG is then formally defined as a directed acyclic graph $\mathcal{N}$ with $n$ leaf nodes and a unique root node, where every node other than the root has in-degree one (tree node) or two (recombination node), every segregating character labels a unique edge in $\mathcal{N}$ (infinite sites assumption), and every sequence in $\mathcal{S}$ labels a unique leaf in $\mathcal{N}$. Moreover, each node in $\mathcal{N}$ is labelled by a $m$-length binary sequence, such that the sequence labelling a tree node differs from the sequence of the parent node only at the character labelling the edge that connects the two nodes; and the sequence labelling a recombination node is a combination of the sequences labelling the two parent nodes. Single-crossover recombinant sequences are formed by taking the first $k$ sites from the sequence of one of the parent nodes (*prefix*) and appending the last $m-k$ sites from the sequence of the other parent node (*suffix*), for $k\in[1,m-1]$.

There is an infinite number of ARGs that can explain a given sample $\mathcal{S}$ . A stochastic model, such as the coalescent model with recombination , would assign probabilities to each possible ARG. Here, however, we adopt a parsimony approach and consider ARGs that are minimal (in a sense defined below), without assuming an underlying probabilistic model. Such a model-independent approach has proven useful in summarizing genetic sequences into evolutionary histories where all events are required.

Specifically, we consider ARGs that contain exactly the minimum number $R_{\rm min}$ of single-crossover recombinations required to explain the sample, and that minimize the function $$\label{eq1}
D(\mathcal{N}) = \sum_{r=0}^{R_{\rm min}}d_{r}$$ where the sum runs over all recombination events in $\mathcal{N}$, and $d_r$ is the Hamming distance between the two parental sequences involved in the $r$-th recombination. This is a more restricted definition of minimal ARG than the one that usually appears in population genetics literature , where the condition on $D(\mathcal{N})$ is generally not required. We use the term *ultra-minimal ARG* to refer to this restricted type of minimal ARG. Ultra-minimal ARGs are thus minimal ARGs where recombination events involve parental sequences that are as genetically close as possible. They introduce a higher level of parsimony than minimal ARGs, being informative not only about the minimum number of recombination events, but also about the minimum genetic distance between the recombining sequences that took part in those events. By construction, an ultra-minimal ARG explaining any given sample always exists. Examples are shown in Figs.  and .

A minimal ARG can be condensed by collapsing all unlabelled edges, so that the resulting graph can be embedded into an $m$-dimensional hypercube and its diagonals (that is, the line segments joining non-consecutive vertices) (Fig. ). The number of edges and vertices of such a condensed representation is $m+2R_{\rm min}$ and $m+R_{\rm min}+1$, respectively, whereas the number of independent loops is $R_{\rm min}$, where a loop is said to be independent if it cannot be embedded in the union of other loops. In this representation, the distance between two nodes is defined as the number of edges in the shortest path connecting the nodes, and is equal to the Hamming distance between the corresponding sequences.

Given a sample $\mathcal{S}$ of genetic sequences, we would like to obtain information about the ultra-minimal ARGs that explain $\mathcal{S}$, without explicitly constructing them. To that end, we consider the undirected graph $\mathcal{G}=(V,E)$, with vertices $V$ and edges $E=E_1\cup\ldots\cup E_l$, that results from the union of all condensed ultra-minimal ARGs $\mathcal{G}_i=(V, E_i)$ explaining $\mathcal{S}$ and having the same set of vertices $V$ (Fig. ). We call this construction *topological ARG* (tARG). A tARG therefore summarizes the collection of most parsimonious histories associated to a sample of genetic sequences. However, unlike minimal ARGs, tARGs are completely determined by their vertices. By considering tARGs instead of minimal ARGs, we are able to reduce an NP-hard problem into a much simpler (but still very informative) topological problem, as we describe in next section.

## Persistent homology and recombination inference

Topological data analysis has emerged during the last decade as a branch of applied topology that attempts to infer topological features of spaces (such as the number of loops and holes) from sets of sampled points . The topological features of a space are preserved under continuous deformations of the space and can be arranged in mathematical structures called *homology groups* . We refer the reader to refs.  for formal definitions and basic introductions to algebraic topology. In brief, the $n^\textrm{th}$ homology group of a space is an algebraic structure that encompasses all $(n+1)$-dimensional holes of the space. Of special interest to us is the first homology group, whose elements correspond to loops.

Homology groups can be computed by replacing the original space with a simpler one, known as *simplicial complex*, which has the same topological features as the original space but consists of a finite set of elements (Fig. B). A simplicial complex is a generalization of a network that, in addition to nodes and vertices, includes higher dimensional elements like triangles and tetrahedra. Simplicial complexes are powerful because they allow the implementation of algebraic operations to extract the topological features of the space.

When only a finite set of points of the space is given, there is still a well-defined notion of homology groups, known as *persistent homology* , which capture the topological features of the underlying space. At each value of a scale parameter $\epsilon$, a simplicial complex (known as Vietoris-Rips complex) can be constructed by considering the intersections of balls of radius $\epsilon$ centred at the sampled points (Fig. B). Points are joined if their corresponding balls intersect. This process produces a sequence of simplicial complexes parametrized by $\epsilon$, from which persistent homology can be computed using available algorithms . Remarkably, the computation time of persistent homology is polynomial in the number of points .

Persistent homology can be represented using barcodes . These are graphical representations where each element of persistent homology is represented by a segment spanning the interval $[\epsilon_b,\, \epsilon_d]$, where $\epsilon_b$ and $\epsilon_d$ are the values of the parameter $\epsilon$ at which the corresponding feature is respectively formed and destroyed in the sequence of simplicial complexes (Fig. C). Thus, each segment in a barcode represents a topological feature inferred from the data, and the position and length of the segment are informative of the size of the topological feature. The values $\epsilon_b$ and $\epsilon_d$ are referred as *birth* and *death time* of the topological feature, respectively.

In the current context, we exploit the use of persistent homology to infer topological features of an unknown tARG, given a set of sampled nodes (Fig. ). The use of persistent homology to detect the presence of recombination in genetic samples was proposed in . However, the relation between persistent homology and explicit evolutionary histories incorporating recombination events was not studied. Our aim is inferring information about the loops of the tARG, as they correspond to recombination events present in the collection of most parsimonious histories explaining the sample. To that end, we consider the Hamming distance matrix of the sample and compute persistent homology using the algorithm developed in ref. . Since computing the distance matrix and persistent homology requires respectively $\mathcal{O}(n^2m)$ and $\mathcal{O}(n^3)$ operations , the running time grows at most cubically with the number of genetic sequences. An advantage of using persistent homology instead of just counting loops in a nearest neighbour graph is that we also obtain valuable information about the genetic distances between recombining sequences.

The barcode that results from this computation contains information about the number and size of the loops in the tARG underlying the sample (Fig. ). Each segment in the barcode represents a loop in the tARG, and therefore a recombination event in an ultra-minimal ARG explaining the sample. The position of each segment provides information about the genetic scales involved in the corresponding recombination event. Specifically, $2\epsilon_d$ sets an upper bound to the mutational distance between the two recombining sequences, since all pairwise distances between nodes in the loop are smaller than $2\epsilon_d$. The number of segments in the barcode (namely, the dimension of the first persistent homology group) or *persistent first Betti number*, $b_1$, is hence a lower bound of the number of recombination events in the tARG, $\overline{R}_{\rm min}$. Note that, since a tARG is the union of multiple minimal histories, $\overline{R}_{\rm min}$ can be larger than $R_{\rm min}$. In particular, $\overline{R}_{\rm min} > R_{\rm min}$ when there are three characters for which all eight possible allele combinations appear in the sample. In general, this can only happen at very large recombination rates.

## The barcode ensemble of a sample

The sensitivity of persistent homology to detect recombination decreases as the number $m$ of segregating characters increases. Indeed, in that case the dimensionality of the ambient space is larger and the sample becomes sparser. For this reason, $b_1$ is in general a loose lower bound of $\overline{R}_{\rm min}$. To address a similar problem, Myers and Griffiths introduced the idea of combining the local bounds that result from partitioning the sequence, building a more stringent global bound . In this way, information about the ordering of characters is incorporated and the location of recombination breakpoints is constrained in the sequence. This general idea was applied in to the haplotype bound, $n-m-1 \leq R_{\rm min}$, to built a stronger lower bound of $R_{\rm min}$, denoted $R_{\rm MG}$.

A similar idea can be applied in the context of barcodes to build a *barcode ensemble*, given by the disjoint union of the persistent first-homology barcodes of a set of optimally chosen, non-overlapping intervals within the sequence alignment (Fig. A). Given a partition of a genetic sequence, the barcode associated to each interval captures information about recombination events with breakpoint in that interval. Due to the curse of dimensionality mentioned in the previous paragraph, the union of the barcodes associated to two contiguous genomic intervals often captures more recombination events than the barcode associated to the union of the two genomic intervals. Therefore, by systematically exploring all possible partitions of the genetic sequence, it is possible to find a partition that maximizes the total number of bars in the barcodes. The solution is often not unique, as different partitions may lead to the same total number of bars. One may reduce this degeneration by considering additional criteria, such as also maximizing the total length of the bars (so that they are more informative about genetic distances). The formal details of the barcode ensemble construction are presented in the Methods section.

The barcode ensemble incorporates information about the full structure of characters in the sample, largely increasing the sensitivity of persistent homology to recombination and providing information on the location of the recombination breakpoints in the sequence. The number of bars in the barcode ensemble, $\bar b_1$, is an improved lower bound of $\overline{R}_{\rm min}$, in the same way as $R_{\rm MG}$ is an improved lower bound of $R_{\rm min}$:

$$\begin{CD}
\textrm{tARG} \ @>>> \ \overline R_{\rm min} \ \geq \ \bar b_1 \ \geq \ b_1 \\
@AAA\\
\textrm{(ultra) minimal ARG} \ @>>> \ R_{\rm min} \ \geq \ R_{\rm MG} \ \geq \ n-m-1
\end{CD}$$

In biological data, $\bar b_1$ and $R_{\rm MG}$ are in general very close to each other (Fig. B), as tARGs with $\overline{R}_{\rm min}>R_{\rm min}$ occur very rarely. However, unlike $R_{\rm MG}$, barcode ensembles provide additional phylogenetic information, such as bounds on the mutational distances between recombining sequences (note that birth and death times in barcode ensembles refer to local genetic distances, namely mutational distances across the genomic interval associated to the particular bar). These features put barcode ensembles at the very interesting interface between the fast, but phylogenetically limited, existing lower bounds to $R_{\rm min}$; and the slow, but phylogenetically rich methods for reconstructing minimal ARGs. We have implemented the computation of barcode ensembles in publicly available software, called `TARGet`.

## Examples

We consider five examples that illustrate how the formal developments presented in previous sections can be used to extract useful phylogenetic information from samples of genetic sequences. The first example is a simple toy model where an explicit minimal ARG can be easily constructed. It displays how the information contained in the barcode ensemble of the sample directly maps to features of ultra-minimal ARGs. The second example, based on simulated data of two sexually reproducing populations exchanging genetic material at low rate, shows the applicability of persistent homology to large datatsets, consisting of several hundreds of sequences. It also demonstrates the use of phylogenetic information contained in the barcode ensemble to distinguish among various biological settings with similar recombination rates. The third and fourth examples consist respectively of 250 and 100 kilobase regions in the HLA and MS32 loci of $\sim 100$ humans, where several meiotic recombination hotspots localize. The fifth example consists of a 9 megabase scaffold in the genome of 112 Darwin's finches . These last three examples serve to illustrate the applicability of barcode ensembles to real datasets.

#### A simple example.

We illustrate the use and interpretation of barcode ensembles with a simple example, consisting of a sample of 4 genetic sequences with 7 binary characters: 1111001, 1111111, 0000110 and 0000000. Minimal ARGs explaining this sample require two single-crossover recombination events. An ultra-minimal ARG is presented in Fig. A. The most ancestral recombination event involves genetically distant parental gametes, leading to a large loop in the ARG. To the contrary, the most recent recombination event involves genetically close parental gametes, leading to a second small loop in the ARG. These features are captured by the barcode ensemble of the 4 sequences (Fig. B), which consists of two bars, corresponding to the two recombination events. The position of the bars represent the genetic scales associated to the recombination events, with the $2\epsilon_d=5$ ($3$) bar corresponding respectively to the large (small) recombination loop. These death times are good upper bounds for the mutational distance between recombinant sequences in the two genomic intervals associated to each bar (characters 1 to 5, and 6 to 7, respectively). The position of the crossover breakpoints associated to these recombination events is also correctly reproduced. Hence, taking as input the 4 sequences, the barcode ensemble extracts phylogenetic information from ultra-minimal ARGs that explain the sample, without requiring complete reconstruction of the ARGs.

We note here the importance of using the barcode ensemble instead of the ordinary barcode, used in previous phylogenetic applications of persistent homology . In this simple example $b_1=1$, and only one of the two recombination events would have been detected if the ordinary first-homology barcode had been used. The barcode ensemble largely increases the sensitivity to detect recombination events.

We can attempt to reconstruct the tARG of the sample by using persistent homology generators (Fig. C). Whereas there are theorems ensuring the stability of barcodes against small perturbations , the generators of persistent homology identified by TDA strongly depend on the sample, and multiple choices of basis are possible. Hence, the use of generators to reconstruct the tARG is usually limited to small datasets. In this simple example, both bars in the barcode ensemble are generated by the four sampled sequences. Therefore, the reconstructed loop enclosing each recombination event is the same in both cases and corresponds to the large enveloping loop in the ultra-minimal ARG (Fig. A). Adding the internal nodes 1111000 and 1111110 to the sample permits disentangling the generators of the two loops (Fig. C), fully reconstructing the topology of the underlying tARG.

#### Genetic exchange between two divergent populations.

We now consider a more involved example consisting of two sexually-reproducing populations, simulated under the coalescent model with recombination. The two populations diverged $24N$ generations before present. Their effective population sizes are taken to be constant and given by $N$ and $N/5$. We consider two different cases, depicted in Fig. . In the first case (Fig. A), the two populations are completely isolated from each other. In the second case (Fig. B), to the contrary, there is a small migration rate between the two populations. The recombination rate is the same in both cases. Alternatively, in a phylogenetic context, this setting describes the incomplete lineage sorting of two species, with or without the presence of gene flow.

We randomly sampled 250 sequences from the large population and 50 sequences from the small population. The full sample consisted of 300 sequences with 300 segregating sites. We present in Fig.  the barcode ensemble for simulated samples without and with migration. The computation took approximately 33 minutes (wall-clock time) in a modern 8-cores desktop computer. Whereas the number of detected recombination events in the tARG, counted by the number bars, is similar in both cases, their genetic scales are very different. Specifically, in the presence of migration the size of some of the loops in the tARG is large, corresponding to migration events followed by a recombination event (Fig. B). This is indicated by the presence of bars with large death time $\epsilon_d$ in the barcode ensemble of the case with migration, corresponding to recombination events with large mutational distances between recombining sequences.

Hence, in this example the barcode ensemble provides rich phylogenetic information that could be hardly obtained by other methods. Methods that attempt to construct a minimal (or nearly-minimal) ARG are computationally inefficient for such large sample sizes . Fast bound methods , on the other hand, do not provide enough phylogenetic information to distinguish between the cases with and without migration, as the total recombination rate is the same in both situations. Sequentially Markov coalescent approaches produce an ARG that is far from being minimal but is a good approximation to the maximum likelihood. However, these methods require an underlying coalescent model, with mutation, recombination and population structure parameters given as priors. Finally, algorithms for constructing phylogenetic split networks  are fast and provide very different outputs in each of the above two cases. However, the interpretation of the output in terms of recombination and migration events is obscure.

#### Human leukocyte antigen (HLA) locus.

The previous examples serve to illustrate the relation between features of the barcode ensemble of a genetic sample and those of the ultra-minimal ARGs explaining the sample. However, both examples are based on simulated data. We now consider a more realistic example, consisting of 180 phased genotypes from a $\sim$`<!-- -->`{=html}250 kilobase region of the HLA locus of 90 individuals belonging to the Luhya in Webuye, Kenya (LWK) population, sequenced as part of the International HapMap Project . In total, the region contains 471 SNPs. Recombination hotspots in this part of the HLA locus have been studied in detail in the past through sperm typing and other high-resolution methods . This example therefore serves to illustrate the capacity of the barcode ensemble to localize recombination events in realistic situations. With this aim, we also considered 194 phased genotypes from a smaller region (40 kilobase) within the same HLA locus of 97 individuals from the same population, sequenced by the 1,000 Genomes Project Consortium . This additional dataset contained a higher density of SNPs (482 SNPs in total), allowing for a higher resolution in the localization of recombination events.

We used `TARGet` to compute the first-homology barcode ensembles of the two datasets and analysed the distribution of bars across the HLA locus (Figs. A and B). The computation took 19 and 14 minutes (wall-clock time) in a modern 8-cores desktop computer, respectively for the HapMap and 1,000 Genomes datasets. Comparison with the African-American recombination map , based on more than 2 million crossovers in 30,000 unrelated African-Americans, shows a large degree of consistency between the recombination rates and the genomic position of recombination events detected by the barcode ensembles (Fig. A). The distribution of mutational distances associated to recombination events is qualitatively consistent with coalescent arguments (Fig. C). In particular, bars with large death time $\epsilon_d$, corresponding to recombination events with a large mutational distance between recombining sequences, are mostly associated to regions of low recombination rate, consistently with the longer coalescence time for these regions .

#### Human MS32 mini-satellite locus.

Similarly, we also considered 194 phased genotypes from a $\sim 100$ kilobase region (416 segregating sites) near the MS32 mini-satellite locus of 97 individuals from the LWK population, sequenced by the 1,000 Genomes Project Consortium . We used `TARGet` to compute the barcode ensemble and studied the distribution of bars across this genomic region (Fig. D). The computation took 10 minutes (wall-clock time) in a modern 8-cores desktop computer. As in the previous example, the genomic position of the recombination events detected by the barcode ensemble (Fig. D) was consistent with the recombination rate across this region, as determined by the African-American recombination map .

#### Darwin's finches.

Our last example consists of the genetic sequences of 112 Darwin's finches, belonging to 15 different species inhabiting the Galápagos archipielago and Cocos Island . We aligned and genotyped a 9 megabase scaffold of their genome and, after filtering for high-quality variants, we focussed on a set of 140 SNPs that were homozygous across the 112 samples, thus avoiding potential phasing artefacts. By considering this set, we mostly restrict to very ancestral recombination/gene flow events, close to the origin of radiation from a common ancestor 1.5 million years ago . We used `TARGet` to obtain the first-homology barcode ensemble of the sample, as well as the partially reconstructed tARG. The computation took 9 minutes (wall-clock time) in a modern 8-cores desktop computer.

The first-homology barcode ensemble (Fig. A) contains 13 recombination events, mostly involving samples from multiple species and usually including samples from the genus *Certhidea* (Fig. B), the most ancestral lineage among the genera present in the sample . These results add support to the evidence for genetic introgression found in . Our analysis also reveals that the crossover breakpoints of these events localize at four different genomic regions within the 9 megabase scaffold that we have considered in this example (Fig. C).

## Parameter estimation

The examples above illustrate the use and interpretation of barcode ensembles in molecular phylogenetics. As we have discussed, an important feature of topological approaches to phylogenetics is that they inform about most parsimonious evolutionary histories. Being model-independent approaches, they describe minimal sets of events required to explain a sample of sequences, without assuming any probabilistic model of evolution. In some situations, however, we are interested in estimating the parameters of a specific evolutionary model from the observed data (e.g. the recombination rate in a coalescent model with recombination). To that end, barcode ensembles can be taken as summary statistics from which to build parameter estimators. For instance, in Fig. A we show the dependence of $\overline{b}_1$ on the recombination rate for a set of 1,000 coalescent model simulations. The expected $\overline{b}_1$ of the barcode ensemble is informative of the recombination rate, growing monotonically with the later. Compared to sequentially Markov coalescent (SMC) approaches for ARG inference , $\overline{b}_1$ is strongly correlated with the number of recombinations in SMC ARGs derived from the same set of sequences (Pearson's $r=0.93$, $p<10^{-100}$, Fig. S1). Although the coefficient of variation is $\sim 35\%$ larger for $\overline{b}_1$ (Fig. S1), its computing time is substantially lower ($> 9$ times faster after parallelizing in a modern 8-cores desktop computer, Fig. S1), being a robust approach to coalescent-model recombination rate estimation in large datasets. Furthermore, unlike the number of recombinations in SMC ARGs, $\overline{b}_1$ is unbiassed at small recombination rates, vanishing when the recombination rate is zero (Fig. A).

Although recombination rate estimation is a very direct example, the barcode ensemble of a sample of genetic sequences contains other rich phylogenetic information apart from $\overline{b}_1$, which can be used for more complex parameter estimation in structured models of evolution. Consider, for instance, the case of two divergent populations with migration and recombination discussed above. In this model, the average genetic distance between recombining sequences is expected to decrease with the migration rate, as the average time to the most recent common ancestor between foreign and local gametes in a population is shorter. In figure B we show the dependence of the average death time ($\langle \epsilon_d\rangle$) on the migration rate parameter, for the barcode ensembles of a set of 900 coalescent model simulations with fixed recombination and variable migration rates. As expected, $\langle \epsilon_d\rangle$ is informative of the migration rate, decreasing monotonically with the later. It is therefore a good measure for estimating migration rates. Consistently, $\langle \epsilon_d\rangle$ correlates with time to the most recent common ancestor of recombining sequences in SMC ARGs obtained from the same data (Pearson's $r=0.55$, $p<10^{-72}$, Fig. S1). Although the coefficient of variation of $\langle \epsilon_d\rangle$ is $\sim 60\%$ larger (Fig. S1), extracting this type of information from SMC ARGs requires the implementation of a greedy algorithm, substantially increasing the running time ($\sim 8$ times slower in a single core of modern desktop computer, Fig. S1) and therefore limiting its applicability to large datasets.

These two simple examples illustrate the utility of barcode ensembles for building parameter estimators in specific models of evolution. Importantly, being model-independent, they are robust and flexible tools which can be applied in an infinitely large number of possible evolutionary models.

# Discussion

As the famous title of the essay by Dobzhansky "Nothing in Biology Makes Sense Except in the Light of Evolution" underscores, evolutionary processes are central orchestrating themes in biology. Mutations, recombinations and other evolutionary processes get imprinted into genomes through selection, reflecting the accumulated history giving rise to an organism. Phylogenetics try to reconstruct the evolutionary history through the comparison of genomes of related organisms. In addition to reporting relationships and elucidating particular histories, one would like to understand and quantify how different evolutionary processes have occurred. The identification and quantification of evolutionary processes can be challenging due to the lack of a well-established universal framework to capture evolutionary relationships beyond trees. In addition, robust statistical inference needs to exploit the large number of genomes that are now becoming available, aggravating the computational burden and obscuring interpretations. Ideally, we would like to have a biologically interpretable framework able to quantify different evolutionary processes by analyzing large numbers of genomes.

In this paper we have proposed a few steps in this direction. We have extended the notion of barcodes in persistent homology to identify the genetic scale and number of recombination events. We have shown that, by correctly studying persistent homology in subsets of segregating sites, it is possible to characterize the genomic regions where recombination takes place and identify the gametes involved in particular recombination events. The persistent homology barcodes derived from each of these sets can be structured as a "barcode ensemble" where each bar captures a recombination event. Barcode ensembles can be interpreted as counting and quantifying the scale of recombination events in a variation of Ancestral Recombination Graphs (ARGs). Topological ARGs represent a summary of potential recombination histories that can explain the data. The method proposed, `TARGet`, is scalable to hundreds of genomes. As an alternative to some phylogenetic networks, barcode ensembles provide robust quantification of events, the distribution of genetic scales, computational scalability and interpretative graphs.

Barcode ensembles are versatile in that they do not assume any specific model of evolution, providing explicit, interpretable summaries of the minimal set of recombination events required to explain a sample of genetic sequences. Here we have illustrated their use in several practical cases. However, the range of possible applications is unlimited. In some cases, it may be convenient to perform minor modifications to the approach described here. For instance, although in our exposition we have only made use of Hamming distance and binary sequences, the main concepts we have presented extend straightforwardly to other genetic distances. The use of these metrics can be particularly useful in cases with rapidly diverging samples or substantial mutational biases. In other cases, information about the ancestral and derived alleles for each character in the sample may be available. Although tARGs have no natural directionality, the inclusion of the ancestral sequence in the original sample may lead in those cases to more stringent bounds on $\overline{R}_{\rm min}$, similarly to what occurs with other approaches to recombination inference . Finally, more efficient integer linear programming algorithms, like the one of , could in principle be also generalized to the computation of barcode ensembles.

# Methods

## First-homology barcode ensemble

We extended the construction of ref.  to persistent homology barcodes. From a geometric perspective, this corresponds to projecting the original space on sets of mutually orthogonal hyperplanes in the ambient hypercube, and computing persistent homology in each of those projections. For that aim, we need to establish an ordering relation on barcodes. Being sets of intervals, it is natural to take the maximum of two barcodes to be given by the one with largest $L^0$-norm, namely largest $b_1$. If both barcodes have the same $L^0$-norm, we may successively compare other norms (e.g. other $L^p$-norms), until the tie is broken or, otherwise, one of the two barcodes is arbitrarily chosen. The algorithm of is then generalized to persistent homology barcodes as follows:

1.  Let $\mathcal{B}_{ik}$ be the first-homology barcode of the sequences that result from the $i$-th to $k$-th characters in $\mathcal{S}$. Set $\mathcal{R}_{ij} = 0$ and $k = 2$.

2.  For $j=1,\ldots,k-1$, set $\mathcal{R}_{jk} = \textrm{max}\{\mathcal{R}_{ji} \cup \mathcal{B}_{ik}\ :\ i = j, . . . , k - 1\}$

3.  If $k < m$, increment $k$ by 1 and go to step 2.

The barcode ensemble of $\mathcal{S}$ is the union barcode $\mathcal{R}_{1m}$ that results from this algorithm.

We implemented the algorithm in a publicly available multi-threaded software, `TARGet`, which is distributed under the GNU General Public License (GPL v3). The application is fully written in Python 2.7, and relies on Dionysus C++ library for persistent homology computations (`http://www.mrzv.org/software/dionysus`). Since considering all possible sequence partitions is unnecessary and computationally infeasible in most cases, we follow the strategy of ref.  and allow the user to limit the number of partitions by the maximum number of segregating characters within each subset of $\mathcal{S}$ (specified by the command line option `-s`), and by the maximum distance between segregating characters in the subset (specified by the command line option `-w`). In addition, we also allow the user to exclude from $\mathcal{S}$ segregating characters that are compatible (namely, that satisfy the Hudson-Kaplan four-gamete test ) with all the other characters in $\mathcal{S}$ (specified by the command line option `-e`). For each genomic interval, a filtration of Vietoris-Rips complexes is constructed using Hamming distance and the persistent first-homology group is computed over $\mathbf{Z}_2$.

## Population genetics simulations

We performed 4,000 simulations of a sample of 40 sequences with 12 segregating sites, using the software `ARGweaver` . The population was simulated using a coalescent infinite sites model with recombination. The population-scaled recombination rate, $\rho$, was randomly generated in each simulation, taking values from a uniform distribution between 0 and 110. For each simulated sample, Myers and Griffiths lower bound $R_{\rm MG}\leq R_{\rm min}$ was computed using the software `RecMin` , with parameters `-s 12 -w 12`. Lower bounds $\bar b_1\leq \overline{R}_{\rm min}$ were computed using our application `TARGet`, with parameters `-s 12 -w 12`.

To study the dependence of $\overline{b}_1$ on the recombination rate parameter in coalescent models, we performed 1,000 simulations of a sample of 200 sequences. The population-scaled recombination rate, $\rho$, was randomly generated in each simulation, taking values between 0 and 216. For each simulated sample, `TARGet` was run with parameters `-s 11 -w 11`, and `ARGweaver`'s tool `arg-sample` was run with parameters `-m 7e-9 -n 400 –sample-step 10`, discarding the first 200 iterations.

Samples of genetic exchange between two divergent populations were simulated using the software `ms` , using the commands

`ms 300 1 -s 300 -r 40 10000 -I 2 250 50 -ej 6.0 1 2 -n 2 0.2 -m 1 2 0.5`

and,

`ms 300 1 -s 300 -r 40 10000 -I 2 250 50 -ej 6.0 1 2 -n 2 0.2`

respectively for the cases with and without migration. The barcode ensemble of each sample was computed using `TARGet` with parameters `-s 12 -w 14 -e`.

To study the dependence of $\langle\epsilon_d\rangle$ on the migration rate in this scenario, we performed 900 simulations using the software `ms` and `seq-gen` , with the commands

`ms 150 1 -T -r 60 10000 -I 2 125 25 -ej 6.0 1 2 -n 2 0.2 -m 1 2 X`

and

`seq-gen -mHKY -l 10000 -s 0.004 -p 50000`

where the migration rate `X` in the first command takes random values from a uniform distribution between 0 and 2. For each simulated sample, `TARGet` was run with parameters `-w 8 -s 8`, and `arg-sample` was run with parameters `-m 1e-7 -n 400 -r 1.5e-7`. We extracted from SMC ARGs the time to the most recent common ancestor of recombining sequences using a greedy algorithm that searches for the shortest non-zero path connecting the two sequences.

## HLA and MS32 loci

We downloaded phased genotype data from HapMap phase III , corresponding to all SNPs of LWK population between rs6457661 and rs3129301 in chromosome 6. We also downloaded phased genotype data from 1,000 Genomes Project , corresponding to all SNPs of LWK population between positions 32,887,978 and 32,927,978 of chromosome 6, and half of the SNPs of LWK population between positions 234,190,031 and 234,291,193 of chromosome 1. All coordinates refer to human assembly hg18. The barcode ensemble of each dataset was computed using `TARGet` with parameters `-s 12 -w 12`.

## Darwin's finches genotyping

Raw paired-end reads from 112 Darwin finches were obtained from SRA archive (accession number PRJNA263122) and aligned against the consensus sequence of *Geospiza Fortis*, version GeoFor_1.0/geoFor1, scaffold JH739904. We followed essentially the same procedure than that of ref.  for the alignment, SNP calling, genotyping and filtering. In short, the alignment was performed with Burrows-Wheeler aligner (BWA) , version `0.7.5`, using BWA-MEM algorithm and default parameters. PCR duplicates were marked using Picard tools (`http://picard.sourceforge.net/`). Indel realignment, SNP discovery and simultaneous genotyping across the 112 samples was performed using Genome Analysis Toolkit (GATK) , following GATK best practice recommendations . SNP calls were filtered by keeping variants with SNP quality $>$ 100, total depth of coverage $>$ 117 and $<$ 1750, ratio between SNP quality and depth of coverage $>$ 2, Fisher strand bias $<$ 60, mapping quality $>$ 50, mapping quality rank $>$ -4 and read position rank sum $>$ -2. In total, 13,980 variant positions passed these filters. To avoid phasing errors, we only considered SNPs that were homozygous across the 120 samples. The resulting genotypes were processed with `TARGet` for barcode ensemble computation, using the options `-s 14 -w 14`.

# Acknowledgments

We thank C. Solis-Lemus, C. Ane, D. S. Rosenbloom, K. Emmett, M. Lesnick, S. Zairis and V. Aleksandrov for useful discussions and comments.

Figure S1. **Comparison to SMC approaches to ARG inference.** (A) The number of recombination events in SMC ARGs , $R_{\rm argweaver}$, plotted against the number of bars in the barcode ensemble, $\overline{b}_1$. Both quantities are strongly correlated (Pearson's $r=0.93$, $p<10^{-100}$). Plot based on 1,000 coalescent model simulations of a sample of 200 sequences. (B) Coefficient of variation of $R_{\rm argweaver}$ (blue) and $\overline{b}_1$ (red) as a function of the recombination rate. (C) Distribution of wall-clock running times for the simulations in (A). (D) Average time to the most recent common ancestor of recombining sequences in SMC ARGs, $\langle T_{\rm argweaver}\rangle$, plotted against the average death time of bars in the barcode ensemble, $\langle \epsilon_d\rangle$, for two divergent populations with recombination and migration. Both quantities are largely correlated (Pearson's $r=0.55$, $p<10^{-72}$). Plot based on 900 simulations of a sample of 150 sequences. (E) Coefficient of variation of $\langle T_{\rm argweaver}\rangle$ (blue) and $\langle \epsilon_d\rangle$ (red) as a function of the migration rate. (F) Distribution of wall-clock running times for the simulations in (D). `ARGweaver` running times also include the time required to extract $\langle T_{\rm argweaver}\rangle$ from SMC ARGs.