Brian Nosek had pretty much given up on finding a funder. For two years he had sent out grant proposals for his software project. And for two years they had been rejected again and again—which was, by 2011, discouraging but not all that surprising to the 38-year-old scientist. An associate professor at the University of Virginia, Nosek had made a name for himself in a hot subfield of social psychology, studying people’s unconscious biases. But that’s not what this project was about. At least, not exactly.
Like a number of up-and-coming researchers in his generation, Nosek was troubled by mounting evidence that science itself—through its systems of publication, funding, and advancement—had become biased toward generating a certain kind of finding: novel, attention grabbing, but ultimately unreliable. The incentives to produce positive results were so great, Nosek and others worried, that some scientists were simply locking their inconvenient data away.
The problem even had a name: the file drawer effect. And Nosek’s project was an attempt to head it off at the pass. He and a graduate student were developing an online system that would allow researchers to keep a public log of the experiments they were running, where they could register their hypotheses, methods, workflows, and data as they worked. That way, it would be harder for them to go back and cherry-pick their sexiest data after the fact—and easier for other researchers to come in and replicate the experiment later.
But in July 2012, Nosek received an email from an institution whose name he didn’t recognize: the Laura and John Arnold Foundation. A Google search told him that the Arnolds were a young billionaire couple in Houston. John, Nosek learned, had made his first millions as a wunderkind natural gas trader at Enron, the infamous energy company, and he’d managed to walk away from Enron’s 2001 collapse with a seven-­figure bonus and no accusations of wrong­doing attached to his name. After that Arnold started his own hedge fund, Centaurus Energy, where he became, in the words of one hedge fund competitor, “the best trader that ever lived, full stop.” Then Arnold had abruptly retired at the ripe age of 38 to focus full time on philanthropy.
As Nosek tells it, John Arnold had read about the Reproducibility Project in The Chronicle of Higher Education and wanted to talk. By the following year, Nosek was cofounding an institution called the Center for Open Science with an initial $5.25 million grant from the Arnold Foundation. More than $10 million more in Arnold Foundation grants have come since. “It completely transformed what we could imagine doing,” Nosek says. Projects that Nosek had once envisioned as modest efforts carried out in his lab were now being conducted on an entirely different scale at the center’s startup-like offices in downtown Charlottesville, with some 70 employees and interns churning out code and poring over research. The skeletal software behind the data-sharing project became a slick cloud-based platform, which has now been used by more than 30,000 researchers.
The Reproducibility Project, meanwhile, swelled to include more than 270 researchers working to reproduce 100 psychology experiments—and in August 2015, Nosek revealed its results. Ultimately his army of volunteers could verify the findings of only about 40 percent of the studies. Media reports declared the field of psychology, if not all of science, to be in a state of crisis. It became one of the biggest science stories of the year.
But as it happens, Nosek is just one of many researchers who have received unsolicited emails from the Arnold Foundation in the past few years—researchers involved in similar rounds of soul-searching and critique in their own fields, who have loosely amounted to a movement to fix science.
John Ioannidis was put in touch with the Arnolds in 2013. A childhood math prodigy turned medical researcher, Ioannidis became a kind of godfather to the science reform crowd in 2005, when he published two devastating papers—one of them titled simply “Why Most Published Research Findings Are False.” Now, with a $6 million initial grant from the Arnold Foundation, Ioannidis and his colleague Steven Goodman are setting out to turn the study of scientific practice—known as meta-research—into a full-fledged field in its own right, with a new research center at Stanford.
British doctor Ben Gold­acre also got an email from the Arnold Foundation in 2013. Famous in England as a sharp-witted scourge of “bad science,” Goldacre spent years building up a case that pharmaceutical companies, by refusing to reveal all their data, have essentially deceived the public into paying for worthless therapies. Now, with multiple grants from the Arnolds, he is leading an effort to build an open, searchable database that will link all publicly available information on every clinical trial in the world.
A number of the Arnolds’ reform efforts have focused on fixing nutrition science. In 2011 the science journalist Gary Taubes received an email from Arnold himself. Having spent more than a decade picking apart nutrition science, Taubes soon found himself cofounding an organization with a substantial grant from the Arnold Foundation, to rebuild the study of obesity from the ground up. And in 2015 the Arnold Foundation paid journalist Nina Teicholz to investigate the scientific review process that informs the US Dietary Guidelines. Just weeks before the federal guidelines were due for an update, Teicholz’s blistering report appeared in the prominent medical journal The BMJ, charging that the government’s panel of scientists had failed to consider evidence that would have done away with long-held worries about eating saturated fat.
And those are just a few of the people who are calling out iffy science with Arnold funding. Laura and John Arnold didn’t start the movement to reform science, but they have done more than anyone else to amplify its capabilities—typically by approaching researchers out of the blue and asking whether they might be able to do more with more money. “The Arnold Foundation has been the Medici of meta-research,” Ioannidis says. All told, the foundation’s Research Integrity initiative has given more than $80 million to science critics and reformers in the past five years alone.
McGovern’s quip about Enron, a company that hasn’t existed in 15 years, was a bit of a potshot. But given the long history of deep-pocketed business interests sowing doubt in research, his underlying question was a fair one: Who is John Arnold, and why is he spending so much money to raise questions about science?
Arnold is still quiet. “Usually the division of labor in most of our work is that I talk,” Laura Arnold says in a phone interview. By all accounts, Laura, who attended Harvard College and Yale Law School and worked as an oil executive, has been equally influential in setting the direction for the foundation. But when I visit the Arnold Foundation’s Houston headquarters in June, Laura has been called away on a family emergency, leaving John to do the talking. Arnold is 5’10”, trim, and blandly handsome, his unusually youthful appearance now somewhat concealed by a salt-and-pepper beard.
Arnold grew up in Dallas. His mother was an accountant (she would later help manage the books at his hedge fund). His father, who died when Arnold was 18, was a lawyer. By kindergarten, Arnold’s talent for math was apparent. “I think I was just born with a natural gift for seeing numbers in a special way,” he says. Gregg Fleisher, who taught him calculus in high school, recalls an occasion when Arnold instantly solved a math puzzle that had been known to stump PhDs. But he also stood out for his skepticism. “He questioned everything,” Fleisher says.
By the time he was 14, Arnold was running his first company, selling collectible sports cards across state lines. Those were the early days of the Internet, and he managed to gain access to an online bulletin board intended only for card dealers. The listings let him see that the same cards were sold at different prices in different parts of the country—which presented an opportunity for arbitrage. “Hockey cards didn’t have much of a market in Texas,” he tells me. “I would buy up all the premium hockey cards and send them to Canada or upstate New York.” He called the company Blue Chip Cards. Arnold estimates that he made $50,000 before he finished high school.
Arnold graduated from Vanderbilt University in 1995, taking only three years to finish his degree. He started working at Enron four days later. A year after that, at age 22, he was overseeing Enron’s Texas natural gas trading desk, one of the company’s core businesses.
Arnold’s work at Enron—seeking to capitalize on seasonal price differences in natural gas—wasn’t all that different from what he’d done as a teenager selling sports cards. In Hedge Hogs, a 2013 book about hedge fund traders, Jeff Shankman, another star trader at Enron, is quoted describing Arnold as “the most thoughtful, deliberate, and inquisitive person” he worked with on the gas floor. But Shankman recognized that he and Arnold were different in one key respect: Arnold had a greater appetite for risk, a quality that seemed at odds with his quiet demeanor. On some days at Enron, Arnold would trade more than a billion dollars’ worth of gas contracts. In 2001, even as Enron was collapsing amid an accounting scandal that covered up billions in debt, he was reported to have earned $750 million for the company. A former executive at Salomon Brothers later told The New York Times that there were very few incidents in the history of Wall Street comparable to Arnold’s success that year.
As Enron neared bankruptcy, executives scrambled to hold its operation together, offering bonuses to keep traders on board. Arnold was given $8 million, the biggest payout of all, just days before Enron filed for bankruptcy. He started Centaurus the next year, bringing along a small group of former Enron traders, who worked out of a single large room.
It wasn’t long before Arnold had the answer to his doubts. In 2006, Centaurus reportedly generated a 317 percent return overall, after taking the opposite side of a risky bet that another hedge fund, Amaranth, had made on fluctuations in natural gas prices. Amaranth, which was gambling with money from large pension funds, suffered a $6 billion loss and collapsed. By 2009, Centaurus was managing over $5 billion and had more than 70 employees. In its first seven years, according to Fortune, the fund never returned less than 50 percent.
THE ARNOLDS had been dabbling in philanthropy for years, supporting a few handpicked programs in education, criminal justice reform, and other areas that were important to them. But now, with their stepped-up ambitions, the couple entered a new realm. Arnold had always been ready to make huge bets, but it was ultimately his hunger for reliable data that made him a brilliant trader. That same hunger would make large-scale philanthropy far more challenging than he had anticipated.
Then, one day in November 2011, he was listening to the podcast EconTalk, hosted by libertarian economist Russ Roberts. The guest that day was science journalist Gary Taubes, and he was talking about how the prevailing dietary wisdom of the past 40 years—that eating too much fat leads to obesity and heart disease—arose from the flimsiest of scientific evidence. The foundational studies, Taubes said, looked at the diets and disease rates in various countries, then essentially guessed at which items in the diet were responsible for the country’s good or bad health statistics. Worse yet, whenever evidence came along that contradicted the consensus about the dangers of eating fat—often evidence that was much stronger than the evidence for the dangers—it was ignored or not even published. Hardly anyone in the world of nutrition science seemed willing to question the science behind the low-fat diet, even after Americans grew fat and diabetic in record numbers.
The picture Taubes painted wasn’t of a flawed study here or there but of a fundamentally broken scientific culture. During the podcast, he mentioned that he was raising money in the hope of funding experiments that might deepen our understanding of the root causes of obesity. Not long after the podcast went online, he received a five-line email from Arnold. “From the little I know about the science of nutrition, your study makes a lot of sense,” Arnold wrote. Like Nosek, Taubes had to Google Arnold to learn who he was. Six months later the Arnold Foundation made a $4.7 million seed grant to the Nutrition Science Initiative (NuSI), the nonprofit Taubes cofounded to support fundamental research on diet and health. The next year the Arnolds promised $35.5 million more. (wired wrote about NuSI in issue 22.09).
Arnold says that now, unless he trusts a researcher’s work, he no longer believes the findings of any scientific study until he or someone on the staff carefully vets the paper. “A new study shows …” are “the four most dangerous words,” Arnold wrote on Twitter.
The Arnolds had already decided that, with decades of life ahead of them and almost unlimited resources, they had the time and money to evaluate charitable programs properly, even when that meant paying for expensive randomized controlled trials that could take years to complete. But now they were widening their scope. If they wanted to embark on truly “transformational change,” as their foundation literature states, it wouldn’t be enough to properly evaluate this or that education or criminal justice program. They would also have to take on a far more ambitious project: The Arnolds would have to try and fix science itself.
In my conversations with Arnold and his grantees, the word incentives seems to come up more than any other. The problem, they claim, isn’t that scientists don’t want to do the right thing. On the contrary, Arnold says he believes that most researchers go into their work with the best of intentions, only to be led astray by a system that rewards the wrong behaviors. Says Goodman, “Scientists really do want to discover things that make a difference in people’s lives. In a sense, that’s the strongest weapon that we have. We can feed off that.” Figuring out exactly which rewards work best and how to simultaneously change the incentives for researchers, institutions, journals, and funders is now a key area of interest for Goodman and Ioannidis.
At the Center for Open Science, Nosek has already begun to experiment with new incentives for scientists. Because investigating and replicating research begins with having the data and materials necessary to do so, he is particularly focused on making science more transparent. In 2014 he partnered with the journal Psychological Science to offer colorful “Open Data” and “Open Materials” badges for papers that met specific criteria for sharing. A 2016 study to determine the effectiveness of the badges showed that the number of articles that reported publicly available data had increased tenfold. “It’s a stupid little badge,” Nosek says, but it works.
Nosek is also still campaigning to convince researchers to preregister what they plan to analyze and report in a study, so that they can’t adjust their experiment on the fly or hide less-than-dazzling results—a problem that Goldacre is also tackling. To promote preregistration, the Center for Open Science offered the first 1,000 scientists who preregister their studies with the organization $1,000 each. Nosek says that the cash offers were Arnold’s idea.
The Center for Open Science’s next big undertaking is another reproducibility project—this one for cancer studies—which recently revealed its initial findings (two of five studies yielded the same results the second time around). In 2012 the former head of cancer research at the biotech firm Amgen revealed the results of the company’s effort to replicate 53 “landmark” papers in hematology and oncology; only six studies’ findings could be confirmed. So there is already widespread concern about reproducibility in the field. The center’s replication efforts, in turn, have inspired economists and even tropical ecologists to plan reproducibility projects of their own.
Whether all this momentum will lead to transformational change decades from now is impossible to know. Arnold figures that some of his specific grants might not work out as planned. (The foundation’s funding of the Nutrition Science Initiative is now scheduled to end in November.) More generally, it may not be possible to truly reform a system where the incentives are already so deeply embedded. “It’s probably too big a lift for us to expect we’re going to change researchers who have been around for decades,” he says. Plus, systems of prestige and advancement die hard. “You don’t shift a culture overnight,” Nosek says. But as many Wall Street veterans can testify, betting against John Arnold is usually a bad idea.
Sam Apple (@samuelapple) teaches science writing at the University of Pennsylvania.