Jan Leighley from American University agrees. The small sample size of the original study “would have tanked the paper from consideration in a serious political science journal,” she says.

There are many reasons why researchers might be unable to replicate the results of an earlier study. It could be that the original experiment was flawed, and its results were a random fluke. Also, psychologists often tamper with details of their studies in ways that produce positive and publishable results, but also illusory and irreproducible ones.

Gerber doesn’t think any of that is necessarily happening here. Instead, he notes that there are many differences between his experiment and Bryan’s. They involved different people, elections, and years. Bryan used an online survey, while Gerber delivered his over the phone. The second study doesn’t necessarily mean that the verb-noun effect isn’t real, just that it might only show up in some situations and not others. “Failure to generalize might be a better phrase than failure to replicate,” Gerber says.

Bryan agrees. He thinks that Gerber’s study, though much larger, had some fatal flaws. First, the timing was off. Gerber’s team deployed their surveys a few days before their respective elections, whereas Bryan’s team did so the day before or the morning of the polls. “This kind of psychology is ephemeral,” he says. “If you do it days before, you might think, ‘Yeah I really should vote’, and then move on to something else. You have to do it at the point where people are making a decision, and going: Okay, where’s my polling station?”

Second, the stakes were lower. “We ran our study in two major elections that got a lot of media attention,” he says. “The elections that Gerber used… most of them didn’t matter, and nearly half were uncontested.” In this context, emphasizing one’s identity as a voter shouldn’t really matter. (Gerber counters that previous studies have shown that voter mobilization should be more effective, not less, in lower-stakes elections.)

“In that context, I don’t think the theory would have predicted a strong effect. They had little or no chance of getting useful results,” Bryan says. “I’ve heard of a number of political candidates who tried to apply the same idea, but most of the time, it was a significant enough deviation from how we did it that I wasn’t at all confident it would be effective. It highlights the perils of treating a complex psychological study as fortune cookie wisdom.”

He takes some responsibility for that. “The space in high-profile journals is limited, but we can all do a better job of more thoroughly articulating the theory behind our ideas,” he says. And that gulf between theory and practice can mean the difference between a wise intervention and a foolish one.

While Bryan was trying to get people to polls, Geoff Cohen (also from Stanford) was working to improve the fates of African American children. In 2003 and 2006, his team worked with 158 black seventh-graders from a Northeastern school. Half of them were randomly chosen to write about something that was important to them, from having friends to being musically adept. The other half wrote about something they deemed unimportant.