Truncated thoughts: What went wrong with the presidential polling pundits?
What went wrong with the presidential polling pundits?
As I write this, it is 5 a.m. November 9, 2016. I got up to see if, perhaps, Clinton had pulled out an unlikely victory. She had not. Trump will be president.
This makes it a challenge to adjust for these nonresponse rates, which are not random. I’d worried earlier that there might be some decent sized pocket of Trump voters who weren’t admitting they were Trump voters, because they thought that was a socially unpopular thing to do. That would technically be a bias, and the bias would be similar (correlated) across all states and polls.
And the results are likely to be within the margin of error of the individual polls. But, still, let’s not gloss over the fact that, in the end, there was a failure.
Oddly enough, this failure seems to me to be similar to the error in financial modeling that was one of the causes of the faulty risk assessments prior to the financial crisis in 2008, that led us into years of recession. That’s a failure to accurately measure the amount of intercorrelation.
The election outcome is a surprise in that it contradicts two pieces of information: Pre-election polls and early-voting tallies. We knew that each of these indicators could be flawed (polls because of differential nonresponse; early-voting tallies because of extrapolation errors), but when the two pieces of evidence came to the same conclusion, they gave us a false feeling of near-certainty.
In retrospect, a key mistake in the forecast updating that Kremp and I did, was that we ignored the correlation in the partial information from early-voting tallies. Our model had correlations between state-level forecasting errors (but maybe the corrs we used were still too low, hence giving us illusory precision in our national estimates), but we did not include any correlations at all in the errors from the early-voting estimates. That’s why our probability forecasts were, wrongly, so close to 100%.
Put simply, if there is either a late surge for Trump in opinion, or there was a hidden batch of Trump supporters, or Trump supporters were more likely to show up to vote than expected by the models, these errors would not be random, they would be correlated. There would be more Trump votes across nearly ALL states. Similarly, if Hillary supporters were less likely to show up to vote than expected, this would be likely to affect nearly ALL states, not occur randomly.
Note Gelman is aware of this problem, but doesn’t feel that he adjusted completely enough for it.
“The damage was foreseeable and, in fact, foreseen. In 1998, before Li had even invented his copula function, Paul Wilmott wrote that "the correlations between financial quantities are notoriously unstable." Wilmott, a quantitative-finance consultant and lecturer, argued that no theory should be built on such unpredictable parameters. And he wasn't alone. During the boom years, everybody could reel off reasons why the Gaussian copula function wasn't perfect. Li's approach made no allowance for unpredictability: It assumed that correlation was a constant rather than something mercurial. Investment banks would regularly phone Stanford's Duffie and ask him to come in and talk to them about exactly what Li's copula was. Every time, he would warn them that it was not suitable for use in risk management or valuation.
So, in both the election forecasting and in the financial forecasting of those tranched mortgage securities we have a problem in not accurately understanding the correlations (and the stability of the correlations) between events. There are a lot of differences, of course, but still those high level similarities.
And there is the human tendency to overconfidence in predictions, which has been amply demonstrated many times, including even in a survey of Messy Matters blog readers (who tend to be professional statisticians) taking part in a survey called “Are You Overconfident”!
So, in both the financial collapse and in the 2016 election predictions we have an inability to accurately understand the correlation between events, combined with the bias toward overconfidence that seems to be a persistently human trait.
Regardless of our posthac understanding, we still had a deep recession after the financial collapse and we will still have Donald Trump as president. So, there may be understanding, but there will also be pain. Can we not have a gain in learning without pain?
I’m trying not to think about the more complex situation of climate models.
Fun Fact: Here's a surprising number. I downloaded the polls data from FiveThirtyEight (polls only forecast), and only looked at polls since Sept 1. The total sample size in these polls? 3,155,370 (these include polls done in states, mostly in swing states). That's a truly staggering number. So, while individual polls have a sampling error margin of error, the error in polling as a whole is due to nonsampling errors (most commonly summarized under the term "biases".
 Josh Katz, The Upshot, New York Times, 2016 Election Forecast: Who Will Be President, updated Monday Nov 7, 2016 6:58a.m.
 http://andrewgelman.com/2016/11/08/election-forecasting-updating-error-ignored-correlations-data-thus-producing-illusory-precision-inferences/ , accessed November 9, 2016.
 Felix Salmon “Recipe for Disaster: The Formula That Killed Wall Street” Wired, February 23, 2009 https://www.wired.com/2009/02/wp-quant/ , accessed November 9, 2016.