This is the website of Abulsme Noibatno Itramne (also known as Sam Minter). Posts here are rare these days. For current stuff, follow me on Mastodon



June 2024

A final look back at my 2012 election models

First of all, I promised to report back after the electoral college vote, just to give the real final results, but I didn’t. There were no “faithless electors” this time around, as fun as that always is, so the end result was as expected from election night. Obama 332, Romney 206.

Now on to the analysis of the accuracy (or lack there of) of my five poll averages:

While my analysis predicted 56 of 56 jurisdictions correctly (50 states, DC, ME & NE congressional districts), this is not the whole story. It is worthwhile to also look at the comparison between the final five point averages and the actual popular vote margins in each of these jurisdictions. Here is what you find:

Screen Shot 2013-07-07 at 22.00.54094

There is no clear overall bias toward either Obama or Romney. The overall count gives 25 jurisdictions where my 5 poll average overestimated Obama’s support, and 31 jurisdictions where my 5 poll average overestimated Romney’s support. If anything, on an overall basis then, there is a slight bias in favor of Romney. But that isn’t the real bias here.

Lets look at this again partitioning based on who won.

  • Romney won 27 jurisdictions. Of those, 7 overestimated Romney’s support, while 20 overestimated Obama’s support.
  • Obama won 29 jurisdictions. Of those, 5 overestimated Obama’s support, while 24 overestimated Romney’s support.

So the overwhelming pattern here is that although there are clearly exceptions, in general, the polls overrepresented the support level of the person who lost the state.

In other words, the polls are not biased toward either candidate. The polls are biased toward showing a CLOSER RACE than the actual results.

The trend can been seen even more clearly if you subtract out the actual results, leaving only the deviation or bias:

Screen Shot 2013-07-14 at 21.13.12403

If the polls were very generally right, you would expect a horizontal line very close to zero.

If there was a direct bias for one candidate or another you would expect a horizontal line, above zero for an Obama bias, below zero for a Romney bias.

We see positive to the left and negative to the right though, meaning the bias is generally toward the underdog, whoever that is. (In other words, again, a bias toward a closer race.)

More than that though, the less close the race is, the greater that bias is. It isn’t just that races that are less close are polled less often and so are generally further from the right answer, but rather the bias toward making it closer increases too.

In Hawaii, where it was clear Obama was going to win by very high margins, the polls overestimated Romney by 14.7%! (Polls said Obama would win by 28%, Obama actually won by 43%.)

Similarly, in West Virginia, where it was clear Romney was going to win by very high margins, the polls overestimated Obama by 11.4%! (Polls said Romney would win by 15%, Romney actually won by 27%.)

Those were the two worst states in terms of accuracy of the five poll average. Rounding out the worst five are Alaska (R+9.6%, a big exception to the trend… where there were no polls at all and the 5 poll average was based on the 2004 and 2008 elections), Tennessee (O+9.4%), Kentucky (O+8.2%).

The best five were: Maine-1 (O+0.03%), Texas (R+0.2%), Virginia (R+0.2%), Washington (O+0.4%), Georgia (R+0.4%).

Now, in the end, the degree to which each candidate wins (especially in the states they are way ahead in) doesn’t really matter. We have a winner take all system, so this bias actually makes a lot of sense in terms of how pollsters work. Having the margin wrong doesn’t matter much in these kinds of races, only the winner matters (and the poll averages were all correct on that front). Also of course, close races are more interesting, and people are more likely to commission polls if it looks closer… so the biases all make sense.

But lets take a closer look at the states that were even remotely in contention, where the margins were under 15%. That actually includes a lot of places I wouldn’t really consider “close” but it gives a good number to look at:

Screen Shot 2013-07-14 at 05.31.28514

Now, I had categorized using “Strong Romney”, “Weak Romney”, “Lean Romney”, “Lean Obama”, “Weak Obama”, “Strong Obama”. Lets look at how things stacked up in terms those categorizations… specifically the “lean” states.

You can see from the chart above, that even among the states near the “tipping point” where you would think polling would be most robust, the error is significant. The tipping point is of course on Obama’s side, since he won. But what we see is that of states that the last five poll average identified as “Lean Obama”… that is, Obama ahead, but with a margin under 5%… so close enough you could conceivably think Romney might take it… only TWO actually ended up with margins under 5%. That would be Florida and Virginia.

On the other hand, the five poll average identified each of these states (and one congressional district) as “close”, but in fact it was pretty much an Obama blow out:

5 Poll O-R Actual O-R Bias
Colorado 2.4% 5.5% R+3.0%
Pennsylvania 4.8% 5.4% R+0.6%
New Hampshire 2.8% 5.6% R+2.8%
Iowa 2.0% 5.8% R+3.8%
Nevada 4.8% 6.7% R+1.9%
Maine-2 2.8% 8.6% R+5.8%
Michigan 4.5% 9.5% R+5.0%

There was only one state with the opposite pattern… that is, the five poll average indicated Obama with a healthy safe lead of more than 5%, and the actual results were a win by less than 5%. That would be Ohio. The 5 Poll average had Obama ahead by 5.5%. In actuality he won by only 3.0%

On the Romney side, there were less close places to look at. There were no cases where the five poll average classified something with Romney having a greater than 5% where it ended up being closer than that. There was ONE case where the five poll average showed a close race where it really wasn’t. That would be Nebraska-2, where the five poll average showed only a 3.8% lead. In fact, Romney won Nebraska-2 by 7.2%.

The lesson in all this for 2016? Probably just that “Lead under 5%” isn’t enough to actually think that a state has a real possibility of going either way, that it is really up for grabs. Especially if the form of bias evident here (bias toward the races looking closer than they really are) repeats next time around, which I think is a reasonable assumption.

Some tighter criteria is needed. Maybe just a narrower percentage lead, but maybe also something else. Perhaps an actual history of the lead bouncing back and forth, or the actual current set of polls including results going both way.

Of the five closest states based on actual results (Florida, North Carolina, Ohio, Virginia, Colorado) only Florida had even one of the last five polls indicating the opposite winner (although North Carolina and Colorado had last minute polls showing a tie). Of course five results showing a lead of 0.1% should probably be considered differently than 5 results showing a lead of 4.9%.

The more mathematically proper thing would probably be to take into account the margins of errors on each poll, and come up with an actual probability of the result being on the opposite side than the simple five poll average. There is a place for more complicated models (and I love them!) but part of the point of my analysis is that you can come up with results that are essentially just as good through very simple models with no fancy stuff, just simple averages.

If we went by a rule saying we would only consider a state to be a true toss up if at least one of the last five polls showed a different result, then on election eve 2012, the only Toss-Up state would have been Florida.

I have yet to decide what I will do in 2016, but considering Florida to be the only true toss-up seems closer to the truth than the analysis that put the seven entities in the table above (and NE-2) in the toss-up category.

This analysis is also posted on my Election 2012 Wiki where you can see the original analysis as well.

OK… now to start setting up the 2016 analysis… :-)

Electoral College: Nov 10 18:00 UTC – Florida finally called – For Obama – 56 of 56!

Just before 18:00 UTC today, Florida was finally “called” for Obama. Absent any faithless electors, this makes the final electoral college results Obama 332, Romney 206.

As those of you following my electoral college tracking this year know, not only was that the final state of my “Current” line on election day, but it is also by far the most common location of the race in the daily updates covering the whole year. As I’d said quite a few times, 332-206 just seemed to be where this race “wanted” to be. Sometimes events would move the polls slightly further in Obama’s direction, sometimes slightly further in Romney’s direction, but 332-206 seemed to attract the race like a magnet, and things would revert back to this state. When things “reverted to the mean” this is where they went.

Also of course, looking at the daily updates going back to January, Romney was NEVER ahead in this electoral college analysis… or any other electoral college analysis. Aside from a few days in October where Romney was threatening to take the lead (but never did) the question was never if Obama would win, but rather by how much. But yet reports are that the Romney campaign, and Romney himself, were shocked by the fact that they lost. They really truly did not believe the overwhelming consensus from the pollsters.

I guess to some degree to run a national campaign like this you HAVE to make yourself think you are winning, but still…

One final point. There are quite a few more complicated models out there doing electoral college analysis. They provide potentially more detail and more kinds of insight than mine, but part of my point in doing this exercise is that even a very simple “last five poll” average can get you very good results. The marginal value from added complexity doesn’t really seem to get you that much more. This isn’t to say that there isn’t value in that complexity. There is. And if I had more time, I’d certainly be tempted to make a more complex method of analysis myself.

But the simple average still got 56 out of 56 right. (That would be the 50 states plus DC, plus the congressional districts in Maine and Nebraska.)

That ain’t bad.

Note added Nov 11 00:25 UTC: I have some final clean up to do on it yet, but the archival wiki page with all the information from this general election season is here. The earlier wiki archiving the primary season is here.

Edit Nov 11 00:34 to change some awkward wording.

Electoral College: 06:15 – Alaska for Romney, Only Florida Left

Romney Obama
Romney Best Case 235 303
Current Status 206 332
Obama Best Case 206 332

I needed to pause to take my wife to the airport, so I couldn’t make this update earlier, but shortly after 06:00 UTC, Alaska was called for Romney.

This means so far the five poll average has called 54/54 contests. Only Florida is left. Even if it ends up missing on Florida, that is pretty good… It is better than it did four years ago.

If I do this again in four years, I may end up doing some things differently based on things I’ve learned this time around, but for the most part I’d say this exercise has been a success. And lots of fun, if somewhat exhausting in the last month.

I have no idea if Florida will end up being called in the next few hours, or if it will end up going almost 15 days like Missouri did in 2008.

Given that, I think it is time to call it a night. I’ll do another update sometime after Florida is called, although it may not be immediate given I’ll now start spending time on non-election things as well. :-)

Thanks again for following my coverage this year. It has been a blast!

Electoral College: 06:00 – Nebraska’s remaining congressional districts to Romney

Romney Obama
Romney Best Case 235 303
Current Status 206 332
Obama Best Case 206 332

The two remaining Nebraska congressional districts were called for Romney shortly before 06:00 UTC.

As of then, that left Alaska and Florida outstanding, but only Florida was ever really in question.

And this wraps up the live coverage of Election 2012 from I have to take my wife to the airport now.

Alaska was actually called for Romney already, and I’ll include that in a 6:15 update when I get back.

And then we wait for Florida to tell us if we do indeed end up at the magical 332-206, or if we end up at 303 to 235 instead.

I’ll make a final update once the results in Florida are known.

And then a really final update once the electoral college votes in December, just in case there are any faithless electors.

But for now, thanks for joining for the election coverage all year long.

Sam out.

Electoral College: 05:45 – Virginia also for Obama

Romney Obama
Romney Best Case 235 303
Current Status 206 332
Obama Best Case 205 333

The five poll average’s streak continues.

Electoral College: 05:30 – Colorado and Maine 2nd for Obama too

Romney Obama
Romney Best Case 248 290
Current Status 206 332
Obama Best Case 205 333

Colorado and Maine’s 2nd also go to Obama. Still haven’t missed one.

Electoral College: 05:00 – Nevada to Obama too

Romney Obama
Romney Best Case 258 280
Current Status 206 332
Obama Best Case 205 333

At 04:49 UTC CNN called Nevada for Obama. The five poll average still hasn’t missed, but there are still a few states left.

Electoral College: 04:30 – Expected States push Obama over 270

Romney Obama
Romney Best Case 264 274
Current Status 206 332
Obama Best Case 205 333

Between 04:15 and 04:30 UTC, three more states were called. Missouri for Romney, and Oregon and Ohio for Obama. All three of these were fully expected, but Ohio put Obama over 270.

CNN waited until they called Ohio at 04:18 UTC to call the election, even though based on the polls by the time we got to election day, Ohio wasn’t even close. Once Iowa was called 9 minutes earlier, Romney had no more ways to win.

At this point, the only question is the margin of Obama’s victory.

As of this update, the five poll average hasn’t missed a state yet. Four states and two congressional districts to go.

(Of the close ones… there is also Alaska and Nebraska’s 3rd, but those will certainly go to Romney.)

Edit 2012 Nov 7 05:13 to add that last parenthetical note.

Electoral College: 04:15 – Romney’s Paths Gone, Obama Wins!

Romney Obama
Romney Best Case 264 274
Current Status 206 332
Obama Best Case 205 333

A bunch of states were called between 04:00 UTC and 04:15 UTC:

  • Obama: CA, HA, WA, WI, IA NM
  • Romney: ID, MT, NC

Only one of these mattered. Iowa. Iowa was a close state. Romney needed every close state he had left in order to win. Without Iowa, even if Romney swept every other close state (and Maine’s 2nd) he would only get to 264.

When CNN called Iowa at 04:09 UTC, I called the race on twitter in these two tweets. Nine minutes later when some more states came in, CNN called the election for Obama. They were slow. More on those next states in the next update.

Anyway, Obama wins reelection!

Electoral College: 04:00 – Some Expected States

Romney Obama
Romney Best Case 270 268
Current Status 206 332
Obama Best Case 190 348

With this update, there are no surprises, and no close states called, so Romney hangs on to his one glimmer of hope…

But the 04:15 update is coming soon, and the news is not good for Romney…