FAQ and Statement of Methodology
FiveThirtyEight.com
Revised 6/9/2008
Site/Meta
Who are you? My name is Nate Silver. For additional background, please see here or here.
What is the significance of the number 538? 538 is the number of electors in the electoral college.
How is this site different from other compilations of polls like Real Clear Politics? There are several ways that the FiveThityEight methodology differs from other poll compilations. Firstly, we assign each poll a weighting based on that pollster's historical track record, the poll's sample size, and the recentness of the poll. More reliable polls are weighted more heavily in our averages. Secondly, we include a regression estimate based on the demograhics in each state among our 'polls', which helps to account for outlier polls and to stabilize the results. Thirdly, we simulate the election 10,000 times for each site update in order to provide a probabilistic assessment of electoral outcomes.
How often is the site updated? Generally, the charts, graphs and polling averages on the site are refreshed once per day to reflect any new polls. Sometimes, there might not be any polling on a given day, and so an update will not take place. Other times, volume may be so heavy that multiple updates are necessary. You can tell that the charts and graphs on the site have been updated any time you see the "Today's Polls" tag in the footer.
What is your political affiliation? My state has non-partisan registration, so I am not registered as anything. I vote for Democratic candidates the majority of the time (though by no means always; my 2006 governor's ballot was cast for a Republican). I have been a supporter of Barack Obama in the Democratic presidential primary.
Are your results biased toward your preferred candidates? I hope not, but that is for you to decide. I have tried to disclose as much about my methodology as possible.
Does this site accept advertising? FiveThirtyEight.com is a commercial site and accepts advertising. Our preferred advertiser is BlogAds. To run an ad at FiveThirtyEight.com, please click here.
Why do you run ads for [insert name of candidate you don't like]? I believe in the right of free speech. Blogging is one form of free speech, and political advertising is another. If I believe an ad is particularly misleading, I will seek to block it, but otherwise, this site takes a non-partisan position toward which advertising it accepts.
How was the site designed? FiveThirtyEight.com is based on a Blogger.com template; the graphs are designed in MS-EXCEL 2007. Each of these platforms are more flexible than they are generally given credit for. Thanks also to Robert Gauldin for his design assistance.
The site isn't showing up properly in my browser. FiveThirtyEight.com should render reasonably well in the latest versions of Firefox and Internet Explorer. Older versions of Internet Explorer have pervasive problems with Blogger.com templates and are not recommended.
The site seems to freeze when I try and load the comments. If the comments are taking too long to load, click on the headline of the article in question. You will be taken to a new page where the comments will render in static form without additional loading time.
How do I contact you? I can be reached at 538dotcom@gmail.com.
Why haven't you responded to my e-mail? Between my various jobs and projects, I receive more e-mail each day than I'm able to respond to in full. However, I read each e-mail and very much appreciate both compliments and constructive criticism. Many of the new ideas and new features on the blog are a direct result of reader feedback. I appreciate your patience.
Are you hiring? Not presently, but if you're a talented programmer with an entrepreneurial bent and are willing to work relatively cheaply, there might be an opportunity down the line. Please e-mail me if you fit this description. Pollsters and Reliability Ratings
What is the reliability rating? It is a weight assigned to each poll based on three factors: the pollster's accuracy in predicting recent election outcomes, the poll's sample size, and the recentness of the poll.
How do you determine a pollster's reliability? For a very thorough explanation, see here.
OK, so just who are the most reliable pollsters? Pollsters are rated by their long-term pollster-introduced error (PIE). This is the amount of error that a pollster introduces to its results because of methodological imperfections, rather the inherent limitations associated with limited sample sizes and conducting poll far in advance of the election.
Current pollster ratings as of 5/28/08 can be found here.
How you do assess the reliability of other polling firms not included in the table above? These polls are treated as being slightly-below average and assigned a PIE of +2.11.
Are polls weighted by the number of respondents? Yes, although the methodology is a little involved. For a fuller explanation, see here.
How do you adjust for the recentness of a poll? Polls are treated as having a half-life of 30 days. Specifically, the weight assigned to each poll is...
0.5^(P/30)
...where 'P' is the number of days transpired since the median date that the poll was in the field.
How did you derive this recentness formula? It is based on an analysis of 2000, 2004, and 2006 state-by-state polling data. Previously, this formula varied based on the number of days until the general election, with the half-life becoming shorter as we got closer to the general election. After further investigation into the data, I discovered that there was really no empirically valid reason for doing this. The 30-day half life did an optimal job, or very close to optimal, across a broad range of time frames, ranging from the evening before the election to 250 days before the election.
Well, I still think you're making a mistake by using 'old' polls. It is your right to think that, but I'd challenge you to present a case based on the evidence. When I attempted to mimic the Real Clear Politics method -- including only the most recent poll from among pollsters who conducted surveys within 10 days of the election -- I found that the average error in my state-by-state projections would have increased by about half a point (from 2.4 points of error to 2.9) over 2000-2006. Also, note that apart from the recentness rating itself, the model has a couple of other mechanisms to tend to weight recent polls more heavily. For one thing, the density of polling becomes closer as you get closer to Election Day -- and so this will naturally tend to place more of the model's attention on recent polls. For another thing, our method has a second way of discounting for "old" polls, which is that it discounts progressively less recent polls from the same pollster (see below).
What do you do when you have multiple polls from the same polling firm? When a specific polling agency comes out with a new poll, we do not drop their previous poll. Instead, its sample sizes are aggregated for purposes of calculating the weight assigned to the poll, which has the effect of penalizing redundant polling data from the same firm. See the bottom one-third of this post for further discussion.
How do you handle tracking polls? I have not yet conceived of a solution for tracking polls, but it is a long time until we have to cross that bridge.
Does a poll ever become so old that you drop it entirely? Yes. Once a poll's weight falls below 0.05, it is dropped from the model for the sake of simplification and aesthetics.
How do you find the polls you include in the analysis? I periodically scan the links you see on the left-hand side of the page. If you've come across a poll that is not included in the analysis, please give it a shout-out in the comments for a recent blog post, and we will get it included in the next update.
Are there any polls you don't include? All scientifically-conducted polls are included provided that they meet our reporting requirements (see below). The lone exception is "leaked" internal polls, as campaigns may be selective about which polls they leak, biasing the results.
What are the reporting requirements for a poll? At a minimum, the poll must list (1) the percentage of the vote for each major candidate -- not simply the margin; (2) the sample size; and (3) the dates that the poll was in the field. We may temporarily list a "BREAKING" poll that is missing some of this information, but if it does not become available promptly, it will be delisted.
What precisely is indicated by the 'date' reported in association with the poll? It will indicate the median date of interviewing for that poll -- not when that poll was reported or posted to the site. For example, a poll which conducted interviews on July 1, July 2 and July 3, and was reported to the media on July 5, would be listed with a date of July 2.
What if a pollster provides multiple versions of their poll -- e.g. with or without third party candidates included, or different versions for registered and likely voters? When these situations arise, I use the likely voter version. I use the version with third-party candidates included if (i) they have officially announced their candidacy, and (ii) they are on the ballot in that state.
The 538 Regression Estimate and the Weighted Average
How is the weighted average determined in each state? It is calculated based on weighting each poll by its reliability score, plus including the results of the 538 regression estimate.
How much weight is given to the regression estimate? The regression results are treated as a single, recent poll of average reliability (see here for how I define 'average' in this context). Therefore, the regression estimate will have comparatively substantial weight in states with little polling data, but very little weight in states with robust polling data.
Why include the results of the regression analysis at all? Because polls are an imperfect measure of voter sentiment, subject to the vagaries of small sample size, poor methodology, and transient blips and trends in candidates' polling averages. For example, the late February Survey USA polls had Barack Obama ahead four points of John McCain in North Dakota, but behind by four points in South Dakota. Since North Dakota and South Dakota are very similar, it is unlikely that there is a true eight-point differential in how Obama polls in each state. The regression estimate is able to sniff out such discrepancies -- and so, for example, it recognizes that Obama is in fact an underdog in North Dakota. Put differently, it is a way not to be held hostage by the results of individual polls that might defy common sense, particularly where polling data in a state is sparse.
Which variables are included in the regression estimate? The regression model evaluates a total of 16 candidate variables. Variables are dropped via a stepwise process, until such time as each remaining variable is statistically significant at the 80% level or higher (t-score >= 1.3).
The 16 variables presently considered by the model are as follows:
Political1.
Kerry. John Kerry's vote share in 2004. Note that an adjustment is made in Massachusetts and Texas, the home states of Kerry and George W. Bush respectively, based on Al Gore's results in Massachusetts in 2000, and Bob Dole's results in Texas in 1996.
2.
$_Obama (Obama model only). The ratio of the amount of funds raised by
Barack Obama to that raised by John Kerry in each state (once again, an adjustment is made in Massachusetts).
3.
$_McCain. This is the corollary to #2 above: the ratio of John McCain fundraising to
George W. Bush fundraising. An adjustment is made in Texas.
4.
Partisan ID index. Per 2004 exit polls, the number of self-identified Democrats less the number of self-identified Republicans.
Religious Identity5.
Evangelical. The proportion of white evangelical protestants in each state.
6.
Catholic. The proportion of Catholics in each state.
7.
Mormon. The proportion of
LDS voters in each state.
Ethnic and Racial Identity8.
African-American. The proportion of African-Americans in each state.
9.
Hispanic. The number of Latino voters in each state as a proportion of overall voter turnout in 2004, as estimated by the Census Bureau. The reason I use data based on turnout rather than data based on the underlying population of Latinos is because Latino registration and turnout varies significantly from state to state. It is much higher in New Mexico, for instance, which has many Hispanics who have been in the country for generations, than it is in Nevada, where many Hispanics are new migrants and are not yet registered.
10. "
American". The proportion of residents who report their ancestry as "American" in each state, which tends to be highest in the Appalachians. See discussion
here.
Economic 11.
PCI. Per
capita income in each state.
12.
Manufacturing. The proportion of jobs in each state that are in the manufacturing sector.
Demographic13.
Senior. The proportion of the
white population aged 65 or older in each state. Because life expectancy varies significantly among different ethnic groups, this version has more explanatory significance than when lookig at the entire (white and non-white) population.
14.
Twenty. The proportion of residents aged 18-29 in each state, as a fraction of the overall adult population..
15.
Education. Average number of years of schooling completed for adults aged 25 and older in each state.
16.
Suburban. The proportion of voters in each state that live in suburban environments, per 2004 exit polls.
As of 4/26/2008, the regression coefficients are as follows:
Obama Model
Variable Coeff t-score
Kerry +.510 8.38
Evangelical -.683 -6.13
$_Obama +8.269 4.62
$_McCain -15.987 -4.03
AfricanAmerican +.409 3.78
Manufacturing +.583 3.21
Senior -.877 -3.03
Suburban -.108 -2.73
Catholic -.182 -2.40
"American" -.609 -2.32
Hispanic +.254 2.08
Education +4.419 1.39
Constant -38.015
Dropped: PCI, Mormon, Partisan, Twenty
Clinton Model
Variable Coeff t-score
Kerry +.726 14.19
$_McCain -14.190 -4.83
"American" +.869 3.62
$_Clinton +3.895 2.80
AfricanAmerican -.231 -2.74
Suburban +.092 2.70
Education -5.779 2.44
Mormon -.291 -2.19
Evangelical -.214 -1.99
Twenty +1.099 1.96
Catholic +.102 -1.59
Senior +.502 1.49
Manufacturing +.172 1.32
Constant +36.609
Dropped: PCI, Hispanic, Partisan
How often is the regression updated? The regression updates automatically based on the latest polling data. Periodically, I will also test out new variables for potential inclusion in the model.
Simulations and Win Probabilities
What is Win % or Win Probability? Simply, the number of times that a candidate wins a given state, or wins the general election, based on each of 10,000 individual simulation runs.
What is the purpose of the simulation runs? To account for three types of error in interpreting polling data: sampling error, state-specific movement, and national movement. Please see my thorough discussion here. The most important concept to grasp is that the error in predicting electoral outcomes is much larger at this stage of the election cycle than would be implied by the margins of errors from the polls alone. That is, the election may 'break' in any number of different and unpredictable directions, both at a state-by-state and at a national level. As we get closer to November 4, the potentiality for these trends will become lesser, and therefore the error assumed by my simulation model will become progressively less. However, even on election eve, the errors in predicting electoral outcomes are larger than those implied by each pollster's margin of error calculation alone.
Charts and GraphsWhat do the percentages mean on the chart along the left-hand side of the page? They are our estimate of the chances that Barack Obama and John McCain will win that state, respectively.
What is the significance of the 'regions' as defined on the state-by-state summary charts? There isn't any, other than as a way to present and organize the data. For additional discussion, see here.
What is the significance of the color of the state outline in the regional summary charts? They indicate the results from that state in 2004 on a blue-purple-red spectrum.
What is the 'Swing State Analysis'? There are two versions of the Swing State Analysis. One focuses on 'Tipping Point States', and the other on 'Must-Win States'.
'Tipping Point States' are those states that tip the election of the outcome from one candidate to the other. In each simulation run, the states are lined up from best to worst for each candidate. The states are marked off sequentially until the candidate reaches 270 electroal votes. The state responsible for putting the candidate over the top to 270 electoral votes is the tipping point state for that simulation run. For a fuller discussion see here.
The notion of the 'Must-Win States' is somewhat more intuitive. Simply put, it the percentage of the time that the candidate who won the election won that state in our simulation runs.
Miscellaneous Wonkery
How are ties broken? Ties (269 electors for both the Republican and Democratic candidates) are assigned to the Democrat based on the assumption that the Democrat would likely carry the day in the incoming House of Representatives. For additional discussion, see here.
Do you account for the potential for split electoral votes in Nebraska and Maine? Nebraska and Maine assign some of their electors based on the election results in individual congressional districts. The win probability and electoral vote averages do in fact account for these contingencies. This is somewhat relevant in this election, as Barack Obama looks to be competitive in both NE-1 and NE-2, while he will probably lose NE-3 (Western Nebraska) badly.
Do you account for home state effects, like in Arizona and Illinois? Directly, no, but indirectly yes. There is a very strong relationship between a candidate's home state and the amount of fundraising that they've received from that state. Since fundraising is one of the variables in our regression model, these effects will in turn show up in our weighted average for that state.
What if any assumptions do you make about turnout? I don't make any assumptions about turnout. The pollsters make various sorts of assumptions about turnout, and I rely on the pollsters. The only exception is in calculating the popular vote percentage shares for each candidate; for this purpose, I assume that turnout will be proportionate to what it was in 2004.
So is this your prediction about what will happen in the election? Not necessarily. The goal of the model is to do absolutely as much as it can with current state-by-state polling data. That is not exactly the same thing as accounting for external contingencies that might move the polling data (and, more importantly, the actual election result) in the future.
Do you have any plans to introduce polling averages for Senate races? Not currently, but it's a possibility down the road.