Lecture 9: Combining Forecasts

Jacob Steinhardt�Stat 157, Spring 2022

Revisiting Reference Classes

“What is the probability that Joe � Biden completes his full term?”

�

​

�(For simplicity, assume we were trying to forecast at the beginning of his term.)

Revisiting Reference Classes

• Focus on two cases for simplicity: dying in office, and resignation/impeachment
• To get more data, let’s look at US, Canada, UK, France, and Spain

Deaths in Office

• US: 8 since 1789 (8/233 years, or 3.4%/year)
• 0 since 1970
• Canada: 2 since 1878, plus one who resigned in illness (2.5/144 = 1.7%/year)
• 0 since 1970
• UK: 4 since 1800 (4/222 = 1.8%/year)
• 0 since 1970
• France: 4 since 1800 (4/22 = 1.8%/year)
• 1 since 1970 ( =1.9%/year)
• Spain: 4 since 1800 ( =1.8%/year)
• 2 since 1970 ( =3.8%/year)

Combining Reference Classes

���������How should we combine this information? Take average? Which time frame?

Country Similarities and Differences

First Approach: Make up Weights (“Ensembling”)

• US: 5, Canada, UK: 4, France: 3.5, Spain: 0.5
• 1970+: 4, Total: 1�������
• Final average:�0.8%/year

A More Principled Approach

Basic approach misses some things:

• We know “Total” should overestimate the 1970+ period
• Some classes have larger/smaller sample size

​

Next: a more sophisticated approach using probability models

• Use this to understand what factors matter, practice, then go with your gut

Ensembling with Probabilities

• π: true probability�
• π_j: probability for reference class j�
• X_j: observed �count for �reference class j

Simplest Model: Gaussians

• π_j ~ Gaussian(π, 𝜎_j²)
• What if we know π_j is an over/underestimate?
• π_j ~ Gaussian(π+𝛥_j, 𝜎_j²)
• 𝛥_j: bias, 𝜎_j²: uncertainty (variance)�
• Maximum likelihood estimate for π: (on board)�

What about X_j? (On board)

• We have X_j ~ Binomial(π_j, N_j), where N_j is total # of “trials” (e.g. 222 years)�
• Gaussian approximation of binomial: X_j ~ Gaussian(N_j*π_j, N_j*π_j*(1-π_j))�
• Let π_j’ = X_j / N_j. Then simplifies to π_j’ ~ Gaussian(π_j, π_j*(1-π_j)/N_j)�
• Trick: estimate variance with empirical variance�
• Simplify to single Gaussian: π_j’ ~ Gaussian(π+𝛥_j, 𝜎_j² + π_j’*(1-π_j’)/(N_j-1))

Rule of Thumb

Adjusted standard deviation dominated either by a priori uncertainty (𝜎_j) or by sample noise (roughly sqrt(X_j) / N_j)

If X_j = 0 then want to adjust slightly (roughly 1/N_j or perhaps 0.5/N_j)

Applying the Probabilistic Approach

Final ensemble answer (see spreadsheet): 0.64%/year

Your Turn! Reference Classes for Resignation

• US: 1 resignation / 233 years (1974)
• Canada: many dissolutions of government; 1 prime minister resigned due to scandals (1873)
• UK: many dissolutions of government; 3 were due to scandals (1855, 1857, 1924), but none involved the prime minister resigning

​

How would you combine this information?

Other Uses of Ensembling

• External opinions: average many “forecasts” together
• Weight by track record�
• This is how I (Jacob) form most opinions about the world!
• Key: pick good people to “trust” in each area, and occasionally find ways to vet track record�
• Also another reason why forecasting in groups, or examining a problem from many angles, is helpful