Effective Altruism
UT
10/29 Forecasting & estimation
Today’s Agenda
A totally obscure unimportant prediction once said:
We judge that Iraq has continued its weapons of mass destruction (WMD) programs in defiance of UN resolutions and restrictions. Baghdad has chemical and biological weapons as well as missiles with ranges in excess of UN restrictions; if left unchecked, it probably will have a nuclear weapon during this decade.� –National Intelligence Estimate, October 2002
The Intelligence Community—including the CIA, the NSA, the Defence Intelligence Agency, and more—with a $50 billion budget and 20,000 analysts made a prediction that led to half a million deaths and a $3 trillion war.
Forecasting matters.
Philip Tetlock’s “Ten Commandments” for effective forecasting
1. Triage
• Focus on questions where your effort will pay off
• Avoid overly simple or overly complex problems
• Target “Goldilocks” zone questions for maximum impact
2. Break Problems into Tractable Sub-Problems
• Break down tough problems like Fermi
• Separate the knowable from the unknowable
• Make best guesses and learn from mistakes quickly
3. Balance Inside and Outside Views
• No event is 100% unique
• Compare seemingly unique events with similar ones
• Use outside-view thinking for better predictions
4. Update Beliefs Carefully
• Constant belief updating is key to forecasting accuracy
• Tease out signals from noisy information
• Avoid overreacting or underreacting to new evidence
5. Look for Clashing Causal Forces
• Acknowledge counterarguments in any situation
• Be open to the possibility of being wrong
• Watch for signs that challenge your views
6. Recognize Degrees of Doubt
• Learn to distinguish between varying levels of uncertainty
• Use numeric probabilities instead of vague hunches
• Patience and practice are essential for improvement
7. Balance Confidence and Prudence
• Manage the trade-off between decisiveness and caution
• Aim for calibration and resolution in your forecasts
• Avoid common forecasting errors: misses and false alarms
8. Conduct Honest Postmortems
• Own your mistakes without excuses
• Learn from both failures and successes
• Avoid hindsight bias in evaluating past decisions
9. Bring Out the Best in Others
• Master team management: perspective-taking, questioning, ______and confrontation
• Encourage constructive collaboration and ______open-mindedness
• Balance between rigid and decisive group dynamics
10. Master the Error-Balancing Bicycle
• Forecasting requires balancing opposing errors
• Learn by doing, not just reading or passively practicing
• Practice deeply and get clear feedback to improve
The Estimation Game
A Estimation Tool: GetGuesstimate.com
Guesstimate example
How many piano tuners are there in Chicago?