The use of quantitative policy analysis
Jonas Eliasson
Director of Transport Accessibility, National Transport Administration
Professor Transport Systems, Linköping University
Vice chair Governmental Expert Group for Public Economics (ESO)
Who am I?
Why do we need quantitative policy analysis in public decision making?��Can we trust transport forecasts?��Are conclusions robust?��Do results actually affect decisions?��Why are cost overruns so common, and what can be done?
Why do we need quantitative, systematic �policy analysis?
Why do we need quantitative policy analysis in public decision making?
Börjesson, M. och Eliasson, J. (2015) Kostnadseffektivitet i valet av infrastrukturinvesteringar. Rapport till Finanspolitiska rådet.
Mackie, P., Worsley, T. and Eliasson, J. (2014) Transport appraisal revisited. Research in Transportation Economics 47, 3-18.
“Anyone who has ever made the effort to understand a really useful economic model learns something important: The model is often smarter than you are.
The act of putting your thoughts together into a coherent model often forces you into conclusions you never intended, forces you to give up fondly held beliefs.
The result is that people who have understood even the simplest, most trivial-sounding economic models are often far more sophisticated than people who know thousands of facts and hundreds of anecdotes, who can use plenty of big words, but have no coherent framework to organize their thoughts.“
Paul Krugman
Krugman, P. (1999) The accidental theorist, Penguin UK.
Can you trust transport forecasts?
Forecasting effects of congestion charges
| Forecast | Actual |
Traffic across cordon | -16% | -20% |
Rush hours | -17% | -18% |
Public transport | +6% | +5% |
Eliasson, J., Börjesson, M., van Amelsfort, D., Brundell-Freij, K., Engelson, L. (2013) Accuracy of congestion pricing forecasts. Transportation Research A 52, 34-46.
Models are better at predicting behaviour �than people themselves
Congestion charges –
predicted and retrospective behavioural change:
Respondents’ own predicted change: ~5-10% less traffic
Actual measured change: ~30% less traffic
Respondents’ own reported change: ~5-10% less traffic
Eliasson, J. (2014) The role of attitude structures, direct experience and framing for successful congestion pricing. Transportation Research A 67, 81-95.
Long-term forecasts are more difficult
Road traffic forecasts
1970-2010
… mostly due to errors in input assumptions!
Road traffic forecasts
1970-2010
with correct input variables
Andersson, M., Brundell-Freij, K. and Eliasson, J. (2017) Validation of reference forecasts. Transportation Research A 96, 101-118.
More recent forecasts seem to have worked better
Road traffic forecasts
1993-2014
Are conclusions robust?
Benefit/cost ratios of 500 investment candidates
Top 150
The ”good” are much better than the ”bad” –
despite being shortlisted by professionals!
Sensitivity to scenario assumptions and benefit valuations
| Changes in Top 150 |
Double freight benefits | 14 |
Double safety benefits | 22 |
Double emission benefits | 5 |
Double person travel benefits | 11 |
Differentiated VoT’s | 5 |
Double oil price | 2 |
No plugin hybrids | 1 |
Trend break in car ownership | 2 |
Climate policy package | 3 |
Börjesson, M., Eliasson, J., Lundberg, M. (2014) Is CBA ranking of transport investments robust? Journal of Transport Economics and Policy 48(2).
Also: Asplund, D. and Eliasson, J. (2016) Does uncertainty make cost-benefit analyses pointless? Transportation Research A 92, 195-205.
Also: Trafikverket (2021) Förslag till nationell infrastrukturplan.
Do results actually affect decisions?
Benefit/cost ratios of investment candidates
Selected investments
Politicians’ selection
Transp.Admin’s selection
Runners
up
Excluded
Eliasson, J. and Lundberg, M. (2012) Do cost-benefit analyses influence transport investment decisions? Transport Reviews 32(1)
Eliasson, J., Börjesson, M., Odeck, J., Welde, M. (2015) Does benefit/cost-efficiency influence transport investment decisions? Journal of Transport Economics and Policy 49(3),
Benefit/cost ratios in the current plan
In the national plan
Outside the national plan
+10%
Roads
Railways
High-speed rail
Sea
BCR=1
Benefit/cost ratio
Emphasis on cost efficiency throughout �seem to filter out the ”worst” investments
Why are cost overruns so common, �and what can be done?
Most of cost increases occur in initial stages�(Welde & Odeck 2014)
Measured in cost overrun studies
This is
what matters!
Idea
National
Plan
Decision
to build
Finished project
Not all investment costs increase
Old cost estimates
New cost estimates
Trafikverket (2021) Förslag till nationell infrastrukturplan.
Causes of cost overruns
Eliasson, J. (2022) Kostnadsöverskridanden – storlekar och orsaker. Trafikverket PM.
What can be done?
If decisions are made on uncertain cost and benefit estimates:
Decisions need to be preliminary until late stages, when benefits and costs are less uncertain
But difficult (esp. for politicians) to overturn preliminary ”promises” , even if costs turn out be higher and benefits lower…
Eliasson, J. (2023) Tillbaka till framtiden – en nygammal planprocess. Trafikverket PM.
Why do we need quantitative policy analysis?�Confirmation bias, division of labour, cognitive limitations.��Can you trust transport forecasts?�Generally yes, but with several caveats. Use sensitivity analyses.��Are conclusions robust?�Rankings are usually surprisingly robust to uncertainties.��Do results actually affect decisions?�It depends, and it is hard to know – but probably often to a considerable extent. �Too few use them voluntarily: it may be painful to have to change your mind. ��Why are cost overruns endemic, and what can be done?�Several mechanisms. Most are unavoidable if final decisions are made too early. �Must be able to reconsider decisions when costs or benefits turn out to be different.