ABCDEFGHIJKLMNOPQRSTUVWXYZAA
1
ORIGINAL REVIEW QUESTIONS
SUPPLEMENTAL FORECAST QUESTIONS
2
REVIEWER (COLLECTED IN SEPTEMBER 2022)
TIMELINES: By 2070, it will become possible and financially feasible to build APS systems.
INCENTIVES: By 2070, and conditional on TIMELINES above, there will be strong incentives to build APS systems.
ALIGNMENT DIFFICULTY: By 2070, and conditional on TIMELINES and INCENTIVES above, it will be much harder to develop APS systems that would be practically PS-aligned if deployed, than to develop APS systems that would be practically PS-misaligned if deployed (even if relevant decision-makers don’t know this), but which are at least superficially attractive to deploy anyway.
HIGH-IMPACT FAILURES: By 2070, and conditional on TIMELINES, INCENTIVES, and ALIGNMENT DIFFICULTY above, some deployed APS systems will be exposed to inputs where they seek power in misaligned and high-impact ways (say, collectively causing >$1 trillion 2021-dollars of damage).
DISEMPOWERMENT: By 2070, and conditional on TIMELINES, INCENTIVES, ALIGNMENT DIFFICULTY, and HIGH-IMPACT FAILURES above, some of this misaligned power-seeking will scale (in aggregate) to the point of permanently disempowering ~all of humanity.
CATASTROPHE: By 2070, and conditional on TIMELINES, INCENTIVES, ALIGNMENT DIFFICULTY, HIGH-IMPACT FAILURES, and DISEMPOWERMENT above, this disempowerment would constitute an existential catastrophe?
EXISTENTIAL CATASTROPHE
Link to review
AGI BY 2043: Will Artificial General Intelligence (AGI) exist by 1 January 2043?
AGI BY 2070: Will Artificial General Intelligence (AGI) exist by 1 January 2070?
AGI BY 2100: Will Artificial General Intelligence (AGI) exist by 1 January 2100?
EXTINCTION:
Assuming that Artificial General Intelligence (AGI) exists by 2070, will humanity either go extinct or have had its future potential drastically curtailed due to loss of control of AGI by 2200?
3
Superforecaster AI Expert #180.0%90.0%37.5%37.5%0.1%0.1%0.000009%
https://goodjudgment.io/AI/Review_1_Superforecaster_AI_Expert_1.pdf
4
Superforecaster AI Expert #210.0%100.0%60.0%1.0%0.0%5.0%0.0%
https://goodjudgment.io/AI/Review_2_Superforecaster_AI_Expert_2.pdf
5
Superforecaster AI Expert #390.0%60.0%70.0%30.0%70.0%98.0%7.7%
Declined to make public
6
Superforecaster AI Expert #467.5%72.0%70.0%7.0%5.0%6.0%5.0%
Declined to make public
7
Superforecaster AI Expert #570.0%30.0%40.0%80.0%12.0%20.0%0.2%
https://goodjudgment.io/AI/Review_3_Superforecaster_AI_Expert_5.pdf
8
Superforecaster AI Expert #69.0%95.0%50.0%2.0%50.0%95.0%0.04%
Declined to make public
9
Superforecaster AI Expert #780.0%95.0%80.0%20.0%33.0%20.0%0.8%
https://goodjudgment.io/AI/Review_4_Superforecaster_AI_Expert_7.pdf
10
Superforecaster AI Expert #835.0%100.0%100.0%0.001%1.0%50.0%0.000002%
Declined to make public
11
Superforecaster AI Expert #990.0%80.0%95.0%40.0%15.0%10.0%15.0%
https://goodjudgment.io/AI/Review_5_Superforecaster_AI_Expert_9.pdf
12
Superforecaster AI Expert #10 (note that this forecaster provided two different forecasts for each question, for two different definitions of "strategic awareness" -- I've here averaged them together)
91.5%89.0%49.0%0.054%29.0%78.0%0.1%
https://goodjudgment.io/AI/Review_6_Superforecaster_AI_Expert_10.pdf
13
Superforecaster AI Non-expert #160.0%95.0%50.0%50.0%20.0%98.0%5.0%
https://goodjudgment.io/AI/Review_7_Superforecaster_AI_Non-expert_1.pdf
14
Superforecaster AI Non-expert #25.0%50.0%50.0%25.0%40.0%95.0%0.1%
https://goodjudgment.io/AI/Review_8_Superforecaster_AI_Non-expert_2.pdf
15
Superforecaster AI Non-expert #360.0%70.0%5.0%60.0%1.0%95.0%"Less than 1%"
https://goodjudgment.io/AI/Review_9_Superforecaster_AI_Non-expert_3.pdf
16
Superforecaster AI Non-expert #460.0%70.0%10.0%60.0%30.0%50.0%0.4%
https://goodjudgment.io/AI/Review_10_Superforecaster_AI_Non-expert_4.pdf
17
Superforecaster AI Non-expert #590.0%40.0%40.0%40.0%4.0%100.0%0.2%
https://goodjudgment.io/AI/Review_11_Superforecaster_AI_Non-expert_5.pdf
18
Superforecaster AI Non-expert #685.0%95.0%100.0%100.0%20.0%99.0%12.0%
https://goodjudgment.io/AI/Review_12_Superforecaster_AI_Non-expert_6.pdf
19
Superforecaster AI Non-expert #720.0%90.0%40.0%35.0%20.0%99.0%0.5%
https://goodjudgment.io/AI/Review_13_Superforecaster_AI_Non-expert_7.pdf
20
Superforecaster AI Non-expert #815.0%20.0%No answer5.0%"Less than .5%"<.5% or 5%
"Well below .5%"
https://goodjudgment.io/AI/Review_14_Superforecaster_AI_Non-expert_8.pdf
21
Superforecaster AI Non-expert #990.0%99.0%50.0%15.0%1.0%75.0%0.1%
https://goodjudgment.io/AI/Review_15_Superforecaster_AI_Non-expert_9.pdf
22
Superforecaster AI Non-expert #1090.0%82.0%10.0%3.0%0.5%20.0%
20% or around 0%
https://goodjudgment.io/AI/Review_16_Superforecaster_AI_Non-expert_10.pdf
23
Superforecaster AI Non-expert #1135.0%85.0%20.0%50.0%10.0%99.0%0.3%
https://goodjudgment.io/AI/Review_17_Superforecaster_AI_Non-expert_11.pdf
24
Median calculated via this spreadsheet (note that some cells do not have numbers)67.5%85.0%50.0%30.0%13.5%76.5%0.3%
25
26
Superforecaster Aggregation May 2023 (note: the numbers above were from Superforecasters doing their work in isolation, after which point all the questions were posted on a platform where superforecasters could act as a team, challenge/question each other's views, and make more updates. Plus other superforecasters added their judgments. The questions were then re-opened in May 2023 for a final updating round, the aggregation here is the median of the last forecast made by everyone at that time.)
80.0%90.0%58.0%25.0%5.0%40.0%1.0%12.0%40.0%60.0%6.0%
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100