ABCDEFGHIJKLMNOPQRSTUVWXY
1
2
To calculate Brier scores based on the 10 percent, 50 percent and 90 percent confidence intervals, I took the highest percentage (denoted by a coloured box) and Brier scored based on this percentage and whether or not the milestone had happened. See below example.This gives an ok approximation but skews the expert forecasts towards being more conservative and unfairly penalises a lack of precision. It could be improved, with effort, by estimating all the probability density functions and scoring based on them (e.g. where does 2023 fall on the experts aggregates cumulative probability of milestone X happening?).
3
4
5
6
7
Milestone/ confidence of it happening by date10 percent50 percent90 percentTrue by Jan 2023? (5.5 + 1 years)Score 10%Score 50%Score 90%
8
Class24.56.5TRUEN/A(1 - 0.5)^2 = 0.25N/A
9
One-shot4.5820FALSE (0.1 - 0)^2 = 0.01N/AN/A
10
11
Brier score = (0.25 + 0.01) / 2
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100