SCROLLS-Benchmark-Scores

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U
1	SubmissionDate	Contributors	ModelName	#Parameters	InputLength	Description	ScrollsScore	GovReport_Rouge1	GovReport_Rouge2	GovReport_RougeL	SummScreenFD_Rouge1	SummScreenFD_Rouge2	SummScreenFD_RougeL	QMSUM_Rouge1	QMSUM_Rouge2	QMSUM_RougeL	NarrativeQA_f1	Qasper_f1	ContractNLI_EM	Quality_EM	QualityHard_EM
2	2022-01-01T11:16:17.065Z	SCROLLS team	LED-base	162M	16K	LED baseline from the original SCROLLS paper, using 16384 input tokens	29.1567411	56.1688	26.5581	28.8213	24.247	4.5013	15.423	25.0822	6.7348	18.8469	18.5167	26.6449	71.5447	25.7519	25.3831
3	2022-01-01T11:15:26.776Z	SCROLLS team	BART-base	139M	1K	BART baseline from the original SCROLLS paper, using 1024 input tokens	29.01326537	47.8825	18.6243	22.6892	27.2061	4.9263	16.7072	30.1597	8.6591	20.7437	15.4418	26.3421	77.4271	25.9868	25.8621
4	2022-01-01T11:19:16.357Z	SCROLLS team	BART-base (512)	139M	512	BART baseline from the original SCROLLS paper, using 512 input tokens	27.58291512	45.598	16.8603	21.7532	26.2887	5.1108	16.2241	29.5309	8.151	20.0773	14.5204	24.6891	71.5925	26.8327	27.3946
5	2022-01-01T11:17:58.956Z	SCROLLS team	LED-base (1024)	162M	1K	LED baseline from the original SCROLLS paper, using 1024 input tokens	27.05940656	40.8798	16.0571	23.0977	22.6554	3.5503	15.1282	24.6112	6.532	18.9944	15.1731	24.3958	73.362	26.5508	27.2031
6	2022-01-01T11:17:10.719Z	SCROLLS team	LED-base (4096)	162M	4K	LED baseline from the original SCROLLS paper, using 4096 input tokens	28.30081092	52.4529	23.3029	26.8086	23.0142	4.1284	15.1053	26.5711	6.9412	19.9434	16.2961	24.9508	71.5447	26.5977	27.2989
7	2022-01-02T14:12:10.899Z	SCROLLS team	BART-base (256)	139M	256	BART baseline from the original SCROLLS paper, using 256 input tokens	26.35207105	41.9422	14.1951	20.2979	24.5465	3.7669	15.2906	29.9337	8.3135	20.4061	13.978	23.3174	69.7752	26.0338	25.7663
8	2022-01-01T11:20:26.412Z	SCROLLS team	Naive	-	-	Naive baseline from the original SCROLLS paper	19.34927021	45.2663	17.9077	20.8428	19.6269	1.7924	10.9921	14.2458	2.0045	9.2774	1.4519	3.445	65.95	25.23	26.0536
9	2022-03-14T19:57:44.110Z	Google Research	UL2	20B	2K		37.8737	53.5814	26.1358	28.8055	32.8671	7.7772	19.3589	31.1174	8.4681	20.4285	24.1724	37.5969	88.7135	45.7707	40.7088
10	2022-08-27T07:02:21.690Z	Ivgi et al.,	BART-large SLED	406M	16K	BART large SLED (c=256) with beam size 4	37.99	57.5	26.3	27.4	35.2	8.7	19.4	34.2	11	22	24.1	46.9	87.3	34.8	34.8
11	2022-08-21T18:18:37.955Z	Meta AI	BART-LS	460M	16K		39.76	59.4	29.8	30.8	37.7	10.2	21.5	35.1	11	22	26.2	48.7	87.1	37.8	34
12	2023-02-28T22:50:01.572Z	Google Research	CoLT5 XL		16K		43.51	61.3	32.2	33.8	36.4	10.1	21.7	36.2	12.9	24.2	31.1	53.9	88.4	48.1	43.8
13	2023-02-28T22:49:17.974Z	Google Research	CoLT5 Large		16K		41.04	60.7	31.3	32.9	36.7	10.6	22	34.9	11.5	23.1	27.7	49.8	88.7	39.9	36.8
14	2023-02-28T22:48:06.808Z	Google Research	CoLT5 Base		16K		37.64	58.7	29.6	31.4	34.5	9.2	20.6	32	9.3	21	23.3	42.1	86.5	36.5	34
15	2023-03-07T04:44:08.834Z	LongT5	LongT5 Base	220M	16K	LongT5 Base model SCROLL performance. All tasks have max output length 512 tokens, except for GovRep which has 1024 tokens.	38.6	57.7	30	31.4	34.8	9.6	21.1	33.9	11	22.8	23	46.6	85.6	37.9	36.6
16	2023-03-07T04:44:57.857Z	LongT5	LongT5 Large	770M	16K	LongT5 Large model SCROLLS performance. All tasks have max output length 512 tokens, except for GovRep which has 1024 tokens.	41.03	60.3	31.1	32.8	35.6	9.2	21.2	35.1	12	23.3	27.2	52.3	87.3	40.6	38.6
17	2023-03-07T04:42:02.675Z	LongT5	LongT5 XL	3B	16K	LongT5 XL model SCROLLS performance. All tasks have max output length 512 tokens except for GovRep which has 1024 tokens.	42.53	61.1	32.3	33.7	35.8	9.6	21.1	34.9	11.8	23.5	29.3	53.1	88.2	46	42.1
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100