BioLaySumm Evaluation Process Worked Example

	B	C	D	E	F	G	H	I	J
1
2	Subtask 1 - Lay Summarisation
3
4	1. Average metric scores will be calculated independently for the lay summaries in the test set of each dataset (PLOS and eLife).
5
6	PLOS
7	Submission	Relevance				Readability		Factuality
8	Submission	Rouge1	Rouge2	RougeL	BERTScore	FKGL	DCRS	BARTScore	FactCC
9	s1	44.31	44.31	44.31	44.31	12.89	11.87	75.33	41.32
10	s2	42.12	13.21	42.23	83.53	14.12	12.43	69.32	42.65
11	s3	41.42	12.11	39.22	81.52	12.43	11.2	74.23	48.53
12	s4	39.09	11.23	39.6	82.88	11.24	11.67	73.23	45.65
13
14	eLife
15	Submission	Relevance				Readability		Factuality
16	Submission	Rouge1	Rouge2	RougeL	BERTScore	FKGL	DCRS	BARTScore	FactCC
17	s1	45.23	12.21	42.65	85.64	10.21	10.42	74.32	50.34
18	s2	47.54	14.63	44.64	84.21	11.53	10.89	70.23	47.53
19	s3	42.32	13.54	40.64	80.53	9.76	9.32	71.34	55.23
20	s4	38.53	10.53	39.54	82.88	9.53	10.02	64.23	42.12
21
22
23
24	2. To obtain the leaderboard, the metric scores for each dataset will be averaged. This is what will be visible to participants on CodaLab during the test phase.
25
26	Submission	Relevance				Readability		Factuality
27	Submission	Rouge1	Rouge2	RougeL	BERTScore	FKGL	DCRS	BARTScore	FactCC
28	s1	44.77	28.26	43.48	64.975	11.55	11.145	74.825	45.83
29	s2	44.83	13.92	43.435	83.87	12.825	11.66	69.775	45.09
30	s3	41.87	12.825	39.93	81.025	11.095	10.26	72.785	51.88
31	s4	38.81	10.88	39.57	82.88	10.385	10.845	68.73	43.885
32
33	Note that the aim is to maximise all Relevance and Factuality scores, and minimise Readability scores.
34
35
36
37	3. After the test phase is complete, we will compute an average score for each individual aspect (i.e., Relevance, Readability, Factuality).
38
39	a) To do this we will first normalise metric values (using min-max normalization), so they share a common value range (0-1).
40
41	min-max normalisation = (score_i - min(score)) / (max(score) - min(score))
42
43		Rouge1	Rouge2	RougeL	BERTScore	FKGL	DCRS	BARTScore	FactCC
44	min	38.81	10.88	39.57	64.975	10.385	10.26	68.73	43.885
45	max	44.83	28.26	43.48	83.87	12.825	11.66	74.825	51.88
46
47	Submission	Relevance				Readability		Factuality
48	Submission	Rouge1	Rouge2	RougeL	BERTScore	FKGL	DCRS	BARTScore	FactCC
49	s1	0.9900332226	1	1	0	0.4774590164	0.6321428571	1	0.2432770482
50	s2	1	0.1749136939	0.9884910486	1	1	1	0.1714520098	0.1507191995
51	s3	0.5083056478	0.1119102417	0.09207161125	0.8494310664	0.2909836066	0	0.6652994258	1
52	s4	0	0	0	0.9476051866	0	0.4178571429	0	0
53
54
55
56	b) Then we will compute an aspect-level score by averaging across the normalized scores of the revelant metrics.
57
58	Submission	Relevance	Readability	Factuality
59	s1	0.7475083056	0.5548009368	0.6216385241
60	s2	0.7908511856	1	0.1610856047
61	s3	0.3904296418	0.1454918033	0.8326497129
62	s4	0.2369012966	0.2089285714	0
63
64	In this case, s2 is best for Relevance, and s3 is best for Readability and Factuality.
65
66
67
68	4. To determine an best system across all three aspects, we simply calculate their cumulative rank across each aspect (lowest == best).
69
70		Rankings
71	Submission	Relevance	Readability	Factuality	Cumulative Rank
72	s1	2	3	2	7
73	s2	1	4	3	8
74	s3	3	1	1	5
75	s4	4	2	4	10
76
77	In this case, s3 obtains the lowest cumulative rank, so would be considered the best overall system.
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100