Prerequisite Skills for Reading Comprehension: Prerequisite Skills Annotation Guideline (Sugawara et al., ACL2017)
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
Annotate each question with skills, by the following steps:
2
Step1Make hypothesis using a question statement and candidate answer (annotators can see a correct one)
Example1:
Q: Why Tom looked angry? A: His sister ate his cake.
-> Hypothesis: Tom looked angry because his sister ate his cake.
Example2:
Q: When did French Revolution occur? A: In 1789
-> Hypothesis: French Revolution occured in 1789.
3
Step2Select sentences (from the context) required to provide the hypothesis
Example1:
Context: (C1) Tom is a student. (C2) Tom looks annoyed because his sister ate his cake. (C3) His sister's name is Sylvia.
Hypothesis: Tom looks angry because his sister ate his cake.
-> Select: C2
4
Step 3Select skills required for understanding an entailment from selected context sentences to hypothesis
Example1:
C2: Tom looks annoyed because his sister ate his cake.
Hypothesis: Tom looks angry because his sister ate his cake.
-> Skill: causal relation ("because"), bridging (lexical knowledge of "annoyed = angry")
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...
Main menu