Aqqu Error Analysis
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
Still loading...
ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
2
For more information on this document and the performed evaluation visit http://ad-publications.informatik.uni-freiburg.de/CIKM_freebase_qa_BH_2015.materials/
3
Note: this document lists all questions not answered correctly but we analyzed a sample of ~50 questions for each dataset, i.e. some questions have no error anlysis
4
5
Legend
6
7
AbbrevationDescription
8
QIDquestion idA unique ID for each question
9
BAPbest answer positionThe position holding the best answer, i.e. the one with highest F1. Note, that this is not necessarily always the one with F1=1.0 (especially on WebQuestions).
A BAP of "-" indicates there was no correct answer for this question.
A BAP of "NA" indicates the gold answer set was empty, i.e. the only way to achieve perfect F1 is to provide no answer as well.
10
Error/Categoryerror categoryOne or more error categories sorted alphabetically (see below). Sometimes several errors happen simultaenously, e.g. there might be a problem with Entity Recognition as well as relation matching. We always try to list all failure reasons. Lexicographical sorting allows the categories to be summed up easily.
11
DescriptiondescriptionA short description of the error reasons.
12
13
14
Error Categories
15
16
AbbrevationDescription
17
GTground truthAn error in the ground truth. For example, some questions have wrong answers or no answers.
18
DIRdirectionThe direction of the recognized relation is wrong, e.g., the semantics is "who are the parents of X" instead of "who is X a parent of"
19
ERentity recognitionAn error recognizing the entities mentioned in the question. Usually the correct entity is not matched at all or not matched strongly enough.
20
RMrelation matchingAn error matching/identifying the relations mentioned in the question. This can be both, not matching a relation at all, or not matching it well enough.
21
RANKrankingAn error caused by ranking. Note that it is not always clear that fixing, e.g., an ER error will result in a perfect answer w/o adjusting the ranking as well.
22
KBI
knowledge base inconsistency
An inconsistency in the knoweldge base (Freebase). For example, usually each relation is contained in Freebase in two directions, which is an assumption in our patterns.
23
INVinvalid patternOur patterns can only express questions with limited complexity, but sometimes a question requires additional restrictions we cannot express (e.g. reasoning over date spans).
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...
 
 
 
Legend
Free917
WebQuestions