A Framework for Categorising AI Evaluation Instruments

Anthony G Cohn¹

José Hernández-Orallo²

Julius Sechang Mboli³

Yael Moros-Daval²

Zhiliang Xiang⁴

Lexin Zhou²

EBeM Workshop 2022 @IJCAI/ECAI 2022 24th July, Vienna

1 School of Computing, University of Leeds, UK; and the Turing Institute, UK�2 VRAIN, Universitat Politècnica de València, Spain�3 Faculty of Engineering and Informatics, University of Bradford, UK�4 IROHMS, School of Computer Science and Informatics, Cardiff University, UK

The background

• Approached by the OECD for their “Artificial Intelligence and the Future of Skills (AIFS) Project”
• Understand the potential as well as the limits of AI capabilities at a detailed task level to describe more precisely how humans and AI are complementary.

​

• Subsequently invited and funded to work on a framework to analyse the landscape of competitions and benchmarks
• Characterise EIs to capture strengths and weaknesses
• Not based on domains but on the values of a set of facets.

​

​

18 Facets to characterise EIs

Validity group

• Does it measure what we want to measure?
• Purpose, Capability, Reference, Coverage, Specifity, Realism

Consistency group

Fairness group

• Does it measure it effectively and verifiably?
• Judgeability, Containedness, Reproducibility, Reliability, Variation, Ajustability

• Does it treat all test takers equally?
• Antecedents, Ambition, Partiality, Objectivity, Progression, Autonomy

Why these 18? It was a long process that started was refining the facets and their possible values iteratively, as the first EI were evaluated.​

AERA, APA, NCME, et al., Standards for educational and psychological testing, 

Facet Values (Validity, Consistency, Fairness)

1. Purpose [RESEARCH, CONFIRMITY, OTHER (specify)]
2. Capability [TASK-PERFORMANCE (specify), CAPABILITY (specify)]
3. Reference [ABSOLUTE, RELATIVE (specify)]
4. Coverage [BIASED (specify), REPRESENTATIVE]
5. Antecedents [CREATED, ADAPTED (specify)]
6. Specificity [SPECIFIC, CONTAMINATED]
7. Judgeability [MANUAL, AUTOMATED]
8. Containedness [FULLY-CONTAINED, PARTIAL-INFERENCE (specify), NOT-CONTAINED (specify)]
9. Reproducibility [NON-REPRODUCIBLE, STOCHASTIC, EXACT]
10. Reliability [RELIABLE, NON-RELIABLE, N/A]
11. Variation [FIXED, ALTERED, PROCEDURAL]
12. Adjustability [UNSTRUCTURED, ABLATABLE, ADAPTIVE]
13. Antecedents [CREATED, RETROFITTED (specify)]
14. Ambition [SHORT, LONG]
15. Partiality [PARTIAL (specify), IMPARTIAL]
16. Objectivity [LOOSE, CUSTOMISED, FULLY-INDEPENDENT]
17. Progression [STATIC, DEVELOPMENTAL]
18. Autonomy [AUTONOMOUS, COUPLED (specify), COMPONENT]

​

23 Selected EIs : some examples

Introduced by Levesque et al

A recognising textual entailment task to test commonsense reasoning

(a replacement for the Turing Test?)

…                  …                        …                                                                     …

Example EI: Winograd Schema Challenge (WSC)

Introduced by Levesque et al

A recognising textual entailment task to test commonsense reasoning

(a replacement for the Turing Test?)

The city councilmen refused the women a permit because they [feared/advocated] violence

​

The trophy would not fit into the case because it was too [big/small] 

Analysis of Rater Consistency

Mercury

Mercury is the closest planet to the Sun and the smallest of them all

Facets Agreements for all 23 EIs

Analysis of Results

Mercury

Mercury is the closest planet to the Sun and the smallest of them all

Analysis of Results

Mercury

Mercury is the closest planet to the Sun and the smallest of them all

Summary

Potential future work

Proposed a framework for categorising EIs

• Initial analysis of 23 EIs
• Framework can be applied fairly reliably, but still some inter-rater disagreements

• Framework could be used by:
• EI designers
• AI system developers
• Multiple organisations
• Apply the framework to further EIs (now 36)
• Track the evolution of AI evaluation

​

Thank you!

Analysis of Results for 36 EIs

Mercury

Mercury is the closest planet to the Sun and the smallest of them all

Analysis of Results for 36 EIs

Mercury

Mercury is the closest planet to the Sun and the smallest of them all