Artificial Intelligence and Validity
Center for Evaluation & Assessment
The Gevirtz School
High quality evaluation
is not a technocratic exercise!!!
AI & Validity
Quantitative Validity | |||
Stages | Types of Validity | Evaluator/ Researcher | Artificial Intelligence |
Methods | Internal validity | Yes | Not Yet |
| External validity | Yes | Yes |
Measurement | Construct validity | Yes | Maybe |
| Concurrent validity | Yes | Yes |
| Predictive validity | Yes | Yes |
Analysis | Statistical conclusion validity | Yes | Yes |
Qualitative Trustworthiness | |||
| Types of Validity | Evaluator/ Researcher | Artificial Intelligence |
| Credibility | Yes | Not Yet |
| Transferability | Yes | Maybe |
| Dependability or trustworthiness | Yes | Not Yet |
Concepts and examples within educational assessment
From: Dorsey, D. W., & Michaels, H. R. (2022). Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment. Journal of Educational Measurement, 59(3), 267-271.
“AI in the form of Natural Language Processing (NLP) methods can “score interviews, essays, or other forms of text-based assessments, while overlooking the fact that language/text exists at multiple levels of abstraction: syntactic, lexical, semantic, and pragmatic, involving a host of construct-based inferences—ones that involve complex interactions between culture and language” pg. 268
Hype Cycle
Source: https://www.gartner.com/en/research/methodologies/gartner-hype-cycle
Thank you