The last three decades have witnessed substantial advances in the development of educational technologies, among them Intelligent Tutoring Systems, including the orchestration of learning interactions, augmenting human capacity, and revealing learning processes. ITSs typically support learning and instruction by leveraging and integrating student, domain, and pedagogical models (e.g., predictive AI algorithms with feedback). While many ITSs have demonstrated success in matching or exceeding human tutoring outcomes, there are nonetheless relatively few ITSs implemented at scale, and many ITSs have been stripped of their intelligence in order to take them to scale. One reason for the dearth of large-scale adoption can be attributed to an incompatibility between educational practice (e.g., covering curricula within the context of a time-limited classroom) and intelligent tutoring (e.g., adapting curricula and pacing to the needs of individual learners). This misalignment might be viewed as a fatal flaw in the conceptualization of ITSs. However, from a different angle, the development of ITSs offers contributions well beyond the end product - beyond an educational platform to support learning and instruction.  This talk will describe how the multiple facets of ITS development have contributed to educational practice, the design of educational learning environments, and our understanding of learning processes.  Dr. McNamara describes the contributions of two ITSs, iSTART and Writing Pal, in terms of five facets of educational technologies: interface design, pedagogy, learner engagement, feedback, and feasibility. She will also discuss the need to consider the future of ITS, and more generally digital learning platforms, including the need for large-scale R&D infrastructures that link DLPs and enable research at scale within authentic learning environments.  

Copyright © 2021 Arizona Board of Regents

The Multiple Facets of Developing Intelligent Tutoring Systems

​

​

Danielle S. McNamara

Copyright © 2021 Arizona Board of Regents

• Last 3 Decades have seen substantial advances
• Educational Technologies have strong potential

Intelligent Tutoring Systems

​

​

• Wikipedia: “An intelligent tutoring system (ITS) is a computer system that aims to provide immediate and customized instruction or feedback to learners, usually without requiring intervention from a human teacher. ITSs have the common goal of enabling learning in a meaningful and effective manner by using a variety of computing technologies. ”
• Domain model
• Student model
• Tutoring model
• User interface model

​

​

​

​

¹ Allen et al., 2014, 2015; ² Crossley, Roscoe, & McNamara, 2013; ³ Crossley et al., 2013; ⁴ Roscoe et al., 2013

​

Many ITSs have demonstrated success in matching or exceeding human tutoring outcomes

there are nonetheless relatively few ITSs implemented at scale

many ITSs have been stripped of their intelligence in order to take them to scale

One reason for the dearth of large-scale ITS adoption can be attributed to an incompatibility between educational practice (e.g., covering curricula within the context of a time-limited classroom) and intelligent tutoring (e.g., adapting curricula and pacing to the needs of individual learners)

This misalignment might be viewed as a fatal flaw in the conceptualization of ITSs (or vice versa)

The Multiple Facets of Developing Intelligent Tutoring Systems

​

​

From a different angle, the development of ITSs offers contributions well beyond the end product - beyond an educational platform to support learning and instruction. 

​

This talk will describe how the multiple facets of ITS development have contributed to educational practice, the design of educational learning environments, and our understanding of learning processes.

McNamara, Arner, Butterfuss, Mallick, Lan, Roscoe, Roediger, & Baraniuk (in press). Situating AI (and Big Data) in the Learning Sciences:

Moving Toward Large-Scale Learning Sciences. In Alavi & McLaren, Artificial Intelligence in STEM Education: The Paradigmatic Shifts in Research, Education, and Technology

Landscape of Learning Sciences

McNamara, Arner, Butterfuss, Mallick, Lan, Roscoe, Roediger, & Baraniuk (in press). Situating AI (and Big Data) in the Learning Sciences:

Moving Toward Large-Scale Learning Sciences. In Alavi & McLaren, Artificial Intelligence in STEM Education: The Paradigmatic Shifts in Research, Education, and Technology

Landscape of Learning Sciences

language

Literacy Technologies

Coh-Metrix

Bellissens et al., 2007, 2008; Bruss et al., 20014; Cai et al.., 3004; Crossley et al., 2007ab, 2008abcd, 2009abc, 2010abcde, 2011abcdefghij, 2012abcdefg; Dempsey et al., 2007, 2009; Dufty et al., 2006; Duran et al., 2006, 2007ab; 2009, 2010; Graesser et al., 2003, 2004, 2007ab, 2010, 2011ab, etc.; Hall et al., 2007; Hempelman et al., 2005, 2006; Jarvis et al., 2012; Lightman et al., 2007; Louwerse et al., 2004, 2006; McCarthy et al., 2006, 2007abc. 2008abc, 2009abcd; McNamara et al., 2006, 2007, 2010, 2012ab; Ozuru et al., 2008; Rus et al., 2008abc, 2009abc, 2011, 2012;

Analyzes texts on many different dimensions of cohesion and language

see McNamara, Graesser, McCarthy, & Cai, 2014

Linguistic Properties

14

linguistic features relating to readability and quality of texts

lexical sophistication

syntactic complexity

cohesion

semantics

Machine Learning �&�Natural Language Processing

Quality Scores

Self-explanation

Paraphrasing

Summaries

Essay quality

​

Component Scores

Text Difficulty

Writing Characteristics

Writing Scores

​

Individual differences

Prior knowledge

Vocabulary

Reading skills

Working memory

Creativity

Authorship

Course performance

​

​

{

Copyright © 2021 Arizona Board of Regents

ISTART

Began with SERT

Adaptive Instruction and Training for Reading Comprehension

Inference Making

Summarization

Question Asking

Provides automated instruction on self-explanation reading strategies through lesson videos

Paraphrasing

Comprehension Monitoring

Bridging Inferences

Elaborative Inferences

Regular and game-based practice with automated feedback on the quality of self-explanation

​

Allen et al., 2014ab, 2015abcd, 2016, 2017abc, 2019; Jackson et al., 2013, 2014abc, 2015, 2017; Jacovina et al., 2015, 2016ab; Johnson et al., 2016, 2017; Kurby et al., 2012; Magliano et al., 2004; K. McCarthy et al., 2018ab, 2017ab,, 2020; McNamara et al., 2004, 2006, 2007, 2015, 2016, 2017, 2019; O’Reilly et al., 2004a, 2004b; Snow et al., 2013ab, 2014abcde, 2015abcdefg, 2016abc, 2017; Soto et al., 2015; Weston et al., 2016

Summary of iSTART Benefits

​

• Increased knowledge and application of self-explanation and comprehension strategies^1,2
• Improves comprehension for low knowledge readers and for both less skilled and skilled readers^4,5
• More positive attitudes toward reading³
• Games enhance engagement, enjoyment, and persistence during practice³
• Positive attitudes toward iSTART:
• “Every school curriculum should use a system as easy and user friendly as this one, making it fun to learn.”
• “The system really helped me more with self-explanations. And I really needed help on it and it informed me more on things I didn't even know about. So, thank you for accepting me in this program. And letting me experience it.”

​

¹ McNamara, 2004, 2016; ² O’Reilly et al., 2004a, 2004b; ³ Jackson & McNamara, 2014; ⁴ Magliano et al., 2004; ⁵ McNamara et al., 2006, 2007

Writing Pal

​

• Strategy Instruction
• Pre-writing
• Introduction, Body, and Conclusion
• Revision
• Nine instructional modules including lesson videos and games
• Animated lesson videos using pedagogical agents
• Educational games to practice strategies related to components of writing
• Writing practice with automated feedback
• Student, Teacher, and Experimenter interfaces

​

-tutoring system combining game-based strategy instruction with automated writing evaluation

Allen et al., 2013ab, 2014abc; 2015abc; 2016abcde; 2017, 2018, 2019; Brandon et al., 2012; Crossley et al., 2009, 1010, 2011abcd, 2012ab, 2013abcd, 2014abcdef, 2015ab; 2016abc; 2019; Dai et al., 2011; Dascalu et al., 2015; DiSano et al., 2012; Guo et al., 2013; Jacovina et al., 2015, 2016; Jung et al., 2019 Likens et al., 2017; McCarthy et al., 2019, 2020; McNamara et al., 2010, 2012, 2013ab, 2015; Proske et al., 2014; Roscoe et al., 2011, 2013abcde, 2014ab, 2015, 2016, 2019; Shum et al., 2016; Skalicky et al., 2016; Snow et al., 2015ab; Weston et al., 2010, 2011, 2012, 2013, 2016; see also McNamara & Allen, 2017, 2019

Automated Evaluation and Feedback�Focuses on Strategies to Improve Writing

Summary of Writing-Pal Benefits

​

• Increased performance on timed, prompt-based writing^1,2,3
• Increased knowledge of writing strategies⁴
• More positive attitudes toward the writing process¹
• Games enhance engagement, enjoyment, and persistence¹
• Positive attitudes toward W-Pal:
• “It was a lot of fun, i learned a lot, and i feel that i've become a stronger and better writer now.”
• “I feel the system helped improve a lot of writing and editing skills.”

​

​

​

¹ Allen et al., 2014, 2015; ² Crossley, Roscoe, & McNamara, 2013; ³ Crossley et al., 2013; ⁴ Roscoe et al., 2013

​

¹ Allen et al., 2014, 2015; ² Crossley, Roscoe, & McNamara, 2013; ³ Crossley et al., 2013; ⁴ Roscoe et al., 2013

​

Enhanced Learning

& Performance

​

iSTART and Writing Pal: Multiple Facets of Development

…beyond effectiveness

Enhanced Learning

& Performance

​

iSTART and Writing Pal: Multiple Facets of Development

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Enhanced Learning

& Performance

​

Machine Learning & AI

Enhanced Learning

& Performance

​

Machine Learning & AI

The Multiple Facets of Developing Intelligent Tutoring Systems

​

​

What Lies Ahead

(AKA what am I currently focused on?)

McNamara, Arner, Butterfuss, Mallick, Lan, Roscoe, Roediger, & Baraniuk (in press). Situating AI (and Big Data) in the Learning Sciences:

Moving Toward Large-Scale Learning Sciences. In Alavi & McLaren, Artificial Intelligence in STEM Education: The Paradigmatic Shifts in Research, Education, and Technology

Landscape of Learning Sciences

Much of the research in the Learning Sciences is focused on a few factors.

​

Much of the data on learning, and from instructional technologies is hard to access at scale, and there are generally a limited number of factors measured or manipulated.

Walls Around Student Data

� Balkanization of digital learning platforms

​

• Data are compartmentalized within each platform, �with each representing only a small slice of a student’s �learning experience and behavior

​

• Data available only through ad-hoc requests, �which prevents systematic research

​

• Most researchers are shut out of platform-based�research

​

36

36

We Need

• Large-scale R&D infrastructures that
• open up data to external researchers
• link a wide variety of digital learning platforms at different education levels
• enable researchers to conduct research in scalable manners in authentic learning environments

• Complete picture of the student �over the long term
• Inclusive research infrastructure and community support
• Institutional support
• Securing, privacy of individual and institution

Copyright © 2021 Arizona Board of Regents

The ASU Learning at Scale

Digital Learning Network Platform

The ASU L@S project is developing foundational infrastructure and protocols to connect a wide range of data available in the ASU data ecosystem on student achievement, learning, and persistence.

The data will be accessible to researchers within and outside of ASU in ways that honor institutional and individual privacy, so that it can be examined through exploratory and experimental methods.

Challenge

Solution

• No widely adopted protocols.
• Changes in instructional practice and student support fails to demonstrate effectiveness beyond initial learner populations, contexts, and scales.
• Key data are not collected or are not representative of increasingly diverse learner populations.

• Aggregated data warehouse.
• Platform that furthers our understanding of education and learning through feature- and context-rich student data.
• Model for other public universities on best practices in developing Digital Learning Platforms (i.e. MSRI).

Copyright © 2021 Arizona Board of Regents

Digital Learning Platforms to Enable Efficient Education Research Network

SEERNet

OpenStax

MATHia

ASSISTments

Canvas + Terracotta

ASU Learning@Scale

Digital Learning Network (DLN)

to meet and share solutions.

https://ies.ed.gov/ncer/projects/program.asp?ProgID=2119

Copyright © 2021 Arizona Board of Regents

Beyond the data….

Scale?

​

Generalization?

​

Impact?

Copyright © 2021 Arizona Board of Regents

Copyright © 2021 Arizona Board of Regents

​

Our Future Depends on

Collaborative

Research and Implementation

​

Thank You!

​

​

soletlab.asu.edu

adaptiveliteracy.com

Danielle McNamara

dsmcnamara1@gmail.com

See also,

tmb.pubpub.org/