The Promise and Challenges of Implementing MMLA in Real-World Settings
Mutlu Cukurova University College London
m.cukurova@ucl.ac.uk @mutlucukurova
Multimodal Learning Analytics:
“tools that have been designed and developed to assist in or replace decision-making processes in education through analysis of multiple modalities of data, and prediction of the best value for a designated outcome variable, which is conveyed through a user interface.”
MMLA: A vision for the future (1 of 3)
x
Human Control
Automation through AI
Most traditional Educational Technology
AH = Human tasks are replaced by MMLA H A
Initial vision of AIED: Early Promises of Intelligent Tutoring Systems
High
Low
High
Low
Multimodal Learning Analytics
Machine Learning Classification of Collaboration Competence
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning.
Journal of Computer Assisted Learning, 34(4), 366-377.
Independent Variables (MMLA Features)
FLS - Number of faces looking at screen
DBL - Mean distance between learners
DBH - Mean distance between hands
HMS - Mean hand movement speed
AUD - Mean audio level
IDEX - Arduino measure of complexity
IDEVHW - Arduino active hardware blocks
IDEVSW - Arduino active software blocks
IDEC - Arduino active blocks
PWR - Student Work Phases
The promise of MMLA
Items | PWR | PW | W | WR |
NB | 0.8 | 0.8 | 0.6 | 0.75 |
SVML | 0.6 | 0.75 | 0.75 | 0.8 |
SVMR | 0.75 | 0.75 | 0.75 | 0.75 |
LR | 0.6 | 0.75 | 0.5 | 0.6 |
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377.
Note. NB = naive Bayesian; LR = logistic regression; SVML = support vector machines with linear kernel; SVMR = support vector machines for regression
Removed feature | Best Result |
No features removed | 0.129 |
All faces data | 0.21 |
All Arduino data | 0.21 |
DBF | 0.15 |
DBH | 0.21 |
HMS | 0.19 |
AUD | 0.18 |
Hand pos | 0.21 |
Arduino comp | 0.19 |
Method | Deep Learning | Traditional |
Task | Regression | Classification |
Input | 18 variables | 9 variables per window |
Output | 6 scores over 5 levels | 1 score with 3 levels |
Metrics | Regression score | Classifier accuracy |
Windowing | 120,240 and 360s | 10, 20, 30, 90 min |
Phase exclusion | Reflection | Reflection |
Method | Multiple layers | NB, LR, SVML, and SVMR |
Detecting Drowsy Learners at the Wheel of e-Learning Platforms with Multimodal Learning Analytics
R. Kawamura et al., Detecting Drowsy Learners at the Wheel of e-Learning Platforms With Multimodal Learning Analytics, IEEE Access, vol. 9, pp. 115165-115174, 2021, doi: 10.1109/ACCESS.2021.3104805
Features | Metrics | Constructs Represented |
Heart rate (HR) | Mean/standard deviation of RRI, HR, LF/HF, Body surface temperature | Heart rate in general represents the activity of the autonomic nervous system. RRI is an index of heart rate variability. HR is the heart rate. Low frequency (LF) power and high frequency (HF) power represent stress and rest states. Body surface temperature is environmental temperature in clothing. |
Seat Pressure (SP) | Mean Pressure | Mean of each frame’s total pressure and mean of pressure per second. They are used to estimate a learner’s motions |
Seat Pressure (SP) | Mean time of MS (moving state) and SS (static state) | Represents how long a learner moves or stays still. |
Seat Pressure (SP) | Ratio of MS (moving state) | Represents how often a learner changes posture. |
Seat Pressure (SP) | Mean of absolute pressure difference between pressure current and previous frame. | Represents how large and how often a learner changes posture along vertical axis. |
Facial Expression (Face) | Mean/standard deviation of AU 2, 15, 26, 45 (occurrence and intensity) | AU2: Outer Brow Raiser, AU15: Lip Corner Depressor, AU26: Jaw drop, AU45: Blink. |
Facial Expression (Face) | Mean/standard deviation of head rotation (yaw, pitch, roll) | Represents how large a learner’s head rotation is. |
Facial Expression (Face) | Mean/standard deviation of head transition along x, y, z | Represents how large a learner’s head transition is. |
MMLA: A vision for the future (2 of 3)
Human Control
Automation through AI
Most traditional Educational Technology
HA = Humans internalise MMLA models H 🡺 A
Analytics and GOFAI: Changing the operations and representations of thought
AH = Human tasks are replaced by AI H A
Initial vision of AIED: Early Promises of Intelligent Tutoring Systems
High
Low
High
Low
Online Group Meeting Analytics
Zhou,Q., Suraworachet, W., Pozdniakov, S., Martinez-Maldonado, R., Bartindale, T., Chen, P. Richardson, D., & Cukurova M. (2021). Investigating Students’ Experiences with Collaboration Analytics for Remote Group Meetings. International Conference of Artificial Intelligence in Education, Springer, Cham.
Cover Topics | Sample Contents |
Acknowledgement of group activities on the brainstorming platform. | Hello Group X, Based on your interaction board, it is great to see you included your note of the group discussion here. I appreciate that you came up with a clear goal in mind and propose promising approaches towards achieving your goal. Well done to you. |
Description of the collaboration analytics (Speech time pie chart and turn-taking network) which highlighted significant interactions/trends and encouraging messages for more equal contributions. | According to the speech time pie chart, Member 1 and Member 2 dominated most of the group speaking time. Member 3, Member 4 and Member 5 spoke less during the meeting. Hope we could hear more from them in future meetings. There was also a significant amount of time spent in silence. Maybe you were working on your notes? This also links to the turn-taking network as the strongest turn-taking link was between Member 1 and Member 2 whereas the number of turn-taking between other group members was significantly low. |
Emphasis on the formative assessment | As usual, the analytics data presented here is for your individual and group reflections, not for summative assessment purposes. Please feel free to let us know if you have any questions or concerns. Best regards, XX |
MMLA: A vision for the future (3 of 3)
Human Control
Automation through AI
Most traditional Educational Technology
HA = Humans internalise MMLA models H 🡺 A
Analytics and GOFAI: Changing the operations and representations of thought
H[A] = Human cognition (H) extended with an MMLA (A), tightly coupled human and artificial systems.
H[A] > (H) + (A)
AH = Human tasks are replaced by MMLA H A
Initial vision of AIED: Early Promises of Intelligent Tutoring Systems
High
Low
High
Low
MMLA
Active
Active
Passive
Kasparova, A., Celiktutan, O., & Cukurova, M. (2020). Inferring Student Engagement in Collaborative Problem Solving from Visual Cues. Companion Publication of the 2020 International Conference on Multimodal Interaction. https://doi.org/10.1145/3395035.3425961
Face patches (64x64)
Deep CNN (FaceNET)
Embeddings
Figure 1: Our proposed framework i) detects body key points from the recording of student interactions; ii) combines face recognition with a Bayesian model to identify and track students with a high accuracy; and iii) classifies student engagement leveraging a Team Long Short-Term Memory (TEAM LSTM) neural network model.
State | Passive | Semi-active | Active | Missing |
Passive | 8 | 107 | 123 | 5 |
Semi-active | 137 | 216 | 523 | 15 |
Active | 372 | 304 | 1265 | 35 |
Missing | 29 | 22 | 69 | 478 |
Calculating Transparent Collaboration Indexes from Machine Observed Features
Associating code triples to behavioral patterns of students
In particular we identify:
Syn (G) = percentage of 222 states in Group G
IA (G) = percentage of (211,222) - percent of (002, 012, 022)
Eq.(G) = ∑[(AI(s₁,t) - AI(S₂,t))2 + (AI(s₃,t))2 + (AI(s₂, t) - AI(s₃,t))2]
IV(Sᵢ) = ∑ⁿ⁻¹ₖ₌₁(AI(Sᵢ, tₖ₊₁) - AI (Sᵢ, tₖ))² / N - 1
Building Theory-driven Models of Collaboration for Human-in-the-loop systems
Some important components of collaboration may be interpreted through nonverbal indexes of students physical interactivity:
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students' physical interactions. Computers & Education, 116, 93-109.
Cukurova, M. (2018). A syllogism for designing collaborative learning technologies in the age of AI and multimodal data. In European Conference on Technology Enhanced Learning (pp. 291-296). Springer, Cham.
OMA to analyse sequences
Zhou,Q., Suraworachet, W., & Cukurova M. (UR). What does shared understanding “look like”?: Towards discovering students’ gaze patterns in collaborative group work in videos. International Conference of Artificial Intelligence in Education, Springer, Cham.
Short-term Challenges of MMLA
Cover Topics | Sample Contents |
Acknowledgement of group activities on the brainstorming platform. | Hello Group X, Based on your interaction board, it is great to see you included your note of the group discussion here. I appreciate that you came up with a clear goal in mind and propose promising approaches towards achieving your goal. Well done to you. |
Description of the collaboration analytics (Speech time pie chart and turn-taking network) which highlighted significant interactions/trends and encouraging messages for more equal contributions. | According to the speech time pie chart, Member 1 and Member 2 dominated most of the group speaking time. Member 3, Member 4 and Member 5 spoke less during the meeting. Hope we could hear more from them in future meetings. There was also a significant amount of time spent in silence. Maybe you were working on your notes? This also links to the turn-taking network as the strongest turn-taking link was between Member 1 and Member 2 whereas the number of turn-taking between other group members was significantly low. |
Emphasis on the formative assessment | As usual, the analytics data presented here is for your individual and group reflections, not for summative assessment purposes. Please feel free to let us know if you have any questions or concerns. Best regards, XX |
Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The Promise and Challenges of Multimodal Learning Analytics. British Journal of Educational Technology, 51(5), pp. 1441-1449, https://doi.org/10.1111/bjet.13015. Zhou,Q., Suraworachet, W., Pozdniakov, S., Martinez-Maldonado, R., Bartindale, T., Chen, P. Richardson, D., & Cukurova M. (2021). Investigating Students’ Experiences with Collaboration Analytics for Remote Group Meetings.
International Conference of Artificial Intelligence in Education, Springer, Cham.
Zhou, Q., Suraworachet, W., Cukurova, M. (2021). Different modality, different design, different results: exploring self-regulated learner clusters’ engagement behaviours at individual, group and cohort activities. https://doi.org/10.35542/osf.io/u3g4n
“The Real” Challenges of MMLA
Normativity
Prediction
Human Agency
Ethics- Human Values