1 of 71

Human-Centered Learning Analytics: Designing for balanced human and computational agency

Prof. Yannis Dimitriadis

GSIC/EMIC group

University of Valladolid, Spain

Intelligent Tutoring Systems 2022

July 1, 2022

Although Artificial Intelligence (AI) and Learning Analytics (LA) have shown their potential in Education, stakeholders’ agency seems to be threatened. On the other hand, multiple issues regarding FATE (Fairness, Accountability, Transparency and Ethics) have been raised when AI or LA-based solutions are designed and implemented. This talk discusses the quest for an optimal balance between human and computational agents, when LA tools and services are employed in a Technology Enhanced Learning (TEL) ecosystem. Through the discussion of relevant conceptual models and examples, it argues for a Human-Centered Learning Analytics (HCLA) approach, where agency and FATE principles are essential design parameters. The talk focuses especially on LA solutions that may position teachers as designers of effective interventions and orchestration actions. Selected Human-Centered Design (HCD) principles are discussed and illustrated, and directions for future research and development are formulated to overcome the main obstacles for adoption of HCD for LA.

2 of 71

Lots of acknowledgements

3 of 71

Learning Analytics (LA) �in Technology Enhanced Learning (TEL)

Learning Analytics

“measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”

Most R&D - Innovation has been devoted to

Mining patterns
Deriving predictive models
Providing dashboards

4 of 71

Predictive models with LA�(At-risk students)

Herodotou, C.; Hlosta, M.; Boroowa, Avinash; R., Bart; Zdrahal, Z. and Mangafa, C. (2019). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology, 50(6) pp. 3064–3079.

How does the predictive model work and how it was trained?

How were the data collected for this prediction model?

Who was involved in its design and who can use the data?

5 of 71

Pattern mining using LA�(Detection of learning strategies)

J. B. J. Huang, A. Y. Q. Huang, O. H. T. Lu and S. J. H. Yang, "Exploring Learning Strategies by Sequence Clustering and Analysing their Correlation with Student's Engagement and Learning Outcome," 2021 International Conference on Advanced Learning Technologies (ICALT), 2021, pp. 360-362, doi: 10.1109/ICALT52272.2021.00115.

How are the proxies for strategies DEFINED AND COMPUTED?

Who can interpret this data and how?

Is there any student bias regarding these strategies?

6 of 71

LA-based dashboards�(Monitoring and sense-making)

S. Charleer, A. V. Moere, J. Klerkx, K. Verbert and T. De Laet, "Learning Analytics Dashboards to Support Adviser-Student Dialogue," in IEEE Transactions on Learning Technologies, vol. 11, no. 3, pp. 389-399, 1 July-Sept. 2018, doi: 10.1109/TLT.2017.2720670

How effective is sense-making out of those dashboards?

Do teachers-students need to improve their data literacy?

Can we compensate the sense-making workload?

7 of 71

Smart Learning Environments (Personalized recommendations-resources)

S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, G. Vega-Gorgojo, A. Ruiz-Calleja and J. I. Asensio-Pérez, "From Informal to Formal: Connecting Learning Experiences in Smart Learning Environments," 2021 International Conference on Advanced Learning Technologies (ICALT), 2021, pp. 363-364, doi: 10.1109/ICALT52272.2021.00116.

How is the student model built?

Do teachers/students get involved in the reaction scripts?

What about privacy in informal learning settings?

8 of 71

Two dilemmas on Agency (I)

Dilemma 1: Learning Analytics (LA) may be helpful when embedded in Technology-Enhanced Learning (TEL) contexts. They are typically designed by researchers and developers, that best know about efficiency and effectiveness. But existing LA solutions mostly ignore teachers as orchestrators (designers and enactors).

What about teachers’ agency?

9 of 71

Two dilemmas on Agency (II)

Dilemma 2: Artificial Intelligence (AI) agents that are using LA may support and eventually maximize students’ learning but how can they be transparent, trustful, responsible or ethical?

What about students’ agency?

10 of 71

What is this talk about

Discuss the dilemma regarding teachers’ agency when designing and orchestrating LA solutions
Analyze models for human-LA complementarity and teachers’ augmentation
Formulate design principles for Human-Centered Learning Analytics (HCLA)
Illustrate the HCLA approach

11 of 71

A definition of teachers’ agency

Priestley, M., Biesta, G., & Robinson, S. (2015). Teacher agency: What is it and why does it matter? In R. Kneyber & J. Evers (Eds.), Flip the System: Changing Education from the Ground Up (pp. 134–148). Routledge. https://doi.org/10.4324/9781315678573 (adapted)

Agency entails the capacity of actors to make practical and normative judgments

among alternative possible trajectories of action, in response to the emerging

demands, dilemmas, and ambiguities of presently evolving situations

The ecological perspective of agency proposed by Priestley et al. (2015) takes the conceptualization of human agency proposed by Emirbayer & Mische (1998), these authors conceived human agency as:

A temporally embedded process of social engagement, informed by the past (in its habitual aspect), but also oriented toward the future (as a capacity to imagine alternative possibilities) and toward the present (as a capacity to contextualize past habits and future projects within the contingencies of the moment) (p.963)

That point of view stresses the importance of the moment, agency is always enacted in the present, but it is affected by the past and directed toward the future. Tackling agency under the ecological perspective implies considering not only the personal resources, but also the environmental ones, as well as social situations. Under the view proposed by Priestley et al. (2015), teachers’ agency is enacted in the present, but it is influenced by personal and professional histories and considers the projective dimension. That dimension considers both short and long-term goals, desires, or fears as a factor of influenced related with agency. The

For this model, the practical-evaluative element is central as it “entails the capacity of actors to make practical and normative judgments among alternative possible trajectories of action, in response to the emerging demands, dilemmas, and ambiguities of presently evolving situations” (Emirbayer & Mische, p.971). However, the iterational and the projective dimension cannot be omitted when studying a particular situation, the study of these elements seems complex as it may imply research involving longitudinal studies and well-structured qualitative studies.

12 of 71

A socio-cultural perspective of professional agency

Eteläpelto, A., Vähäsantanen, K., Hökkä, P., & Paloniemi, S. (2013). What is agency? Conceptualizing professional agency at work. Educational Research Review, 10, 45–65. https://doi.org/10.1016/j.edurev.2013.05.001 (adapted)

Within this model, agency is considered in terms of individual and social aspects of human behavior (Eteläpelto et al., 2013). According to Eteläpelto et al. (2013, p.47) “such a perspective understands individuals as being embedded in and imbued by their socio-cultural contexts; however, they are not seen as passive carriers of their contextual conditions, but rather as capable of transforming these conditions”. The role of professional identity is key, and it is important for individuals to have enough opportunities to practice agency within the organization (e.g., school in the case of teachers) Vähäsantanen et al. (2008).The figure 4 depicts the main features of the subject-centered socio-cultural perspective.

When refereeing to teachers, professional agency is to be practiced at school and classroom contexts: influencing and negotiating core pedagogical and instructional practices, making decisions on one’s ways of teaching as well as developing one’s work (Eteläpelto et al., 2015). Moreover, the study of Vähäsantanen et al. (2008) uncovered that professional agency is needed by teachers to fully commit with their work.

13 of 71

Teachers as producers and shapers

Jenkins, G. (2020). Teacher agency: the effects of active and passive responses to curriculum change. Australian Educational Researcher, 47(1), 167–181. https://doi.org/10.1007/s13384-019-00334-2

Following Bandura (2006), there are four core properties of human agency. Self-reflectiveness (1), pointing out that people are self-examining of their own functioning and not just agents of action; intentionality (2) makes reference to people’s intention to realize action plans as well as strategies for carrying them out; self-reactiveness (3) involves the ability to operate under appropriate courses of action and to maintain the motivation and regulate their execution; and forethought (4) which involves the temporal dimension of agency, visualized futures are brought into the present as guides and motivators of behavior.

The Triadic Reciprocity Framework Core Agency Concepts is a model proposed by Jenkins1 (2020) with the purpose to study teachers’ agency. This model has a double purpose: identify and map the complex dimensions of the act of teacher agency and enable the exploration of the contextual factors that impacted teacher while acting. For Jenkins (2020) “teachers exercise agency in their workplace when they attempt to influence or control aspects of curriculum change in their schools” (p.170). There exists a triadic interconnection among the personal determinants of a person, their behavioral determinants, and the environmental determinants where they live and work, figure 2 depicts this interrelation. Jenkins followed Bandura’s view on agentic theory, which basically situates teachers as producers and shapers of events; exerting influence on things that are to be influenced by them help teachers to achieve desired futures as well as preventing to avoid undesired ones.

14 of 71

Digital agency

Passey, D., Shonfeld, M., Appleby, L., Judge, M., Saito, T., & Smits, A. (2018). Digital Agency: Empowering Equity in and through Education. Technology, Knowledge and Learning, 23(3), 425–439. https://doi.org/10.1007/s10758-018-9384-x (adapted)

Control over and adapt to …

Be proactive producers

Be aware of the data

Decide what data is relevant

According to Passey et al. (2018) Digital Agency (DA), consists “of digital competence, digital confidence and digital accountability—is the individual’s ability to control and adapt to a digital world” (p.426).

In this line, the difficulty that the role of technology entails has been discussed on many occasions. Who dominates whom? If the technology has a greater load, we can speak of moving in technological determinism, however, if the technologies take into account the contextual needs and the "person" it can be said that we are maintaining the balance between the technology and the social agents that employ them. Figure 2 exemplifies this delicate balance

Digital Agency, then, can be characterized as the capacity for individual to choose ‘control’ over and ‘adapt’ to carry out social participation, being involve in decision making in terms of being proactive producers in a social environment accompanying the spread of digital technology (Passey et al., 2018). Teachers need digital agency in order to feel that they are in control of the situation and are fully aware of any sort of data coming out from surrounding educative or non-educative technologies. If data from Learning Analytics3 (LA) implementation practice are to be used, teacher must take into account contexts of action in order to derive meaning from that LA data (Donoghue et al., 2019). Teachers need to decide what data are relevant in those contexts, and that is another example of agency put into motion as teachers are taking decisions and exercising choices with regard to school policies (Passey et al., 2018).

Using learning analytics uncontrolled practices as a resource may end up in having a large volume of educational data that, in my opinion, can be a destabilizing factor for the agency of any individual and, of course, for teachers. As this data comes more frequently from pervasive technologies it is needed to have digital agency (Passey et al., 2018).

As stated previously, in accordance to Passey et al. (2018), digital agency is formed by three dimensions or components, digital competence, digital confidence and digital accountability.

15 of 71

Human-centered design landscape

Sanders, E. B. N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. Co-design, 4(1), 5-18.

16 of 71

From User-Centered Design to Co-Design

User-centred design

Co-creation (co-design)

User

Researcher

Designer

Sanders, E. B. N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. Co-design, 4(1), 5-18.

17 of 71

Human-Centered Learning Analytics

What is human-centeredness in Learning Analytics then?

18 of 71

Human-Centered Learning Analytics

Human centeredness has been identified in other fields as a characteristic of systems that have been carefully designed by:

identifying the critical stakeholders,
their relationships, and
the contexts in which those systems will function

19 of 71

Human-Centered Learning Analytics

HCD should involve:

Inclusion via stakeholder participation in the design process

Empathic experiences (particularly when making design decisions).

Giacomin, J. (2014). What is human centred design? The Design Journal, 17(4), 606–623. https://doi.org/10.2752/175630614X140561854801.

20 of 71

Human-Centered Learning Analytics

Human-centered design considered harmful…

“Most items in the world have been designed without the benefit of user studies and the methods of Human-Centered Design. Yet they do quite well.”

What Adapts? Technology or People?

Don Norman proposes stronger focus on tasks and activities

Norman, D. A. (2005). Human-centered design considered harmful. interactions, 12(4), 14-19.

21 of 71

Human-Centered Learning Analytics

the human centered (not centric)

All the human factors,

social factors and

technology factors

interact together under the human activity umbrella.

22 of 71

Augmented teacher �(Human-AI complementarity)

Augmentation

Complementary strengths and weaknesses
Improvement (co-learning) over time

Goals

Optimized objective functions + design decisions

Perceptions

Sense, attention, interpretation

Holstein, K., Aleven, V., Rummel, N. (2020). A Conceptual Framework for Human–AI Hybrid Adaptivity in Education. In: Bittencourt, I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds) Artificial Intelligence in Education. AIED 2020. Lecture Notes in Computer Science(), vol 12163. Springer, Cham. https://doi.org/10.1007/978-3-030-52237-7_20

Ultimate goals and intermediate targets. Knowledge from behavior patterns High dependence on student modeling, reduce/optimize teachers’ attentional and orchestration load. Instrucional moves of teacher’w repertoire. Mirroring and guiding, Student as a direct recipient of action or mediated through peers and teacher. Select alternative strategies or alert peers/teacher? Adaptive per step/task/dialogue turn/design iteration, or prior, during, afterwards. AIEd technologies encode the assumptions and goals of those who design and develop them. In some cases, teachers’ or

peer tutors’ goals may be fundamentally at odds with known instructional best practices. Lumilo teacher smartglasses are designed to direct teachers’ attention, during a class session, to

situations that an AI tutor may be poorly suited to handle on its own, or which require a teacher’s further assessment. help humans learn to notice relevant features. allow teachers and parents

to update individual student models. poll students regarding their subjective feeling of knowing particular skills. these systems perform a considerable

amount of pre-interpretation on behalf of human facilitators or learners. explore the design of interfaces that can more

actively guide humans towards particular interpretations of learning data (e.g., [17]) or

interfaces that can scaffold humans in more productive forms of reflection (e.g., [19]).

However, it remains an open question for future research how best to productively guide

human interpretation, while still leveraging (rather than diminishing) humans’ unique

inferential capacities. detecting cases where the AI misinterprets learning

data (e.g., by misclassifying patterns in collaborating groups’ behaviors) and to provide

corresponding feedback in order to shape these interpretations in more meaningful

directions. Adaptively provide feedback or resources. Custommizing and creating new actions through authoring tolos. that adaptively,

dynamically delegate instructional roles between AI systems, teachers, and peers, based

upon an awareness of trade-offs between the instructional ability and capacity of each

teaching the AI tutor (as discussed below),

transferring their unique expertise and pedagogical preferences

diverse contexts by adapting the way these systems are implemented in use to

the needs of their local contexts

With knowledge of a facilitator’s instructional goals,

future AIEd systems might help the facilitator make more informed trade-offs between

competing goals or nudge them away from practices that are at odds with their goals

machine teaching

for AIEd suggests promise for approaches in which humans teach the AI to teach

through feedback and demonstrations [37, 3

23 of 71

Augmented teacher �(Human-AI complementarity)

Actions

Action space, scalability and capacity

Decisions

Link perception and action – take effective pedagogical interventions

Timing and granularity

e.g., adaptation by teachers through LA dashboards, during learn time, regarding a task

In the following we present a conceptual framework for human–AI hybrid adaptivity

in education, examining the same set of basic components (goals/targets, perception,

action, decision policies, and granularity/timing) while broadening our focus. We use

this framework both to characterize prior work and to envision new possibilities, based

upon distinct ways in which humans and AIEd systems might augment one another:

(1) Goal Augmentation, (2) Perceptual Augmentation, (3) Action Augmentation, and

(4) Decision Augmentation. Within each category, possibilities exist both for augmenting

performance (in which humans and AI systems, assumed to have complementary

strengths and weaknesses, augment one another’s abilities at “runtime”, but without necessarily

producing lasting changes in behavior) and for co-learning (inwhich humans and

AI systems help one another improve over time). Finally,we discuss howthe Granularity

and Timing of adaptivity might be understood in human–AI systems.

A key way for humans and AIEd systems to support one another is by influencing each

other’s goals. To a large extent, AIEd technologies encode the assumptions and goals

of those who design and develop them—whether explicitly, via objective functions that

a system’s adaptive policies optimize towards, or implicitly, through design decisions

that promote certain goals over others. However, the goals baked into an AIEd system

may not always align with those of humans in real-world educational contexts [24, 46,

53]. For example, ITSs used in K-12 school contexts often implement mastery-based

activity selection policies, allowing each student to progress through the curriculum at

their own pace. Yet prior work suggests that teachers often struggle to balance their

desire to implement such personalized classrooms with external pressure to keep classes

“on schedule”. In practice, teachers often opt to manually push students forward in the

curriculum if they are slower to master certain skills [24, 53], sometimes even if they

are aware that doing so may harm students’ learning [24, 28]. As of yet, little work in

AIEd has explored the design of supports for goal augmentation.

Goal Augmentation: Informing Each Other’s Instructional Goals

A second way for AIEd systems and humans to augment one another is by enhancing

each other’s abilities to perceive instructionally relevant information, or opportunities

for action. This may take the form of (1) extending what the other is able to sense (i.e.,

what information is made available to them, prior to further interpretation [11, 20]); (2)

guiding how the other distributes their attention; or (3) guiding how the other interprets

incoming information. Each of these broad possibilities is discussed in turn, below.

Perceptual Augmentation: Leveraging Complementarity in Perception

Augmenting Sensing and Attention

Augmenting Interpretation

Athirdway inwhich AIEd systems and humans canwork together to supportmore adaptive

instruction is by augmenting and extending the other’s capacities for instructional

action. In particular, AIEd systems and humans can (1) enhance each other’s ability

to perform particular kinds of instructional actions, and relatedly expand the range of

actions available to each; and (2) enhance each other’s scalability and capacity for

action.

Enhancing Ability and Expanding the Availability of Actions

Enhancing Scalability and Capacity

Beyond informing each other’s goals or augmenting each other’s capacities for perception

and action, a fourth major way in which AIEd systems and humans can work

together is by helping each other make more effective pedagogical decisions (i.e., helping

each other more effectively link between perception and action). Prior work has

explored forms of both AI→human and human→AI decision augmentation. However,

much additional research is needed in order to fully realize the visions of AIEd systems

as, for instance, effective decision support and professional development tools [5, 19,

23, 24, 66] and as teachable machines [37, 39, 60, 74].

Finally,we briefly discuss howgranularity and timing might be understood in human–AI

systems.WhenAIEd systems and humanswork together, theymay each adapt instruction

at different grain sizes. For instance, in classrooms using step-based tutoring software,

teachers may provide substep feedback on-the-spot (i.e., feedback on a step while the

student is, from the system’s perspective, still in themidst of completing the step) [23, 27,

30]. While an AIEd system waits for the student to submit their input, a human facilitator

might perceive an opportunity to intervene within a long pause in student typing. The

timing of adaptation may also vary across humans and machines. For instance, Aleven

et al’s “design loop adaptivity” [3] can be viewed as involving a form of shared adaptivity

in which human facilitators or instructional designers repeatedly adapt an AIEd system’s

design (informed by educational data and/or their own observations) before or after an

instructional activity, while the AIEd system in turn takes care of adapting to learning

situations during the activity.

Our case study illustrates the design and evaluation of a successful human–AI partnership in a�challenging real-world context: K-12 education. While AI has shown great potential to enhance�learning and teaching in K-12, the work of human teachers has been recognized as unlikely to be�fully automated (7, 24). We concur: as in other care-based professions where relationship building�is central, AI systems appear to have the greatest potential for positive impact where they are�designed to augment and complement the abilities of human practitioners. In the Lumilo project,�we took a participatory approach to research and design, deeply engaging teachers both in framing�the challenges to be addressed through an improved partnership and in shaping how this�partnership would function. Through iterative piloting in live K-12 classrooms, we observed the�interplay of human and AI decision-making under real-world conditions, refining the design of�Lumilo based on our observations to shape human–AI decision-making in more positive�directions. Throughout this process, we worked to develop our understanding of what�complementary strengths human teachers and AI tutors hold in our context, and how Lumilo’s�design might be optimized to effectively combine these strengths.�We view these components – taking a participatory approach to research and design,�measuring and shaping human–AI decision-making in context, and working towards a theory of�complementarity – as essential ingredients in the design of effective human–AI partnerships, both�within and beyond the domain of education. Accordingly, to advance scientific and design progress�in this area, we highlight three broad recommendations for future research. First, to support the�development of human–AI partnerships that are aligned with real-world needs and work practices,�it is critical to engage practitioners throughout the entire design and development lifecycle for a�new technology. However, deeply involving practitioners becomes challenging when designing�data-driven AI systems, given that practitioners may know very little about AI. The current case�study illustrates the value of developing new design and prototyping methods. However, further�research is urgently needed to develop new technical and design methods that can meaningfully�engage non-technical stakeholders in understanding and working with AI as a design material, e.g.,�see (12, 40). Second, to support more reliable scientific and design insight into the behavior and�dynamics of human–AI partnerships, future research should strive to study these systems in real-�world contexts, with authentic tasks and relevant human experts. Although the use of artificial�tasks and contexts may appear convenient, recent work has demonstrated that results from such�studies often fail to translate to the real-world settings for which partnerships are intended, e.g.,�see (4–5, 25). Third, to support cumulative scientific progress and more systematic design�exploration in this area, future research should work to develop and build upon both domain-�general and domain-specific theories of human–AI complementarity. While we have begun to�pursue this direction within the current ca

24 of 71

Augmented teacher �(Human-centered approach)

Lumilo project (CMU) on human-AI partnership in real-world K-12 education
Co-orchestration (ITS and teachers) of transitions from individual to group activities
Adoption of participatory (human-centered) approach to design and development lifecycle

Holstein, K., & Aleven, V. (2022). Designing for human-AI complementarity in K-12 education. }, ArXiv, abs/2104.01266

Echevarría, V. Yang, K., Lawrence, L., Rummel, N., Aleven V., (2020). Exploring Human–AI Control Over Dynamic Transitions Between Individual and Collaborative Learning, In Proceedings of ECTEL 2020

“

25 of 71

Levels of human-centeredness

Human in command

Oversee when and how to use AI/ITS

Human on the loop

Participate in design and operation

Human in the loop

Get involved in every lifecycle phase

Smuha N.A. (2023). “Pitfalls and pathways for trustworthy Artificial Intelligence in education” in The Ethics of Artificial Intelligence in Education Practices, Challenges, and Debates, W. Holmes, K. Porayska-Pomsta (Eds). Taylor and Francis.

26 of 71

Human-Centeredness in MMLA-AIED

Kukurova, M. (2022). “Multimodal Learning Analytics in Real-world Practice: A Bridge Too Far?”, Webinar at Spanish Network of Learning Analytics (SNOLA), May 2022. https://snola.es/2022/05/03/webinar-multimodal-learning-analytics-in-real-world-practice-a-bridge-too-far-mutlu-cukurova/

27 of 71

Some elements to consider

LA solutions were eventually pushed by new technological (Data and AI) affordances
Teachers as designers were not always considered in complex real-world TEL spaces
The hybrid AI-human models and their trade-offs were not fully studied
Learning theories have not been used extensively while designing LA solutions

28 of 71

The complexity of TEL ecosystems�(Hybrid Learning Spaces)

Gil, Mor, Dimitriadis & Köppe (2022): Hybrid Learning Spaces, Springer https://doi.org/10.1007/978-3-030-88520-5

29 of 71

Design and orchestration

Prieto, L. P., Y. Dimitriadis, J. I. Asensio-Pérez, C. K. Looi (2015). “Orchestration in learning technology research: evaluation of a conceptual framework”. In: Research in Learning Technology 23.0

How to support teachers as designers and reduce/optimize their orchestration load?

30 of 71

Teachers as designers

Pedagogical knowledge

Eventually embedded in tools
Complements / cooperates with the tacit and explicit knowledge of the teachers

Teachers

Are and can serve as designers
Should participate in the design and orchestration of the teaching and learning processes

Kali, McKenney & Sagy (2015)

31 of 71

The METIS ILDE

LD and orchestration tools

Villasclaras-Fernández, Hernández-Leo, Asensio-Pérez & Dimitriadis (2013)
Håklev, Faucon, Hadzilacos & Dillenbourg (2017) - Figure
Laurillard, Kennedy, Charlton, Wild & Dimakopoulos (2018) - Figure

32 of 71

Balancing computer-human agents

Sharples, M. (2013). Shared Orchestration Within and Beyond the Classroom. Computers & Education. 69. 504-506. 10.1016/j.compedu.2013.04.014.

33 of 71

Mirroring, Advising, Guiding through LA

Soller, A., Martínez-Monés, A., Jermann, P., Muehlenbrock, M. (2005) From Mirroring to Guiding: A Review of the State of the Art Technology for Supporting Collaborative Learning International Journal of Artificial Intelligence in Education (ijAIED). 15:261-290

34 of 71

Distributed scaffolding

Puntambekar, S. Distributed Scaffolding: Scaffolding Students in Classroom Environments. Educ Psychol Rev (2021). https://doi.org/10.1007/s10648-021-09636-3
https://www.imec-int.com/en/research-portfolio/steams: Supporting TEAMS in ambient learning spaces

Across

Tools and social scaffolds
Levels (individual, group, and whole class)
Time and Contexts

35 of 71

A Hybrid human-AI learning model

Molenaar, I. (2021), "Personalisation of learning: Towards hybrid human-AI learning technologies", in OECD Digital Education Outlook 2021: Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots, OECD Publishing, Paris, https://doi.org/10.1787/2cc25e37-en.

36 of 71

Human-AI extended model

Teacher monitors and controls

the learning design prior to execution (configuration phase)
the orchestration of the lesson (runtime)

Learner monitors and controls learning

Orientation and planning prior to execution
Monitoring and control during execution
Reflection after execution

37 of 71

Human-AI extended model

Timing and phases

Detect (data) 🡪

Diagnose (technique/algorithm) 🡪

Act (action)

Act components

LA Perspective

Inform, Advise, Guide, Recommend

ITS Perspective

Step, Task, Curriculum

38 of 71

Human-AI extended model

The transitions of control and monitoring have profound implications for the professional functioning (agency) of teachers

Giving up task has positive sides (less time on correction, more feedback)
but also, negative sides (less insights and control).

This friction cannot be resolved easily but co-creation processes do allow for a careful articulation of this friction

39 of 71

Human-AI extended model

Static or dynamic balance

redesign and reconfiguration
self-, co-, socially shared regulation

Operators for teachers’ augmentation

Transparency, agency, explainability, …

40 of 71

Hybrid Intelligence

“… the most likely paradigm for the division of labor between humans and machines in the next years, or probably decades, is hybrid intelligence. … to try to combine the complementary strengths of heterogeneous intelligences (i.e., human and artificial agents) into a socio-technological ensemble. We envision hybrid intelligence systems, … to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results than each of the could have done in separation and continuously improve by learning from each other”

D. Dellermann, P. Ebel, M. Soellner, J.M. Lerimesiter, “Hybrid Intelligence”, arXiv:2105.00691v1 [cs.AI]

41 of 71

Hybrid Intelligence

“… Hybrid intelligence (HI) can go well beyond this by creating systems that operate as mixed teams, where humans and machines cooperate synergistically, proactively, and purposefully to achieve shared goals, showing AI’s potential for amplifying instead of replacing human intelligence”
“Collaborative HI: How do we develop AI systems that work in�synergy with humans?�› Adaptive HI: How can these systems learn from and adapt to humans and their environment?�› Responsible HI: How do we ensure that they behave ethically and responsibly?�› Explainable HI: How can AI systems and humans share and explain their awareness, goals, and strategies?”

Z. Akata et al., (2020) "A Research Agenda for Hybrid Intelligence: Augmenting Human Intellect With Collaborative, Adaptive, Responsible, and Explainable Artificial Intelligence," Computer, 53(8), 18-28, doi: 10.1109/MC.2020.2996587

42 of 71

AI and the future of learning

Roschelle, J., Lester, J. & Fusco, J. (Eds.) (2020). AI and the future of learning: Expert panel

Report. Digital Promise. https://circls.org/reports/ai-report.

Investigate AI Designs for an Expanded Range of Learning Scenarios
Develop AI Systems that Assist Teachers and Improve Teaching
Intensify and Expand Research on AI for Assessment of Learning
Accelerate Development of Human-Centered or Responsible AI
Develop Stronger Policies for Ethics and Equity
Inform and Involve Educational Policy Makers and Practitioners.
Strengthen the Overall AI and Education Ecosystem

Seven recommendations from US expert panel

43 of 71

Human-centered and trustworthy AI

Delgado Kloos, C., et al. (2022), H2O Learn - Hybrid and Human-Oriented Learning: Trustworthy and Human-Centered Learning Analytics (TaHCLA) for Hybrid Education. IEEE Global Engineering Education Conference, EDUCON 2022,
HLEG-AI (High-Level Expert Group on Artificial Intelligence) (2019), “Ethics Guidelines for Trustworthy AI: Requirements of Trustworthy AI,” Available: https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines/1

44 of 71

Human-Centered Approaches ...

See also “sister” initiatives for Human-Centered approaches for the design and development of truly mixed human-AI initiatives for human empowerment

e.g., Recent EU call for funding of A HUMAN-CENTRED AND ETHICAL DEVELOPMENT OF DIGITAL AND INDUSTRIAL TECHNOLOGIES 2022 (HORIZON-CL4-2022-HUMAN-02)

45 of 71

And a few suggestions …

Bring together LA and Learning Design (LD)
Consider multiple needs and paths to use LA, implemented as adaptive (by system/agent) or adaptable (by users)
Bring the teacher in the loop and orchestrate LA with all stakeholders (OrLA)
Consider the consolidated model for LA
Adopt human-oriented workflows for LA solutions
Consider data storytelling and explanatory LA

46 of 71

LD-based process for LA solutions

1 – LA design: LD elements selected as targets for LA solution

2 – LA implementation:

2a. Data from LA targets is analyzed by the LA tool

Resulting LA informs: 2b.) orchestration, 2c.) assessment

Dimitriadis, Martínez-Maldonado & Wiley (2020)

47 of 71

Consolidated model for LA

Gasevic, Dawson & Siemens (2015)
Saint, Gasevic, Matcha, Ahmad & Pardo (2020)
Gasevic, Kovanovic & Joksimovic (2017) - figure
Reimann (2016)

48 of 71

Orchestrating LA (OrLA)

Prieto, Rodríguez-Triana, Martínez-Maldonado, Dimitriadis & Gašević (2019) - figures

49 of 71

LATUX workflow for LA solutions

Martinez-Maldonado, Pardo, Mirriahi, Yacef, Kay & Clayphan (2016) - figure
Holstein, McLaren & Aleven (2019)

50 of 71

Datastorytelling and explanatory LA

Echeverria, Martinez-Maldonado, Buckingham Shum, Chiluiza, Granda & Conati (2018) - figures

51 of 71

HCD principles for actionable LA solutions

Agentic positioning of teachers and other stakeholders
Integration of the learning design cycle and the LA design process
Reliance on educational theories to guide the LA solution design and implementation

Y.Dimitriadis, K.Wiley, & R.Martínez-Maldonado (2021)

52 of 71

Illustrative study

From Theory to Action:

Developing and Evaluating Learning Analytics for Learning Design

K. Wiley, Y. Dimitriadis, A. Bradford, & M. Linn (2020)
K. Wiley (2020)
Y. Dimitriadis, K. Wiley, & R. Martínez-Maldonado (2021)

53 of 71

An overview of the study

Design and development of Teacher Action Planner, a LA tool that supports teachers’ orchestration actions:

Grounded on learning theory (Knowledge Integration) and using the Inquiry Based Learning approach.
Aligned with the Learning Design (Global Climate Change and Photosynthesis Units) and platform (WISE)
Aligned with stakeholders’ needs (OrLA)
Functional within the constraints of the technical and learning environments

54 of 71

DBR research approach

Sandoval & Bell (2004) – figure adapted from

55 of 71

The role of Theory: KI framework

56 of 71

LD informed by Learning Theory

57 of 71

The feature (and data) to focus on

58 of 71

Validating the usefulness of LA (I)

Are the data used to generate the learning analytics

useful for understanding student learning?

59 of 71

Validating the usefulness of LA (II)

High Number of Attempts Predicts Performance on

Subsequent Explanation Item

60 of 71

Optimizing the LD based on LA

Optimization: Fuse two steps of the unit

61 of 71

Creating a useful LA solution towards pedagogical action

62 of 71

LA report

63 of 71

Evaluating the LA solution

64 of 71

The Teacher Action Planner (TAP)

65 of 71

Recommend orchestration actions

66 of 71

Human-Centered Design of LA

Eventually the benefits of enhanced agency, adoption and impact of the LA solutions overcome the costs of difficult, time and resource consuming participatory processes
All the important aspects of learning (cognitive, metacognitive, affective and social) are highly sensible and dependent on the context

Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred Learning Analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1

67 of 71

Some take-home messages (I)

Technology-enhanced learning (TEL) ecosystems

Especially hard to design and orchestrate (advising)

Teachers are essential stakeholders

LD and LA are both about learning and teaching

Human-Centered design is necessary despite its cost

Move from “demonstrators in a greenfield” to embedded tools and practices in authentic contexts

Tools are necessary to support stakeholders

Balanced use of AI agents and human expertise and actions through orchestration technology and distributed scaffolding

Keep the power of LA-based models

But complement with explanations, trust, privacy

68 of 71

Some take-home messages (II)

LA for understanding and optimizing learning

oriented to pedagogical interventions based on actionable insights

LA benefits from

Data Science, Learning Theory and Design

LA and LD are intrinsically interconnected

They should be jointly employed

Inter-stakeholder communication is essential

using multiple design techniques and approaches

Bring the human in the loop

Through participatory user-centered design processes

Support teachers (and learners) with

technological and conceptual tools

69 of 71

Some HCLA challenges

Can design processes from other disciplines, such as HCI, Co-Design and Participatory design, be unproblematically adopted for HCLA, or do they require adaptation?
What are the obstacles to the adoption of HCLA design processes?
How can the voice of students be taken more into account, besides the dominant thread of involving teachers?
What are the lessons learnt from mid-to-long term HCLA studies and how do they inform the aforementioned topic of adoption?
HCLA beyond conventional higher education
A wider view of human-centeredness
Human-AI complementarity

70 of 71

The Future of HCLA?

71 of 71

Let’s remember:

Learning Analytics are about

… Learning

… Learners

…Teachers

… Humans

… Society

This is why

Human-Centered Learning Analytics

may be worth considering