NSF GRFP 2022-2023 Activities Report

1 Intellectual Merit

Over the past year, my research has led to several publications in peer-reviewed venues. My co-author and I published a paper titled “Logs or Self-Reports? Misalignment Between Behavioral Trace Data and Surveys When Modeling Learner Achievement Goal Orientation'' which explores the discrepancies between behavioral trace data and self-report surveys when applied to goal complex theory. Specifically, we find that although learners tend to report a desire to master their learning, the trace data suggests that they may not always engage in activities that support this goal. I have also published “Disparities in Students’ Propensity to Consent to Learning Analytics'' in the International Journal of Artificial Intelligence in Education (IJAIED). This work investigates the privacy considerations that influence learners’ decisions to share educational data and how that may be characterized across various subpopulations.

Additionally, I am excited to report that I have made progress on my dissertation and expect to defend it in the coming year. As part of my thesis, I am conducting a series of studies that explore how students reflect on their own learning when engaging with an AI model. Data collection for 2 of the 3 experiments is now complete. Last year, I worked on a project that explored student interactions with what they perceived to be an AI code annotation tool. Specifically, I am interested in judgments of learning (JOL) and shifts in someone’s confidence in their own knowledge. It has been shown that making accurate judgments is an important part of being an effective learner, but this is a skill with which many still struggle.

In my second study, we construct an OLM (open learner model) and pay particular attention to how these models are used during the learning process by incorporating eye-tracking technology. I also account for the type of task (whether problems involve simply recalling information versus applying knowledge), which may affect students’ ability to make accurate JOLs. There is a push to make AI more explainable, but even when provided, it is critical to understand how learners make use of this information. With the advent of even more sophisticated and popular models such as ChatGPT, these studies are critical in understanding how to best integrate new technology in the classroom to support self-regulation for all learners.

2 Broader Impacts

The IJAIED paper mentioned above was also presented at the AIED conference in Durham, UK, which provided a forum for me to disseminate my findings and share how university stakeholders can help avoid potential biases in predictive models. Furthermore, I had the opportunity to participate in a guest interview for the online course “Data Analytics in the Public Sector with R,” which is a collaboration between the Grow with Google initiative and the University of Michigan. The course is available on Coursera and is targeted for current or early-career professionals working in the public sector who are interested in learning how to effectively analyze public data. As part of this project, I met with the Center for Academic Innovation team to develop new content and material, which was recorded in a professional studio. In my interview, I presented my research data and demonstrated population bias in an R script to students.

Lastly, the paper “A LAK of Direction: Misalignment Between The Goals of Learning Analytics and its Research Scholarship,” which I co-authored, was accepted in the Journal of Learning Analytics. We examined the current state of learning analytics research and found that a significant portion of articles do not analyze data from learners in an educational setting or include measures of learning. These findings indicate that the scholarship in learning analytics research may lack a clear direction toward its stated goals. As a result, we spur critical discussion by inviting others in the community to respond. It is worth noting that this type of paper is a new venture for the journal and we hope that through open peer commentary, these issues can be collaboratively addressed to guide future research efforts.