Intelligent Adaptive Interventions Lab - Research Statement: tiny.cc/williamsresearch Conference Papers: tiny.cc/williamspapers Journal Papers: tiny.cc/williamsjournalpapers
(This CV Updated April 2025)
Research Summary: We develop methods for designing Intelligent Interventions for human Behaviour Change, by developing novel techniques for Adaptive Experimentation. High-impact applications span learning, mental & physical health, charitable giving, decision-making: Any situation where technology can influence people to do one action vs another. For example, we've developed interventions that use emails to boost students' performance on high-stakes tests by 4%, in just 3 minutes [AC.34].
To make Interventions more Intelligent and Adaptive, we develop and democratize tools & methods for Adaptive Experimentation, which expedite easier experimentation & evaluation of intervention components. We invented the AdaptEx (Adaptive Experiment) framework [AC.29] to transform technology touchpoints used by billions of people (e.g. text messages, emails, websites, apps), into agents that perpetually experiment, to discover and deploy whichever interventions best help a person in a particular context. We achieve this by reimagining randomized 'A/B' experiments as a bridge between human intelligence (e.g. crowdsourced input from users, designers, & scientists) and artificial intelligence (e.g. reinforcement learning, LLMs).
The AdaptEx framework has impacted over 500 000 people, in applications from education to mental & physical health. AdaptEx won a prestigious $1M Xprize for transforming education through AI-driven experimentation, and a $3M NSF grant to enable others access to this "Cyberinfrastructure".
To bridge research with real-world impact, my group publishes across human-computer interaction, experimental psychology, applied AI (LLMs), applied statistics and machine learning, digital education, mental health, & behaviour change. Our 10 year target is to investigate how to scale access to adaptive experimentation techniques, and generalize to many kinds of personalized interventions: Millions of people and organizations can benefit from changing their behaviour.
Our lab's work is represented in over 85 papers (www.intadaptint.org/papers), 2 Best Paper Awards (1 at CHI), 4 Runner Up/Honorable Mention for Best Paper (CHI, EDM, LAS), and 1st place in a $1M Xprize competition for the future of technology for experimentation. We've received over $2M in grant funding, enabling interventions impacting over 500 000 people. Over 4 graduate students have gone into research positions (2 with faculty positions, 1 a research scientist, and 1 a prestigious independent postdoc at Stanford).
Bio: Joseph Jay Williams is an Assistant Professor at the University of Toronto in Computer Science, with courtesy appointments supervising PhD students in Statistical Science, Psychology, and the Vector Institute for Artificial Intelligence, and courtesy appointments in Economics, Industrial Engineering & the Faculty of Information. His PhD Students span HCI (Human Computer Interaction), Cognitive/Social/Clinical/Health Psychology, applied ML (reinforcement learning), applied AI (LLMs), & Statistics. Joseph was previously an Assistant Professor in Information Systems & Analytics at National University of Singapore, Research Scientist at Harvard, Postdoc at Stanford, and did his PhD at UC Berkeley. He is originally from Trinidad and Tobago. Talks for different audiences are at tiny.cc/williamstalk, and his research agenda on Intelligent Adaptive Experimentation is summarized at the URL tiny.cc/williamsresearch.
Click on a link below to go to that section. E.g. Archival Refereed Conference Proceedings Papers (53) or Refereed Journal Articles (32).
Selected Ongoing Work: Working Papers/Under Review/Under Revision 2
Archival Refereed Conference Proceedings Papers (54) 3
Refereed Journal Articles (32) 8
Refereed Conference Proceedings Papers (14) 11
Fellowships, Grants, and Honors 15
Symposia & Workshops Organized/Chaired (16) 18
Undergraduate courses taught 21
Guest Lecturer for MOOC on Applications of Calculus 22
Instructor for Short Courses 22
CHI Course “Online A/B Tests & Experiments” 22
Scientific and Technology mentor for Lean Launchpad course (based on NSF Innovation Corps) 22
Group Leader for Workshop teaching management skills to Stanford Graduate Students 22
Mentor for "Educational Data Mining" Track, Graduate Summer School 22
Teaching Assistant/Graduate Student Instructor 22
K12 Tutoring & Teaching Experience 23
Graduate Student Supervision 23
PhD Dissertation Committee Member 23
Graduate Student Examination/Prelim Committees 23
Mentor for NIH PhD Training Grant on Online Cognitive Therapy 23
Mentorship of >60 Undergraduate Students at University of Toronto (2018-present) 23
Mentorship of 27 Undergraduate Students at UC Berkeley (2008-2013) 23
Outstanding Graduate Student Mentor Award, UC Berkeley (2013) 24
2018- present Assistant Professor, University of Toronto, Department of Computer Science (Budgetary).
2021- present Department of Statistical Sciences, Courtesy Graduate-Level Appointment.
2019- present Department of Psychology, Courtesy Graduate-Level Appointment.
2019- present Vector Institute for Artificial Intelligence Faculty Affiliate.
2024- present Faculty of Information, Courtesty Graduate-Level Appointment.
2023- present Department of Mech. & Industrial Engineering, Courtesy Graduate-Level Appointment.
2022- present Department of Economics, Courtesy Graduate-Level Appointment.
2017- 2018 Assistant Professor, National University of Singapore. Department of Information
Systems and Analytics, School of Computing.
2014- 2017 Harvard University Research Fellow & Research Scientist.
VPAL (Vice Provost for Advances in Learning) Research Group (& HarvardX).
Intelligent Interactive Systems Group, Computer Science, School of Engineering & Applied
Sciences.
2015- 2018 Research Scientist, Worcester Polytechnic Institute, Department of Computer Science.
2013- 2014 Stanford University Postdoctoral Research Scholar.
Open Learning Initiative, Lytics Lab, Office of the Vice Provost for Online Learning.
2007- 2013 Ph.D. Computational Cognitive Science.
University of California, Berkeley. Dissertation: The Subsumptive Constraints Account of
why explaining “why?” helps learning. (Advisors: Tania Lombrozo & Tom Griffiths).
2003- 2007 B.Sc. Cognitive Science & Artificial Intelligence, Mathematics. University of Toronto.
WP.5 Kumar, H, Musabirov, I., Shi, J., Lauzon, A., Choy, K.K., Gross, O., Kulzhabayeva, D., Williams, J.J. Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk. https://arxiv.org/pdf/2209.11344
WP.4 Athey, S., Byambadalai, U., Hadad, V., Krishnamurthy, S. K., Leung, W. & Williams, J. J. Contextual Bandits in a Survey Experiment on Charitable Giving: Within-Experiment Outcomes versus Policy Learning https://arxiv.org/abs/2211.12004
WP.3 Li, T., Nogas, J., Song, H., Kumar, H., Durand, A., Rafferty, Aa., Deliu, N., Villar, S.S., Williams, J.J. Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization. https://arxiv.org/abs/2112.08507
WP.2 Deliu, N., Villar, S., Williams, J. J. Efficient Inference Without Trading-off Regret in Bandits: An Allocation Probability Test for Thompson Sampling. https://doi.org/10.48550/arXiv.2111.00137
WP.1 Williams, J.J., Nogas, J., Deliu, N., Shaikh, H., Villar, S.S., Durand, A., Rafferty, A. Challenges in Statistical Analysis of Data Collected by a Bandit Algorithm: An Empirical Exploration in Applications to Adaptively Randomized Experiments. https://arxiv.org/pdf/2103.12198.pdf
This section [with paper identifiers "AC"] are archival conference proceedings papers at Computer Science venues, and conferences that have similar norms: Conference publications are seen as a top-tier target, and the paper is archival.
AC.54 Bhattacharjee, A., Xu, S., Rao, P., Zeng, Y., Meyerhoff, J., Ahmed, S. I., Mohr, D. C., Liut, M.,
Mariakakis, A., Kornfield, R. & Williams, J. J. “It Explains What I am Currently Going Through Perfectly to a Tee”: Understanding User Perceptions on LLM-Enhanced Narrative Interventions. Proceedings of the 2025 ACM Designing Interactive Systems Conference (ACM DIS).
AC.53 Bhattacharjee, A., Williams, J.J., Beltzer, M., Meyerhoff, J., Kumar, H., Song, H., Mohr, D.C.,
Mariakakis, A. and Kornfield, R., 2025. Investigating the Role of Situational Disruptors in Engagement with Digital Mental Health Tools. (To Appear) 2025 ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (CSCW 2025).
AC.52 Platform-based Adaptive Experimental Research in Education: Lessons Learned from Winning the XPrize Digital Learning Challenge. Musabirov, I., Reza, M., Moore, S., Chen, P., Kumar, H., Li, T., Song, H., Shi, J., Choy, K., Price, T., Stamper, J., Bier, N., Deliu, N., Villar, S., Rafferty, A., Durand, A…. & Williams, J. J. (2025). Proceedings of the 15th International Learning Analytics and Knowledge Conference (ACM LAK).
AC.51 Kumar, H., Yoo, S., Bernuy, A., Shi, J., Luo, H., Williams, J. J., Kuzminykh, A., Anderson, A. & Kornfield, R.. Large Language Model Agents for Improving Engagement with Behavior Change Interventions: Application to Digital Mindfulness. (To Appear) 2025 ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (CSCW 2025).
AC.50 Reza, M., Laundry, N., Musabirov, I., Dushniku, P., Yu, Z.Y., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A. and Williams, J.J. (2024) ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI 2024).
AC.49 Bhattacharjee, A., Zeng, Y., Xu, S., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S.I., Mariakakis, A., Czerwinski, M., Kuzminykh, A., Liut, M., and Williams, J.J. (2024). Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastination. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI 2024). *Honorable Mention for Best Paper (top 5%).
AC.48 Kumar, H., Li, T., Shi, J., Musabirov, I., Kornfield, R., Meyerhoff, J., Bhattacharjee, A., Karr, C., Nguyen, T., Mohr, D., Rafferty, A., Villar, S., Deliu, N., Williams, J. J. (2024, March). Using Adaptive Bandit Experiments to Increase and Investigate Engagement in Mental Health. Proceedings of the AAAI Conference on Artificial Intelligence (IAAI-24 Innovative Applications).
AC.47 Bernuy, A. Z., Sibia, N., Chen, P. Xu, J. J., Tran, E., Ye R., Pammer-Schlindler, V., Petersen, A., Williams, J. J., Liut, M., 2024. "Does the Medium Matter? An Exploration of Voice-Interaction for Self-Explanations." Proceedings of the 2024 ACM Designing Interactive Systems Conference (ACM DIS).
AC.46 Bhattacharjee, A., Gong, Z., Wang, B., Luckcock, T.J., Watson, E., Abellan, E.A., Gutman, L., Hsu, A. and Williams, J.J., 2024. “Actually I Can Count My Blessings'”: User-Centered Design of An Application to Promote Gratitude among Young Adults. 2024 ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (CSCW).
AC.45 Kumar, H., Musabirov, I., Reza, M., Shi, J., Wang, X., Williams, J. J., Kuzminykh, A., & Liut, M. (2024). Guiding Students in Using LLMs in Supported Learning Environments: Effects on Interaction Dynamics, Learner Performance, Confidence, and Trust. 2024 ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (CSCW).
AC.44 Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A., Stamper, J., & Liut, M. (2024). Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. In Proceedings of the Tenth Annual ACM Conference on Learning at Scale (ACM LAS).
AC.43 Musabirov, I., Zavaleta Bernuy, A., Chen, P., Liut, M., & Williams, J., J. (2024). Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science Education. In Proceedings of the 26th Western Canadian Conference on Computing Education (pp. 1-7)
AC.42 Bernuy, Z., A., Ye, R., Sibia, N., Nalluri, R., Williams, J. J., Petersen, A., Smith, E., Simion, B. & Liut, M. (2024). Student Interaction with Instructor Emails in Introductory and Upper-Year Computing Courses. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1(pp. 1477-1483) (SIGCSE).
AC.41 Sibia, N., Bui, G., Wang, B., Tan, Y., Zavaleta Bernuy, A., Bauer, C., Williams, J. J., Liut, M., & Petersen, A. (2024). Examining Intention to Major in Computer Science: Perceived Potential and Challenges. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (pp. 1237-1243). (SIGCSE)
AC.40 Bhattacharjee, A., Song, H., Wu, X., Tomlinson, J., Reza, M., Chowdhury, A.E., Deliu, N., Price, T., & Williams, J.J., 2023. Informing Users about Data Imputation: Exploring the Design Space for Dealing With Non-Responses. Proceedings of the 11th AAAI Conference on Human Computation and Crowdsourcing (AAAI HCOMP).
AC.39 Zavaleta Bernuy, A., Ye, R., Tran, E., Sibia, N., Mandal, A., Shaikh, H., Simion, B., Liut, M., Petersen, A. & Williams, J. J. (2023). Do Students Read Instructor Emails? A Case Study of Intervention Email Open Rates. In Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (pp. 1-12). (Koli)
AC.38 Zavaleta Bernuy, A., Sibia, N., Chen, P., Huang, C., Petersen, A., Williams, J. J., & Liut, M. (2023). VoiceEx: Voice Submission System for Interventions in Education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (pp. 585-586). (ITiCSE)
AC.37 Zavaleta Bernuy, A., Xu, J. J. N., Sibia, N., Williams, J. J., Petersen, A., & Liut, M. (2023). Self-Explanation Modality: Effects on Student Performance?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (pp. 641-641). (ITiCSE)
AC.36 Sibia, N., Zavaleta Bernuy, A., Williams, J. J., Liut, M., & Petersen, A. (2023). Student Usage of Q&A Forums: Signs of Discomfort?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (pp. 33-39). (ITiCSE)
AC.35 Bhattacharjee, A., Williams, J.J., Meyerhoff, J., Kumar, HA., Mariakakis, A. and Kornfield, R. (2023) Investigating the Role of Context in the Delivery of Text Messages for Supporting Psychological Wellbeing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI). *Best Paper Award* [Top 1% of papers]
AC.34 Reza, M., Zavaleta Bernuy, A., Liu, E., Li, Tong., Liang, Z., Barber, C., & Williams, J. J. (2023) Exam Eustress: Designing Brief Online Interventions for Helping Students Identify Positive Aspects of Stress. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI).
AC.33 Ye, R., Chen, P., Mao, Y., Wang-Lin, A., Shaikh, H., Zavaleta Bernuy, A., & Williams, J. J. (2022) Behavioral Consequences of Reminder Emails on Students’ Academic Performance: a Real-world Deployment. In The 23rd Annual Conference on Information Technology Education (SIGITE). Best Paper Award.
AC.32 Zavaleta Bernuy, A., Han, Z., Shaikh, H., Zheng, Q. Y., Lim, L.A., Rafferty, Anna., Petersen, A., & Williams, J. J. (2022) How can Email Interventions Increase Students’ Completion of Online Homework? A Case Study Using A/B Comparisons. Proceedings of the 12th International Conference on Learning Analytics & Knowledge (ACM LAK).
AC.31 Bhattacharjee, A., Williams, J.J., Chou, K., Tomlinson, J., Meyerhoff, J., Mariakakis, A., & Kornfield, R. (2022) “I Kind of Bounce off It”: Translating Mental Health Principles into Real Life Through Story-Based Text Messages. ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (CSCW).
AC.30 Kornfield, R., Meyerhoff, J., Levin, H., Bhattacharjee, A., Williams, J. J., Reddy, M., & Mohr, D. C. (2022) Meeting Users Where They Are: User-centered Design of an Automated Text Messaging Tool to Support the Mental Health of Young Adults. Proceedings of the 2022 ACM Conference on Human Factors in Computing Systems (CHI).
AC.29 Reza, M., Kim, J., Bhattacharjee, A., Rafferty, A.N., & Williams, J.J. (2021) The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, & Personalization in Online Courses. In Proceedings of the Eighth Annual ACM Conference on Learning at Scale (ACM LAS).
AC.28 Yang, K. B., Nagashima, T., Yao, J., Williams, J. J., Holstein, K., & Aleven, V. (2021) Can Crowds Customize Instructional Materials with Minimal Expert Guidance? Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor. In Proceedings of the ACM on Human-Computer Interaction (CSCW).
AC.27 Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Nogas, J., Rafferty, Anna., Petersen, A., & Williams, J. J. (2021) Using Adaptive Experiments to Rapidly Help Students. Artificial Intelligence in Education (AIED).
AC.26 Zavaleta Bernuy, A., Zheng, Q. Y., Shaikh, H., Petersen, A., & Williams, J. J. (2021) Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE).
AC.25 Solyst, J., Thakur, T., Dutta, M., Asano, Y., Petersen, A., & Williams, J. J. (2021) Procrastination and Gaming in an Online Homework System of an Inverted CS1. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE).
AC.24 Xia, M., Asano, Y., Williams, J. J., Qu, H., & Ma, X. (2020) Using Information Visualization to Promote Students' Reflection on" Gaming the System" in Online Learning. In Proceedings of the Seventh ACM Conference on Learning@ Scale (ACM LAS).
AC.23 Price, T. W., Marwan, S., Winters, M., & Williams, J. J. (2020) An Evaluation of Data-Driven Programming Hints in a Classroom Setting. In the International Conference on Artificial Intelligence in Education (AIED).
AC.22 Li, Z., Yee, L., Sauerberg, N., Sakson, I., Williams, J. J., & Rafferty, A. N. (2020) Getting Too Personal(ized): The Importance of Feature Choice in Online Adaptive Algorithms. In Proceedings of the 13th International Conference on Educational Data Mining (EDM).
AC.21 Daskalova, N., Yoon, J., Wang, L., Beltran, G., Araujo, C., Nugent, N., McGeary, J., Williams, J.J. & Huang, J. (2020) SleepBandits: Guided Flexible Self-Experiments for Sleep. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI).
AC.20 Price, T.W., Williams, J.J., Solyst, J. & Marwan, S. (2020) Engaging Students with Instructor Solutions in Online Programming Homework. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI).
AC.19 Asano, Y., Solyst, J., & Williams, J. J. (2020) Characterizing and Influencing Students’ Tendency to Write Self-explanations in Online Homework. Proceedings of the 10th International Conference on Learning Analytics & Knowledge (ACM LAK).
AC.18 Shaikh, H., Modiri, A., Williams, J. J., & Rafferty, A. N. (2019) Balancing Student Success and Inferring Personalized Effects in Dynamic Experiments. Proceedings of the 12th International Conference on Educational Data Mining (EDM).
AC.17 Wanigasekara, N., Liang, Y., Goh, S. T., Liu, Y., Williams, J. J., & Rosenblum, D. S. (2019) Learning Multi-Objective Rewards and User Utility Function in Contextual Bandits for Personalized Ranking. Proceedings on the 28th International Joint Conference on Artificial Intelligence (IJCAI).
AC.16 Marwan, S., Williams, J. J., & Price, T. (2019) An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. Proceedings of the 15th International Computing Education Research Conference (ICER).
AC.15 Marwan, S., Lytle, N., Williams, J. J., & Price, T. W. (2019) The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. Proceedings of the Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE).
AC.14 Zhang, L., Craig, M., Kazakevich, M., & Williams, J. J. (2019) Experience Report: Mini Guest Lectures in a CS1 Course via Video Conferencing. Proceedings of the 1st ACM Global Computing Education Conference.
AC.13 Rafferty, A., Ying, H., & Williams, J. J. (2018) Bandit assignment for educational experiments: Benefits to students versus statistical power. Proceedings on the 19th International Conference on Artificial Intelligence in Education (AIED).
AC.12 Segal, A., David, Y. B., Williams, J. J., Gal, K., & Shalom, Y. (2018) Combining Difficulty Ranking with Multi-Armed Bandits to Sequence Educational Content. Proceedings on the International Conference on Artificial Intelligence in Education (AIED).
AC.11 Williams, J. J., Rafferty, A., Tingley, D., Ang, A., & Kim, J. (2018) Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems (CHI).
AC.10 Shin, H., Ko, E., Williams, J. J., & Kim, J. (2018) Understanding the Effect of In-Video Prompting on Learners and Instructors. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems (CHI).
AC.9 Foong, P. S., Zhao, S., Tan, F., & Williams, J. J. (2018) Harvesting Caregiving Knowledge: Design Considerations for Integrating Volunteer Input in Dementia Care. In CHI 2018, 36th Annual ACM Conference on Human Factors in Computing Systems (CHI).
AC.8 Macina, J., Srba, I., Williams, J. J., & Bielikova, M. (2017) Educational Question Routing in Online Student Communities. Proceedings of the 10th ACM Conference on Recommender Systems (ACM RecSys).
AC.7 Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K., & Heffernan, N. (2016) AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. Proceedings of the Third Annual ACM Conference on Learning at Scale (ACM LAS). *Nominee for Best Paper*.
AC.6 Williams, J. J., Lombrozo, T., Hsu, A., Huber, B., & Kim, J. (2016) Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalous Facts. Proceedings of CHI (2016), 34th Annual ACM Conference on Human Factors in Computing Systems (CHI). *Honorable Mention for Best Paper (top 5%).
AC.5 Ostrow, K., Selent, D., Wang, Y., VanIngwen, E., Heffernan, N., & Williams, J. J. (2016) The Assessment of Learning Infrastructure (ALI): The Theory, Practice, and Scalability of Automated Assessment. 6th International Learning Analytics & Knowledge Conference (ACM LAK).
AC.4 Whitehill, J., Williams, J. J., Lopez, G., Coleman, C., & Reich, J. (2015) Beyond Prediction: First Steps Toward Automatic Intervention in MOOC Student Stopout. In Proceedings of the 8th International Conference on Educational Data Mining (EDM). *Nominee for Best Paper.
AC.3 Krause, M., Mogale, M., Pohl, H., & Williams, J. J. (2015) A Playful Game Changer: Fostering Student Retention in Online Education with Social Gamification. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (ACM LAS).
AC.2 Pacer, M., Williams, J. J., Chen, X., Lombrozo, T., Griffiths, T. L. (2013) Evaluating computational models of explanation using human judgments. In Nicholson, A., & Smythe, P. (Eds.), Proceedings of the Twenty Ninth Conference on Uncertainty in Artificial Intelligence (UAI).
AC.1 Griffiths, T. L., Lucas, C. G., Williams, J. J., Kalish, M. L. (2008) Modeling human function learning with Gaussian processes. Advances in Neural Information Processing Systems 21 (NeurIPS).
J.32 Aguilera, A., Avalos, M. A., Xu, J., Chakraboty, B., Figueroa, C., Garcia, F., Rosales, K., Hernandez-Ramos, R., Karr, C., Williams, J. J., Ochoa-Frongia, L., Sarkar, U., Yom-Tov, E. & Lyles, C. 2024. Effectiveness of a Digital Health Intervention Leveraging Reinforcement Learning: Results From the Diabetes and Mental Health Adaptive Notification Tracking and Evaluation (DIAMANTE) Randomized Clinical Trial. Journal of Medical Internet Research.
J.31 Deliu, N., Williams, J.J., Chakraborty, B. 2024. Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions. International Statistical Review.
J.30 Bhattacharjee, A., Chen, P., Mandal, A., Hsu, A., O’Leary, K., Mariakakis, A., & Williams, J.J., 2024. Exploring User Perspectives on Brief Reflective Questioning Activities for Stress Management: Mixed-Methods Study. JMIR Formative Research.
J.29 Kornfield, R., Stamatis, C.A., Bhattacharjee, A., Pang, B., Nguyen, T., Williams, J.J., Kumar, H., Popowski, S., Beltzer, M., Karr, C.J., Reddy, M., Mohr, D.C., Meyerhoff, J., 2023. A text messaging intervention to support the mental health of young adults: User engagement and feedback from a field trial of an intervention prototype. Internet Interventions Volume 34, December 2023, 100667.
J.28 Choi, H., Jovanovic, J., Poquet, O., Brooks, C., Joksimović, S., Williams, J.J. 2023. The benefit of reflection prompts for encouraging learning with hints in an online programming course. The Internet and Higher Education, Volume 58, June 2023, 100903.
J.27 Xu, J., Yan, X., Figueroa, C., Williams, J.J., Chakraborty, B., 2023. A flexible micro-randomized trial design and sample size considerations. Statistical Methods in Medical Research.
J.26 Meyerhoff, J., Beltzer, M., Popowski, S., Karr, C.J., Nguyen, T., Williams, J.J., Krause, C.J., Kumar, H., Bhattacharjee, A., Mohr, D.C. and Kornfield, R., 2023. Small Steps over time: A longitudinal usability test of an automated interactive text messaging intervention to support self-management of depression and anxiety symptoms. Journal of Affective Disorders.
J.25 Kornfield, R., Stamatis, C.A., Bhattacharjee, A., Pang, B., Nguyen, T., Williams, J.J., Kumar, H., Popowski, S., Beltzer, M., Karr, C.J. and Reddy, M., 2023. A text messaging intervention to support the mental health of young adults: User engagement and feedback from a field trial of an intervention prototype. Internet Interventions, p.100667.
J.24 Bhattacharjee, A., Pang, J., Liu, A., Mariakakis, A., Williams, J.J., 2023. Design Implications for One-Way Text Messaging Services that Support Psychological Wellbeing. ACM Transactions on Computer-Human Interaction (TOCHI), 30 (3), pp. 1-29.
J.23 Figueroa, C.A., Gomez-Pathak, L., Khan, I., Williams, J.J., Lyles, C.R., Aguilera, A. Ratings and experiences in using a mobile application to increase physical activity among university students: implications for future design. Information Society, 05 January 2023
J.22 Meyerhoff, J., Nguyen, T., Karr, C.J., Reddy, M., Williams, J.J., Bhattacharjee, A., Mohr, D.C. and Kornfield, R., 2022. System design of a text messaging program to support the mental health needs of non-treatment seeking young adults. Procedia Computer Science, 206, pp.68-80.
J.21 Cai, W., Grossman, J., Lin, Z. J., Sheng, H., Wei, J. T. Z., Williams, J. J., & Goel, S. (2021). Bandit algorithms to personalize educational chatbots. Machine Learning, 110, 2389-2418. Doi
J.20 Pathak, L.E., Aguilera, A., Williams, J.J., Lyles, C.R., Hernandez-Ramos, R., Miramontes, J., Cemballi, A.G., & Figueroa, C. (2021). Combining user centered design and crowdsourcing to develop messaging content for a physical activity smartphone application tailored to low-income patients. JMIR mHealth and uHealth, 9(5), e21177
J.19 Figueroa, F.A., Aguilera, A., Chakraborty, B., Modiri, A., Aggarwal, J., Deliu, N., Sarkar, U., Williams, J.J., & Lyles, C.R. (2021). Adaptive learning algorithms to optimize mobile applications for behavioral health: guidelines for design decisions. Journal of the American Medical Informatics Association, ocab001, 1-10. doi
J.18 Aguilera, A., Figueroa, C.A., Hernandez-Ramos, R., Sarkar, U., Cemballi, A., Gomez-Pathak, L., Miramontes, J., Yom-Tov, E., Chakraborty, C., Yan, X., Xu, J., Modiri, A., Aggarwal, J., Williams, J.J., & Lyles, C.R. (2020). mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE Study. BMJ Open, 10, e034723. doi
J.17 Bernecker, S.L., Williams, J.J., Caporale-Berkowitz, N.A., Wasil, A.R., & Constantino, M.J. (2020). Nonprofessional peer support to improve mental health: randomized trial of a scalable web-based peer counseling course. Journal of Medical Internet Research, 22(9), e17164. Doi
J.16 Kizilcec, R.F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., Turkay, S., Williams, J.J., & Tingley, D. (2020). Scaling up behavioral science interventions in online education. Proceedings of the National Academy of Sciences, 117(26), 14900–14905. doi
J.15 Khosravi, H., Kitto, K., Williams, J.J. (2020) RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities. Journal of Learning Analytics, 6(3), 91-105. doi
J.14 Fischer, C., Pardos, Z. A., Baker, R. S. Williams, J.J., Smyth, P., Yu, R., Slater, S., Baker, R., & Warschauer, M. (2020). Mining Big Data in Education: Affordances and Challenges. Review of Research in Education.
J.13 Rafferty, A., Ying, H., & Williams, J. J. (2019). Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. Journal of Educational Data Mining.
J.12 Edwards, B.J., Williams, J.J., Gentner, D. & Lombrozo, T. (2019) Explanation recruits comparison in a category-learning task. Cognition. 185, 21-38. doi
J.11 Heffernan, N., Ostrow, K., Kelly, K., Selent, D., Vanlnwegen, E., Xiong, X., & Williams, J. J. (2016). The Future of Adaptive Learning: Does the Crowd Hold the Key? International Journal of Artificial Intelligence in Education, 1 -30. doi
J.10 Miyamoto, Y. R., Coleman, C. A., Williams, J. J., Whitehill, J., Nesterko, S., & Reich, J. (2015). Beyond Time-on-Task: The Relationship Between Spaced Study and Certification in MOOCs. Journal of Learning Analytics, 2(2), 47 - 69.
J.9 Walker, C. M., Lombrozo, T., Williams, J. J., Rafferty, A. N., Gopnik, A. (2016). Explaining Constrains Causal Learning in Childhood. Child Development, 88(1), 229-246.
J.8 Ostrow, K., Williams, J. J., & Heffernan, N. (in press). The Future of Adaptive Learning: Infusing Educational Technology with Sound Science. Teachers College Record.
J.7 Gumport, N. B., Williams, J. J., & Harvey, A. G. (2015). Learning cognitive behavior therapy. Journal of Behavior Therapy and Experimental Psychiatry, 48, 164-169. doi
J.6 Lucas, C. G., Griffiths, T. L., Williams, J. J., Kalish, M. L. (2015). A rational model of function learning. Psychonomic Bulletin & Review, 1-23. doi
J.5 Harvey, A.G., Lee, J., Williams, J. J., Hollon, S. Walker, M.P., Thompson, M. & Smith, R. (2014). Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science, 9, 161-179. doi
J.4 Williams, J. J., & Griffiths, T. L. (2013). Why are people bad at detecting randomness? A statistical argument. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1473-1490. doi
J.3 Williams, J. J., Lombrozo, T., & Rehder, B. (2013). The hazards of explanation: overgeneralization in the face of exceptions. Journal of Experimental Psychology: General, 142(4), 1006-1014. doi
J.2 Williams, J. J., & Lombrozo, T. (2013). Explanation and prior knowledge interact to guide learning. Cognitive Psychology, 66, 55-84. doi
J.1 Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and generalization: evidence from category learning. Cognitive Science, 34, 776-806. doi
B.1 Leung, W. & Williams, J.J. (2019). Multi-Armed Bandits in Education. In A. Sinatra, A. Graesser, X. Hu, K. Brawner, & V. Rus (Eds.), Design Recommendations for Intelligent Tutoring Systems: Volume 7 - Self-Improving Systems (pp. 71 - 76). Orlando, FL: U.S. Army Research Laboratory.
B.2 Williams, J. J., Kim, J., Glassman, E., Rafferty, A. (2016). Making Static Lessons Adaptive through Crowdsourcing & Machine Learning. In R. Sottilare, A. Graesser, X. Hu, A. Olney, B. Nye, and A. Sinatra (Eds.). Design Recommendations for Intelligent Tutoring Systems: Volume 4 - Domain Modeling (pp. 127 - 137). Orlando, FL: U.S. Army Research Laboratory.
RC.14 Kulzhabayeva, D., Williams, J.J., Danks, D., 2024. Dynamics of Causal Attribution. In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci 2024)
RC.13 Oliver, M., Renken, M., & Williams, J. J. (2018). Revising biology misconceptions using an online activity with retrieval practice and explanation prompts. Proceedings of the 13th International Conference of the Learning Sciences.
RC.12 Edwards, B. J., Williams, J. J., Gentner, D., & Lombrozo, T. (2014). Effects of comparison and explanation on analogical transfer. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
RC.11 Williams, J. J., Walker, C., Maldonado, S. G., Lombrozo, T. (2013). Effects of Explaining Anomalies on the Generation and Evaluation of Hypotheses. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
RC.10 Edwards, B.J., Williams, J.J., & Lombrozo, T. (2013). Effects of explanation and comparison on category learning. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
RC.9 Williams, J. J., Walker, C. M., Lombrozo, T. (2012). Explaining increases belief revision in the face of (many) anomalies. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society (pp. 1149-1154). Austin, TX: Cognitive Science Society.
RC.8 Walker, C. M., Williams, J. J., Lombrozo, T., & Gopnik, A. (2012). Explaining influences children's reliance on evidence and prior knowledge in causal induction. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society (pp. 114-119). Austin, TX: Cognitive Science Society.
RC.7 Williams, J. J., Lombrozo, T., & Rehder, B. (2011). Explaining drives the discovery of real and illusory patterns. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1352–1357). Austin, TX: Cognitive Science Society.
RC.6 Williams, J. J., & Lombrozo, T. (2010). Explanation constrains learning, and prior knowledge constrains explanation. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 2912–2917). Austin, TX: Cognitive Science Society.
RC.5 Williams, J. J., Lombrozo, T., & Rehder, B. (2010). Why does explaining help learning? Insight from an explanation impairment effect. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 2906–2911). Austin, TX: Cognitive Science Society.
RC.4 Williams, J. J., & Lombrozo, T. (2009). Explaining promotes discovery: evidence from category learning. In N.A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1186-1191). Austin, TX: Cognitive Science Society.
RC.3 Williams, J.J. & Lombrozo, T. (2009). Explaining promotes discovery: evidence from category learning. In Proceedings of the 35th annual conference of the Society for Philosophy and Psychology.
RC.2 Williams, J. J., & Griffiths, T. L. (2008). Why are people bad at detecting randomness? Because it is hard. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Meeting of the Cognitive Science Society (pp. 64-70). Austin, TX: Cognitive Science Society.
RC.1 Williams, J. J., & Mandel, D. R. (2007). Do evaluation frames improve the quality of conditional probability judgment? In D. S. McNamara & J. G. Trafton (Eds.), Proceedings of the 29th Annual Meeting of the Cognitive Science Society (pp. 1653-1658), Mahwah, NJ: Erlbaum.
EA.21 Kumar, H., Agrawaal, T. S., Choy, K. K., Shi, J., Williams, J.J. (2022). "Sounds like a Cheesy Radio Ad": Using User Perspectives for Enhancing Digital COVID Vaccine Communication Strategies for Public Health Agencies. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-7.
EA.20 Kumar, H., Yu, K., Chung, A., Shi, J., Williams, J.J. (2023). Exploring the Potential of Chatbots to Provide Mental Well-being Support for Computer Science Students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2, pp. 1339-1339.
EA.19 Reza, M. Liut, M., Williams, J.J. (2023). ABScribe: A/B Scribe: Leveraging a Human-AI Co-Writing Interface for Digital Experimentation on Content Variations in Education & Beyond. In the 10th Annual Conference on Digital Experimentation @ MIT.
EA.18 Bernuy, A.Z., Sibia, N., Chen, P., Huang, C., Petersen, A., Williams, J.J., Liut, M. (2023). VoiceEx: Voice Submission System for Interventions in Education. In 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (ITiCSE 2023). New York, NY: Association for Computing Machinery.
EA.17 Kumar, H., Wang, Y., Shi, J., Musabirov, I., Farb, N.A.S., Williams, J.J. (2023). Exploring the Use of Large Language Models for Improving the Awareness of Mindfulness. In 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23). New York, NY: Association for Computing Machinery.
EA.16 Bhattacharjee, A., Kulzhabayeva, D., Reza, M., Kumar, H., Seong, E., Wu, X., Rifat, M.R., Bowman, R., Kornfield, R., Mariakakis, A., Ahmed, S.I., Choudhury, M., Doherty, G., Czerwinski, M.P., Williams, J.J. (2023). Integrating Individual and Social Contexts into Self-Reflection Technologies. In 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23). New York, NY: Association for Computing Machinery.
EA.15 Solyst, J., Asano, Y., Williams, J. J. (2019). The instructor reads what you write: Encouraging introductory programming students to engage in self-explanation online. In the 6th Annual Conference on Digital Experimentation @ MIT.
EA.14 Liu, J., Yang, C., Wang, H., & Williams, J. J. (2019). NoteStruct: Scaffolding Note-taking while Learning from Online Videos. In CHI’19 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.
EA.13 Williams, J. J., Rafferty, A., Ang, A., Tingley, D., & Kim, J. (2017). Connecting Instructors and Learning Scientists via Collaborative Dynamic Experimentation. In CHI'17 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.
EA.12 Williams, J. J., Rafferty, A., Maldonado, S., Ang, A., Tingley, D., & Kim, J. (2017). MOOClets: A Framework for Dynamic Experimentation and Personalization. Proceedings of the Fourth ACM Conference on Learning@Scale.
EA.11 Krause, M., Hall, M., Williams, J. J., Caton, S., & Pripc, J. (2016). Connecting Online Work and Online Education at Scale. In CHI'16 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.
EA.10 Williams, J. J. & Heffernan, N. (2015). A Methodology for Discovering how to Adaptively Personalize to Users using Experimental Comparisons. In Late-Breaking Results Workshop of the 23rd Conference on User Modelling, Adaptation, and Personalization. Dublin, Ireland.
EA.9 Williams, J. J., Krause, M., Paritosh, P., Whitehill, J., Reich, J., Kim, J, Mitros, P., & Heffernan, N. (2015). Connecting Collaborative & Crowd Work with Online Education. Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (pp. 313-318). Vancouver, BC.
EA.8 Williams, J. J., Ostrow, K., Xiong, X., Glassman, E., Kim, J., Maldonado, S. G., Reich, J., & Heffernan, N. (2015). Using and Designing Platforms for In Vivo Educational Experiments. Proceedings of the Second ACM Conference on Learning@Scale, 409-412.
EA.7 Williams, J. J., Kim, J., Keegan, B. (2015). Supporting Collaborations between Instructors and Researchers using MOOClets. Proceedings of the Second ACM Conference on Learning@Scale, 413-416.
EA.6 Williams, J.J., Kizilcec, R., Russell, D., & Klemmer, S. (2014). Learning Innovations at Scale. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.
EA.5 Williams, J. J., Kovacs, G., Walker, C., Maldonado, S. G., & Lombrozo, T. (2014). Learning Online Via Prompts to Explain. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery.
EA.4 Williams, J. J., Sohl-Dickstein, J., Heffernan, N., Thille, C., & Mitchell, J. C. (2015). Practical and Scientific Benefits of Experiment-Guided Instructional Design of MOOClets and Educational Modules. In J. Davenport (chair), Theory Driven Design of Online Learning Systems, Data Structures and Analytic Techniques. Annual Meeting of the American Educational Research Association.
EA.3 Williams, J. J., Maldonado, S., Williams, B. A., Rutherford-Quach, S., & Heffernan, N. (2015). How can digital online educational resources be used to bridge experimental research and practical applications? Embedding In Vivo Experiments in “MOOClets”. Spring 2015 Conference of the Society for Research on Educational Effectiveness, SREE. Washington, D. C.
EA.2 Williams, J. J. & Lombrozo, T. (2014). Why does explaining “why?” help learning? A Subsumptive Constraints Account. In B. Rittle-Johnson (chair), Different Perspectives on the Role of Explanation and Exploration in Learning. Annual Meeting of the American Educational Research Association.
EA.1 Williams, J. J. & Lombrozo, T. (2010). Explaining promotes discovery: Evidence from category learning. In Proceedings of ICLS ‘10, 9th International Conference of the Learning Sciences, 490-491.
WP.1 Ho, A. D., Chuang, I. R., Reich, J., Coleman, C. A., Whitehill, J., Northcutt, C.G., Williams, J. J., Hansen, J. D., Lopez, G., & Petersen, R. (March 30, 2015). HarvardX and MITx: Two Years of Open Online Courses Fall 2012-Summer 2014.
W.11 Ritter, S., Heffernan, N., Williams, J.J., Lomas, D., Bicknell, K., Roschelle, J., Motz, B., McNamara, D., Baraniuk, R., Mallick, D.B., Kizilcec, R., Baker, R., Fancsali, S., Murphy, A. (2023). Fourth Annual Workshop on A/B Testing and Platform-Enabled Learning Research. L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale July 2023 Pages 254–256
W.10 Kumar, H., Musabirov, I., Williams, J.J., Liut, M. QuickTA: Exploring the Design Space of Using Large Language Models to Provide Support to Students. Workshop on Partnerships for Co-Creating Educational Content at the Learning Analytics and Knowledge Conference 2023 (LAK ’23), March 13, 2023, Arlington, TX, USA
W.9 Reza, M., Musabirov, I., Liut, M., Laundry, N., & Williams, J. J. (2023). A/B Testing as a Pedagogical Tool for Experiment-Inspired Design in HCI Classrooms. Paper presented at The 5th Annual Symposium on HCI Education (EduCHI)
W.8 Reza, M., Chowdhury, A., Li, A., Gandhamaneni, M., Williams, J. J. (2023). Experimenting with Experimentation: Rethinking The Role of Experimentation in Educational Design. Paper presented at the 3rd Annual Workshop on A/B Testing and Platform-Enabled Learning Research
W.7 Leung, W., & Williams, J. J. (2019). Enhancing education with instructor-in-the-loop algorithms. Paper presented at the Human+AI Modeling & Design workshop at the annual ACM CHI Conference on Human Factors in Computing.
W.6 Bernecker, S. L., Williams, J. J., & Constantino, M. J. (2017). Enhancing mental health through scalable training for peer counselors. Paper presented at the Computing and Mental Health symposium of the annual ACM CHI Conference on Human Factors in Computing Systems.
W.5 Williams, J. J., Li, N, Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L., & Heffernan, N. (2015). Using the MOOClet Framework as a Problem Formulation to apply Machine Learning to automatically improve modular online educational resources through Experimentation and Personalization. Paper presented at the Human-Propelled Machine Learning Workshop at the Conference on Neural Information Processing Systems.
W.4 Williams, J.J. & Williams, B. A. (2013). Using Randomized Experiments as a Methodological and Conceptual Tool for Improving the Design of Online Learning Environments. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems.
W.3 Williams, J.J. (2013). Applying Cognitive Science to Online Learning. Paper presented at the Data Driven Education Workshop at the Conference on Neural Information Processing Systems.
W.2 Williams, J.J. (2013). Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN.
W.1 Williams, J.J., & Poldsam, H. (2013). Providing implicit formative feedback by combining self-generated and instructional explanations. Paper presented at the Formative Feedback in Interactive Learning Environments Workshop, at the International Conference on Artificial Intelligence in Education, Memphis, TN.
G.15 UToronto DSI Data Science Institute. Developing Algorithms & Statistical Analysis Techniques for Adaptive Experimentation. $100,000. Principal Investigator.
G.14 Acceleration Consortium. Comparison of Traditional and Adaptive Experiments to Accelerate the Identification of MicroRNA in a high through-put Acute Respiratory Disease Syndrome in vitro model. $100,000. Principal Investigator.
G.13 DARPA AI Tools for Adult Learning Competition. QuickTA. $150 000.
G.12 Microsoft: Personalized Messaging Intervention for Young Non-Treatment Seeking Adults Delivered as a Public Health Service. 2022. Co-Principal Investigator.
G.11 NSF National Science Foundation: Frameworks for Intelligent Adaptive Experimentation: Enhancing and Tailoring Digital Education. (Award #2209819). August 1, 2022 - July 31, 2027. Co-Principal Investigator.
G.10 National Institute of Mental Health: Digital Mental Health Service for Non-Treatment Seeking Young Adults. (R34MH124960). 2021-2024. Co-Principal Investigator.
G.9 U.S Office of Naval Research: Investigating the Analysis of Data from Adaptive Experiments. (#N00014-21-1-2576). Principal Investigator.
G.8 Microsoft: Personalized Messaging Intervention for Young Non-Treatment Seeking Adults Delivered as a Public Health Service. 2021. Co-Principal Investigator.
G.7 NSERC: Enhancing Online User-Facing Systems through Dynamic Experimentation. $140,000, 2019 - 2025. Principal Investigator. #RGPIN-2019-06968
G.6 Connaught Fund: Continually Improving User Interfaces. $20,000, 2019 - 2021. Principal Investigator.
G.5 Office of Naval Research: Personalizing Explanations in Online Problems Using Multi-Armed Contextual Bandits. (N00014-18-1-2755). 2018-2021. Principal Investigator.
G.4 National Science Foundation: CIF21 DIBBs: PD: Enhancing and Personalizing Educational Resources through Tools for Experimentation. (1724889) $544,644. 2017 - 2020. Co-Principal Investigator.
G.3 National Science Foundation: SI2-SSE. Adding Research Accounts to the ASSISTments Platform: Helping Researchers Do Randomized Controlled Studies with Thousands of Students. (1440753) $486,000. 2014 - 2017. Co-Principal Investigator.
G.2 Gates Foundation & Athabasca MOOC Research Initiative. Investigating the benefits of embedding motivational messages in online exercises. 2013-2014. ($22 500). Principal Investigator.
G.1 National Institutes of Health Grant Supplement. Improving Learning from Online Cognitive Therapy (2012, $100 000). Co-wrote successful award with Principal Investigator Allison Harvey.
Natural Sciences & Engineering Research Council of Canada Post-Graduate Doctoral Scholarship (2009–2012).
University of California at Berkeley Regents Intern Fellowship (2007-2009).
“Domain Modeling Techniques.” 2015 collaborative project by Army Research Lab, University of Memphis IIS, and ADL.
1st place in $1M Xprize: Transforming Education through Experimentation (2023).
DARPA Learning Tools for Adult Education Winners (with Tutorgen).
Best Paper Award, ACM CHI Conference on Human Factors in Computing Systems (2023). [top 1% papers]
Best Paper Award, SIGITE (2022).
Vector Best Poster Award, Toronto Machine Learning Conference (2018).
Nominee for Best Paper Award, 3rd ACM Conference on Learning at Scale (2016). [top 4 papers]
Nominee for Best Paper Award, 8th Conference on Educational Data Mining (2015). [top 3 papers]
Honorable Mention, CHI (2022, 2016). [top 5% papers]
Outstanding (Undergraduate Student) Mentor Award, UC Berkeley (2013).
Semi-finalist, Best poster award, 35th Conference of Society for Philosophy and Psychology (2009).
Trinidad and Tobago Undergraduate Full National Scholarship (2003-2007).
ACM Learning @ Scale Conference (A/B Testing) Co-Organizer of Annual Conference
CHI (Computer Human Interaction) Subcommittee Co-Chair: Learning, Education and Families
EDM (Educational Data Mining) Workshops Co-Chair
Statistical Machine Learning. NeurIPS (Advances in Neural Information Processing Systems), AAAI (Association for the Advancement of Artificial Intelligence), AI Stats (Artificial Intelligence and Statistics Conference).
Human-Computer Interaction. ACM Learning at Scale, CHI (Association for Computing Machinery Conference on Human Factors in Computing Systems), CSCW (Computer Supported Cooperative Work and Social Computing), Human-Computer Interaction (Journal), MobileHCI (International Conference on Human-Computer Interaction with Mobile Devices and Services).
Cognitive Science & Psychology. Annual Conference of the Cognitive Science Society, Social Cognition, Society for Philosophy and Psychology, WIREs (Wiley Interdisciplinary Reviews) Cognitive Science, Journal of Experimental Psychology: Learning, Memory & Cognition, TopiCS in Cognitive Science.
Education. ACM Learning at Scale, EDM (Educational Data Mining Society Conference), Educational Psychology Review, Journal of Science Education and Technology (JOST), IEEE Transactions on Learning Technologies, Instructional Science, Cambridge University Press.
AAAI, Association for the Advancement of Artificial Intelligence
AI Stats
ACM Learning at Scale
ACM UMAP, User Modeling, Adaptation and Personalization
AIED, Artificial Intelligence in Education
CHI (Specific Applications), ACM Conference on Human Factors in Computing Systems
CogSci, Annual Meeting of the Cognitive Science Society
EDM, Educational Data Mining Conference
HComp (AAAI Conference on Human Computation and Crowdsourcing)
ITS, Conference on Intelligent Tutoring Systems
LWMOOCs, Learning with MOOCs conference
S.21 Ritter, S., Fancsali, S. E., Murphy, A., Heffernan, N., Motz, B., Mallick D. B., Roschelle, J. McNamara, D., Williams, J. J. (2024) 5th Annual Educational A/B Testing at Scale Workshop. ACM Learning @ Scale Workshop 2024 (LAS).
S.20 Musabirov, I., Zhang, R. Williams, J. J. (2023) Learnersourcing for Co-Designing Motivational Systems. Partnerships for Cocreating Educational Content Workshop at LAK.
S.19 Ritter, S., Heffernan, N., Williams, J. J., Lomas, D., Bicknell, K., Roschelle, J., Motz, B., McNamara, D., Baraniuk, R., Mallick D. B., Kizilcec, R., Baker, R., Fancsali, S. E., Murphy, A. (2023) 4th Annual Educational A/B Testing at Scale Workshop. ACM Learning @ Scale Workshop 2023 (LAS).
S.18 Ritter, S., Heffernan, N., Williams, J. J., Lomas, D., Motz, B.Mallick D. B., Bicknell, K., McNamara, D., Kizilcec, R., Roschelle, J., Baraniuk, R., Baker, R. (2022) 3rd Annual Educational A/B Testing at Scale Workshop. ACM Learning @ Scale Workshop 2022 (LAS).
S.17 Ritter, S., Heffernan, N., Williams, J. J., Lomas, D., Bicknell, K. (2021) 2nd Annual Educational A/B Testing at Scale Workshop. ACM Learning @ Scale Workshop 2021 (LAS).
S.16 Ritter, S., Heffernan, N., Williams, J. J., Settles, B., Grimaldi, P., Lomas, D. (2020) 1st Annual Educational A/B Testing at Scale. ACM Learning @ Scale Workshop 2020 (LAS).
S.15 Diving in to Educational Experiments: Process, Evaluation, and Reasoning in Support of Learning (DEEPER Support of Learning). ACM Learning Analytics & Knowledge Workshop 2019 (ACM LAK).
S.14 Azizsoltani, H., Ausin, M. S., Rafferty, A., Williams, J. J., Kim, Y. J., Barnes, T., Chi, M. (2019). Reinforcement Learning for Educational Data Mining. Educational Data Mining Workshop (EDM).
S.13 Williams, J. J., Heffernan, N., & Poquet, O. (2018). Design and Application of Collaborative, Dynamic, Personalized Experimentation. Workshop conducted at the 19th International Conference on Artificial Intelligence in Education (AIED).
S.12 Williams, J. J., Abbasi, Y., Doshi-Velez, F. (2015). Machine Learning From and For Adaptive User Technologies: From Active Learning & Experimentation to Optimization & Personalization. 29th Annual Conference on Neural Information Processing Systems (NeurIPS).
S.11 Krause, M., Hall, M., Williams, J. J., Caton, S., & Pripc, J. (2016). Connecting Online Work and Online Education at Scale. In CHI'16 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery (CHI).
S.10 Williams, J. J., Krause, M., Paritosh, P., Whitehill, J., Reich, J., Kim, J, Mitros, P., & Heffernan, N. (2015). Connecting Collaborative & Crowd Work with Online Education. Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing (pp. 313-318). Vancouver, BC (CSCW).
S.9 Williams, J.J., Kizilcec, R., Russell, D., & Klemmer, S. (2014). Learning Innovations at Scale. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery (CHI).
S.8 Williams, J. J., Linn, M., Edwards, A., Trumbore, A., Chae, H. S., Natriello, G., Siemens, G., & Mitros, P. (2014). How online resources can facilitate interdisciplinary collaboration. Featured presentation & panel at the Special Interest Group on Computer and Internet Applications in Education, Annual Meeting of the American Educational Research Association (AERA).
S.7 Krause, M., Paritosh, P., & Williams, J. J. (2014). Crowdsourcing, Online Education, and Massive Open Online Courses. Workshop conducted at the Second AAAI Conference on Human Computation and Crowdsourcing (AAAI HCOMP).
S.6 Williams, J.J., Goldstone, R.L., Rafferty, A., McClelland, J, & Mozer, M. (2014). Computational Models for Learning: From Basic Processes to Real World Education. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA (APS).
S.5 Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium conducted at the annual convention of the Association for Psychological Science. San Francisco, CA (APS).
S.4 Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society (CogSci).
S.3 Williams, J. J., Saxberg, B., Means, B., Mitros, P. (2013). Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science (APS).
S.2 Williams, J. J., Legare, C., Gentner, D., McNamara, D. (2013). Enhancing education: The role of comparing and explaining examples in promoting abstraction and transfer. Symposium conducted at the annual convention of the Association for Psychological Science (APS).
S.1 Chi, M. T. H., DeJong, G., Legare, C., Lombrozo, T., Williams, J. J. (2011). Explanation-based mechanisms for learning: An interdisciplinary approach. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 2756–2757). Austin, TX: Cognitive Science Society (CogSci).
University of Michigan (Academic Innovation at Michigan Analytics), University of Edinburgh (School of Informatics), University of Pittsburgh (Intelligent Systems Program AI Forum), Brown University (Department of Cognitive, Linguistic, and Psychological Sciences), Worcester Polytechnic Institute (Computer Science), University of Freiburg (Educational Psychology), University of Amsterdam (Psychology), Army Research Lab Generalized Intelligent Framework for Tutoring Symposium, Carnegie Mellon University (Human Computer Interaction Institute, School of Computer Science), Qualtrics Insight Summit, McGraw Hill Education Meetup on Predictive Analytics for Education, MIT (CSAIL Computational Cognitive Science Group), Harvard University (HarvardX), Udacity, Declara, Stanford University (Psychology, Cognition Frisem), Arizona State University (Learning Sciences Institute), Coursera, Stanford University (Instructional Design Special Interest Group), Stanford Psychological Interventions in Educational Settings (PIES) group, Stanford University (Human Computer Interaction brown Bag Seminar), International Association for K-12 Online Learning (iNACOL) Webinar Series, EdX, Pittsburgh Science of Learning Center LearnLab Summer School, Stanford University (Lytics Online Education Lab), UC Berkeley (Graduate School of Education), UC Berkeley (Psychology), Google (Tech Talk), Khan Academy, EdX, Bloomsburg Corporate Advisory Council, Vector Institute for Artificial Intelligence (Vector Talks), Element AI, Spring Symposium on Artificial Intelligence for K-12 Education, UniSA Keynote, Smith School of Business, HarvardX, Singapore Mental Health Workshop, Vector, Michigan Stats.
University College London Interaction Centre (UCLIC). Perpetually Enhancing and Personalizing Technology for Learning & Health Behavior Change: Using Randomized A/B Experiments to integrate Psychology, Crowdsourcing & Statistical Machine Learning
MILA (a Quebec Institute for Machine Learning & Artificial Intelligence). Combining Reinforcement Learning & Human Computation for A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces
North Carolina State University, Computer Science & Center for Educational Informatics. Enhancing & personalizing educational & health technology: Randomized A/B Experiments to bridge human-computer interaction, psychology, and statistical machine learning
University of Washington Design. Use. Build. (DUB). Perpetually Enhancing and Personalizing Technology for Learning & Health Behavior Change: Using Randomized A/B Experiments to integrate Human-Computer Interaction, Psychology, Crowdsourcing & Statistical Machine Learning. (Video: https://vimeo.com/393368072)
Stanford Pervasive Wellbeing Technology Lab (Medical School). Combining Reinforcement Learning, Psychology & Human Computation for Randomized A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces.
University of Alberta Tea Talk. Combining Reinforcement Learning & Human Computation for A/B Experimentation: Perpetually Enhancing and Personalizing User Interfaces.
Carnegie Mellon University LearnLab. Applying Multi-Armed Bandits (Reinforcement Learning)
to A/B Testing to Enhance & Personalize Education.
University of Toronto Social-Personality Research Group. Conducting Adaptive Field Experiments that
Enhance and Personalize Education and Health Technology.
University of Columbia, Statistics. Challenges & Opportunities in using Reinforcement Learning for Dynamic Field Experimentation in User Interfaces: Tradeoffs between Statistical Inference and Enhancing User Experience.
University of Pennsylvania Duckworth Lab. Doing Psychology Research in the Real World: Using Technology for Adaptive Field Experiments in Education and Health.
Office of Naval Research Program Meeting. Enhancing Digital Resources using Multi-Armed Bandits for Crowdsourced & Dynamic Experimentation.
University of Delft. Changing Behaviour with Adaptive Interventions: How AI and MicroExperimentation can Enhance Health Behaviour, Education & Design.
TEDx Port-of-Spain (Trinidad). Is the internet replacing teachers? bit.ly/tedxwilliams
Taught 4th year undergraduate course at the University of Toronto (Computer Science, CSC 428), focused on understanding human behaviour as it applies to user interfaces.
Cross-listed as graduate course 2514
Taught CS1 1st year undergraduate course at University of Toronto (Computer Science, CSC 108), introduction to programming using python.
Supervisor for a 2nd/3rd year undergraduate research course at University of Toronto (Computer Science, CSC299/CSC399). Supervised >45 undergraduate students from 2020-2024 as they completed research projects in my lab.
Supervisor for 4th year undergraduate research course at University of Toronto (Computer Science, CSC 494/495). Supervised >6 undergraduate students from 2019-2020 as they completed individual research projects in my lab.
Taught 4th year undergraduate course at National University of Singapore (Information Systems & Analytics, School of Computing).
Graduate seminar at University of Toronto (Computer Science, CSC 2558, Multidisciplinary Topics in HCI), covering approaches from multiple disciplines (HCI, machine learning, psychology/cognitive science) related to use of randomized A/B experiments to enhance user-facing technology.
Graduate course at University of Toronto (Computer Science, CSC 2514) focused on understanding human behaviour as it applies to user interfaces.
Created 4 lecture videos on applying statistics to educational assessment (item response theory) and dynamic experimentation (multi-armed bandits).
“In Vivo Experimentation” Track, Graduate Summer School, Carnegie Mellon Learning Technology Summer School.
Taught by Steve Blank and Jerry Engels (2013). Mentored Ed-Tech & other startups, using the Launchpad Central online tool and in-person classes similar to what the instructors founded as NSF’s Innovation Corps (which teaches scientists to translate research into businesses).
Taught management skills to a group of Stanford graduate students, through a program sponsored by the Vice Provost for Graduate Education (2014).
Carnegie Mellon Learning Technology Summer School.
Research and Data Analysis.
Basic Issues in Cognition.
Classroom tutor placed to support new math teachers, 9th grade Geometry and Algebra.
Developed & delivered K-12 & university problem-based classes on chemistry.
(Angela Zavaleta Bernuy, Computer Science PhD Student, University of Toronto, 2020-2025)
(Ananya Bhattacharjee, Computer Science PhD student, University of Toronto, 2020-2025)
(Fred Haochen, Statistics PhD student, University of Toronto, 2022-present)
(Mohi Reza, Computer Science PhD student, University of Toronto, 2020-2025)
(Ilya Musabirov, Computer Science PhD student, University of Toronto, 2021-2025)
(Tong Li, Statistics PhD student, University of Toronto, 2021-present)
(Nathan Laundry, Computer Science PhD student, University of Toronto, 2023-present)
(Zahra Hassanzadeh, PhD Student, University of Toronto, 2024-present)
(Arghavan Modiri, Computer Science Master's Student, University of Toronto, 2019-2021)
(Jacob Nogas, Computer Science Master's Student, University of Toronto, 2019-2021)
(Sam Cox, Computer Science PhD Student, National University of Singapore, 2018-2019)
Lillio Mok, University of Toronto, Computer Science
Laura Niss; University of Michigan, Statistics
Mohammad Kian Kianpisheh, University of Toronto, Computer Science
Vaunam Venkadasalam, University of Toronto, Ontario Institute for Studies in Education – OISE
Zhicong Lu, University of Toronto, Computer Science
Amna Liaquat, University of Toronto, Computer Science
Devangini Patel, National University of Singapore, Computer Science
Merrin Oliver, Georgia State University, Psychology
Laura Niss; PhD Committee Member; University of Michigan, Statistics
Ding Dan & Luo Kai; Graduate Paper Examiner; National University of Singapore, Information Systems
Yuanxin Xiang; Master's Thesis Examiner; National University of Singapore, Computer Science
(Samantha Bernecker, University of Massachusetts Amherst, 2013-2017).
Including: Emmy (Meng) Liu (CMU), Yuya Asano (UPitt), Ben Prystawski (Stanford), Jai Aggarwal (UofT), Samantha Quinto, Gavin Zheng.
Students: Stephen Frey, Joyce Liu, Lindsay Reno, Christopher Gioia, Jerry Ling, Tram Dinh, Tim Ko, Summer Kim, Evan Kim, Lucie Vosicka, Jing Wang, Emily Margolin, Vanessa Ing, Sean Trott, Kelly Whiteford, Adam Krause, Samuel Maldonado, Norielle Adricula, Amanda Hu, Dhruba Banerjee, Preeti Talwai, Christopher Hur, Hava Edelstein, Jared Lorenz, Ania Jarosewicz, Olivia Lin, David Oh.
Departmental award for mentoring undergraduate students in conducting research.