ILYA GOLDIN
Outcomes-Aligned Data Science

Most recent resume: goo.gl/mHPtvn

LinkedIn: linkedin.com/in/ilyagoldin

Publications: researchgate.net/profile/ilya_goldin2


SUMMARY

I aim to understand complex social problems through data and to address them through technology.

ilyagoldin@gmail.com

+1 (412) 329 4592

US Citizen


EDUCATION

University of Pittsburgh —
Ph.D., 2011; M.S., 2007

Stevenson University —
B.S., Magna Cum Laude, 1999


EXPERIENCE

Director, Data Science, 2U, Inc.

2015-present

Provided organizational guidance on data science, learning science and educational technology. Worked with product management to design new feature agenda, and to articulate market positioning and differentiation.

Developed data-driven algorithms for digital learning: to describe fundamental limitations on discovery of domain models; to align noisy log files to expected event schedules; to define a computational model of course content and learning outcomes. Primarily working in R, these inventions led to internal products and scholarly publications.

Research Scientist, Center for Digital Data, Analytics & Adaptive Learning, NCS Pearson

2013-2015

Developed data-driven algorithms for digital learning: ways to probabilistically estimate student mastery of a skill, to discover prerequisite skill relationships, to flag learning activities for editorial attention, and others. Primarily working in R, these inventions led to internal prototypes, patent applications and scholarly publications.

Planned and supervised field-testing of Pearson technologies with hundreds of K-12 students in Utah and Michigan, and derived insights from data that drove product revisions

Leveraged academic background and relationships to bring outside expertise to Pearson projects, to support colleagues'​ professional development, and to enhance Pearson's position as a thought leader in educational technology.

Post-doctoral Fellow, Carnegie Mellon University

2011-2013

Studied the effect of hints on learning by creating novel statistical models for data from tutoring software, and by designing and conducting in-school studies. Presented research results at academic conferences.

Data Administrator, Magee-Womens Research Institute and Foundation

2010-2011

Processed hundreds of gigabytes of data for biomedical informatics tasks using custom and third-party software and scripting in a Unix environment.

Research Assistant, JustSystems Evans Research, Inc.

2006-2009

Designed, developed and evaluated software and interfaces in machine learning, recommender systems, semantic modeling, visualization, natural language processing and ontologies.

Research & Development Intern, McCabe & Associates

1997-1998

Worked on software quality assurance, including unit, regression and function testing.


STRICTLY-REFEREED FULL-LENGTH PUBLICATIONS

Goldin, I. M. & Galyardt, A. (under review) On Feasibility of Using Learning Curve Analysis to Refine Domain Models

Galyardt, A., & Goldin, I. M. (2015). Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining.

Galyardt, A., & Goldin, I. (2015). Evaluating Simplicial Mixtures of Markov Chains for Modeling Student Metacognitive Strategies. In R. E. Millsap, D. M. Bolt, L. A. van der Ark, & W.-C. Wang (Eds.), Quantitative Psychology Research (Vol. 89, pp. 377–393).

Goldin, I., Pinkus, R.L., Ashley, K. (2015) Sensitivity and reliability of an instrument for assessing case analyses in Bioengineering Ethics education. Science and Engineering Ethics

Goldin, I., Renken, M., Galyardt, A., Litkowski, E. (2014) Individual differences in identifying sources of science knowledge. Proceedings of 9th European Conference On Technology Enhanced Learning

Almond, R. G., Goldin, I. M., Guo, Y., & Wang, N. (2014). Vertical and stationary scales for progress maps. Proceedings of 7th International Conference on Educational Data Mining

Goldin, I., Carlson, R. (2013) Learner differences and hint content. 16th International Conference on Artificial Intelligence and Education

Goldin, I., Ashley, K. (2012) Eliciting formative assessment in peer review. Journal of Writing Research, 4(2), 203-237.

Goldin, I., Koedinger, K., Aleven, V. (2012) Learner differences in hint processing. Proceedings of the 5th International Conference on Educational Data Mining. [Finalist for Best Paper Award]

Ashley, K., Goldin, I. (2011) Toward AI-enhanced computer-supported peer review in legal education. Proceedings of the 24th International Conference on Legal Knowledge and Information Systems

Goldin, I., Ashley, K. (2011) Peering inside peer review with Bayesian models. Proceedings of the 15th International Conference on Artificial Intelligence and Education

Goldin, I., Ashley, K. (2010) Eliciting informative feedback in peer review: importance of problem-specific scaffolding. Proceedings of the 10th International Conference on Intelligent Tutoring Systems

Goldin, I., Ashley, K., Pinkus, R.L. (2006) Assessing case analyses in Bioengineering Ethics education: Reliability and training.  Proceedings of the 8th International Conference on Engineering Education

Goldin, I., Ashley, K., Pinkus, R.L. (2001) Introducing PETE: computer support for teaching ethics. Proceedings of the 8th International Conference on Artificial Intelligence & Law


ADDITIONAL PUBLICATIONS

Goldin, I. M., Narciss, S., Foltz, P., & Bauer, M. (2017). New Directions in Formative Feedback in Interactive Learning Environments. International Journal of Artificial Intelligence in Education.

Goldin, I. M., Pavlik Jr, P., & Ritter, S. (2016). Discovering Domain Models in Learning Curve Data. Design Recommendations for Intelligent Tutoring Systems: Domain Modeling (Vol 4). US Army Research Laboratory

Goldin, I. M., & Galyardt, A. (2015). Convergent Validity of a Student Model: Recent-Performance Factors Analysis. Proceedings of 8th International Conference on Educational Data Mining (EDM)

Goldin, I.M. & Galyardt, A. (2015). On Feasibility of Using Learning Curve Analysis to Refine Domain Models. Meeting of the Advisory Board of the Generalized Intelligent Framework for Tutoring

Goldin, I., & Galyardt, A. (2015). Viz-R: Using Recency to Improve Student and Domain Models. Proceedings of 2nd ACM Conference on Learning at Scale

Forsgren Velasquez, N., Goldin, I. M., Martin, T., & Maughan, J. (2014). Learning aid use patterns and their impact on exam performance in online developmental mathematics. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Galyardt, A., & Goldin, I. M. (2014). Recent-Performance Factors Analysis. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Scheines, R., Silver, E., & Goldin, I. M. (2014). Discovering prerequisite relationships among knowledge components. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Goldin, I. M., & McLaren, B. M. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of the Workshop on Social, Motivational and Affective Dimensions of Learning through Social Interaction, at the 11th International Conference of the Learning Sciences (ICLS)

Goldin, I. M., & McLaren, B. M. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of 3rd Workshop on Intelligent Support for Learning in Groups, at the 12th International Conference on Intelligent Tutoring Systems (ITS)

Renken, M. D., Litkowski, E., Goldin, I. M., & Oliver, M. (2014). Students’ Recognition of the Value of Scientific Process When Evaluating Science Explanations. Presented at the American Psychological Association Annual Conference (APA)

Goldin, I. (2013) Towards Supporting Students and Instructors with Models of Peer Assessment. Invited symposium. 78th Annual Meeting of the Psychometric Society (IMPS)

Galyardt, A., Goldin, I. (2013). Modeling Student Metacognitive Strategies in an Intelligent Tutoring System. Presented at 78th Annual Meeting of the Psychometric Society (IMPS)

Goldin, I., Koedinger, K., Aleven, V. (2013) Hints: You Can't Have Just One. 6th International Conference on Educational Data Mining (EDM)

Goldin, I. (2013) Examining Peer Reviewer Calibration with Bayesian Models. It's About Time: Addressing the Many Challenges of Analyzing Multi-Scale Temporal Data. Workshop at Alpine Rendez-Vous

Goldin, I., Carlson, R. (2013) Learner Differences and Hint Content. Data Analysis and Interpretation for Learning Environments (DAILE). Workshop at Alpine Rendez-Vous

Goldin, I., Ashley, K., Schunn, C. (2012) Redesigning Educational Peer Review Interactions Using Computer Tools: An Introduction. Journal of Writing Research, 4(2), 111-119.

Goldin, I. (2012) Accounting for Peer Reviewer Bias with Bayesian Models. Proceedings of the Workshop on Intelligent Support for Learning Groups at the 11th International Conference on Intelligent Tutoring Systems

Goldin, I., Ashley, K., Pinkus, R.L. (2006) Teaching case analysis through framing: Prospects for an ITS in an ill-defined domain. Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains at the 8th International Conference on Intelligent Tutoring Systems

Goldin, I., Chapman, W.W. Learning to detect negation with ‘not’ in medical texts. Proceedings of the Workshop on Text Analysis and Search for Bioinformatics at the 26th International ACM SIGIR Conference


PATENT APPLICATIONS

Scheines, R.; Silver, E.; Goldin, I. "Discovering Prerequisite Relationships among Knowledge Components." U.S. Patent Application 62/188,184

Gonzalez-Brenes, J., Goldin, I.M., Larusson, J.A., Behrens, J., McTavish, T. "Teacher Facing Adaptive System." U.S. Patent Application 62/073,751

Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Anchoring Expressions Based on an Ontological Model of Semantic Information.” U.S. Patent Application 20080294426

Hull, D.; Evans, D.A.; Bennett, J.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantic Update and Replace Operations.” U.S. Patent Application 20080294425

Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantically Informed Text Operations.” U.S. Patent Application 20080295013

Cheng, H.; Evans, D.A.; Bennett, J.; Hull, D.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing a Semantically Informed Merge Operation.” U.S. Patent Application 20080294427

Qu, Y.; Evans, D.A.; Goldin, I. “Method and Apparatus for the Automated Construction of Models of Activities from Textual Descriptions of the Activities.” U.S. Patent Application 20080294398


GRANTS

Community Care Behavioral Health Organization. Analysis of Disengagement in Mental Health Provider Networks. PI, 2011, $12,000.

Provost's Advisory Council on Instructional Excellence, University of Pittsburgh. A Peer Review-Based Student Model for Ill-Defined Problem-Solving. Co-author with K. Ashley, 2009, $25,000.


PUBLIC SOFTWARE

Curtis, S.M. (author), Goldin, I. (contributor). (2012) mcmcplots: visualization of Markov-Chain Monte Carlo output for Bayesian modeling. http://cran.r-project.org/web/packages/mcmcplots

Goldin, I. (2012) ProfHelp and ProfHelp-ID: multilevel Bayesian models of learner differences with hint processing.

Goldin, I. (2012) Peer reviewer bias: multilevel Bayesian models of bias in peer assessment and bias-instructor score relationship.

Goldin, I. (2011) Rubrics in peer review: multilevel Bayesian models of multidimensional rubrics in peer review.

Goldin, I. (2010) Comrade: web-based peer review software. Used in 4 studies by 200 students and 3 instructors.

Goldin, I. (2001) Professional Ethics Tutoring Environment (PETE): web-based ethics tutoring software.


INVITED TALKS

Applications of Data Science in Education. Centre for Teaching, Learning and Technology, University of British Columbia (2016)

Domain Modeling using Learning Curve Analysis. Learning Analytics Summer Institute-local, New York University (2016)

On Formative Assessment in Interactive Learning Environments. Learning Sciences Institute, Arizona State University (2013)

Peering into Peer Review with Bayesian Models. Institute for Language, Cognition and Computation, University of Edinburgh, UK (2013)

Peering into Peer Review with Bayesian Models. Institute of Educational Technology, The Open University, UK (2013)

Towards Effective Feedback for All: Learner Differences in Hint Processing. Learning Analytics Seminar Series, Teachers College, Columbia University (2013)

Towards Effective Feedback for All: Learner Differences in Hint Processing. Artificial Intelligence Forum, Intelligent Systems Program, University of Pittsburgh (2012)


ACADEMIC SERVICE

Grant Application Reviewer

National Science Foundation, USA (2013)

Economic and Social Research Council, UK (2008)

Chair

Workshops, International Conference on Educational Data Mining (2016)

Interactive Events track, International Conference on Artificial Intelligence in Education (2015), with Olga Santos

Intelligent Support for Learning in Groups, a workshop at International Conference on Intelligent Tutoring Systems (2014) and International Conference on Artificial Intelligence in Education (2015), with Roberto Maltinez-Maldonado, Erin Walker, Jihie Kim, and Cindy Hmelo-Silver

Formative Feedback in Interactive Learning Environments, a workshop at International Conference on Artificial Intelligence and Education (2013), with Taylor Martin, Ryan Baker, Vincent Aleven and Tiffany Barnes

Crosspollination: Multidisciplinary Approaches to Human Cognition and Behavior, a workshop at inter-Science of Learning Centers (iSLC) Conference (2012), with Amy Ogan and So-One Hwang

Computer-Supported Peer Review in Education: Synergies with Intelligent Tutoring Systems, a workshop at International Conference on Intelligent Tutoring Systems (2010)

Guest Co-editor

International Journal of Artificial Intelligence in Education, Special Issue: Formative Feedback in Interactive Learning Environments (2017), with Peter Foltz, Susanne Narciss, Malcolm Bauer

Journal of Writing Research, Special issue: Redesigning Peer Review Interactions Using Computer Tools (2012)

Program Committee Member

International Conference on Intelligent User Interfaces (2016)

International Conference on Artificial Intelligence in Education (2015-2017)

International Conference on Educational Data Mining (2013-2017)

Artificial Intelligence and Feedback, a workshop at International Joint Conferences on Artificial Intelligence (2015)

Data Mining for Educational Assessment and Feedback (ASSESS), a workshop at KDD 2014

International Conference on Creating, Connecting and Collaborating through Computing (2011)

Journal Reviewer

International Journal of Artificial Intelligence in Education (2013, 2015, 2016)

Technology, Instruction, Cognition and Learning (2013)

IEEE Transactions on Learning Technologies (2013)

Journal of Educational Psychology (2012)

Journal of Educational Data Mining (2011)

Conference Reviewer

ACM CHI Conference on Human Factors in Computing Systems (2014)

Society for Research on Educational Effectiveness (Spring 2013)

International Conference on Artificial Intelligence in Education (2011)

International Conference on Educational Data Mining (2012-14, 2010)

International Conference on Computers and Education (2009)

Conference of Florida Artificial Intelligence Research Society (2009, 2008)

Internal Volunteering

Liaison to Executive Committee of the Pittsburgh Science of Learning Center (PSLC) from PSLC postdoctoral fellows (2012-13)


MENTORING

Laura Malkiewich, intern, 2U, Inc.

Yetian Chen, intern, Pearson Education, with Jose Gonzalez-Brenes

Ryan Carlson, Ph.D. student, Carnegie Mellon University, with Kenneth Koedinger and Carolyn Rose

LearnLab Summer School: Educational Data Mining track, Pittsburgh Science of Learning Center (2012, 2013)

Timothy Hirsh, intern, JustSystems Evans Research, Inc.


SKILLS & EXPERTISE

Educational technology: intelligent tutoring systems, computer-
supported collaborative
learning

Systems development: visualization, machine learning, data mining, information retrieval, natural language processing

Research: experiment design, qualitative and quantitative data collection and analysis

Statistics: R, generalized linear mixed effects, Bayesian and graphical models


AWARDS

MacArthur Foundation /
Educational Testing Service Edmund W. Gordon Fellow, 2013-2015

PostPIER Postdoctoral Training Grant, Institute of Educational Sciences, 2011-2013

Outstanding Paper Presentation Award, University of Pittsburgh Grad Expo, 2006, 2001

Faculty of Arts and Sciences Fellowship, University of Pittsburgh, 1999

President’s Award, Stevenson University, 1999

Outstanding Scholar Award, Stevenson University, 1999

First place, Computer Scholarship Competition, Stevenson University, 1996