IX International scientific and practical conference �“Applied Information Systems and Technologies in the �Digital Society” (AISTDS-2025) Kyiv, Ukraine 01 October�

1

MACHINE LEARNING FOR PREDICTIVE EVALUATION OF STUDENTS’ INTERACTIONS WITH AI-GENERATED CONTENT AND THEIR CRITICAL THINKING LEVELS

​

Nataliia Dziubanovska, Vadym Maslii

�

In contemporary educational settings, generative artificial intelligence (AI) models are increasingly used by students as sources of information, ideas, and ready-made solutions. On the one hand, this creates new opportunities to accelerate learning; on the other, it heightens the risks of uncritical acceptance of automatically generated responses and diminished motivation for independent thinking. Under these conditions, the ability to verify the reliability of AI-generated content and to seek corroboration in authoritative sources becomes particularly important. Verification practices, in turn, may function as a factor that either supports – or, conversely, weakens – the development of critical thinking among higher-education students.

​

The relevance of this study stems from the need to empirically determine how students interact with AI-generated content and which behavioral strategies are most closely associated with high levels of critical thinking. To this end, we use data from a survey on AI-use practices in learning and results from the standardized Watson–Glaser Critical Thinking Appraisal. Subsequent application of machine learning methods enables the classification of student behavior types in the context of AI use and allows for a more precise assessment of how specific strategies for engaging with AI-generated content may influence the development of critical thinking.

​

The study’s methodological approach is aimed at empirically assessing how the use of generative AI models for educational purposes may influence the development of students’ critical thinking. The empirical base was formed from a survey of first- and second-year full-time bachelor’s students enrolled in the Faculties of Economics and Management and Finance and Accounting at West Ukrainian National University (Ternopil). We focused on students in humanities-oriented economics programs, proceeding from the assumption that students in technical fields – owing to more intensive mathematical training – typically possess more advanced analytical competencies, which may serve as a natural factor supporting critical thinking [10]. In turn, we argue that economics students with a humanities profile warrant dedicated investigation into how interaction with generative AI affects the development of argumentative and evaluative cognitive skills.

To establish the internal consistency of the adapted Watson–Glaser test and to examine the structure of relationships among its components, we conducted a correlation analysis at the level of individual subtests. Pearson correlation coefficients were used to test the extent to which the five cognitive components of the test are associated with the overall critical-thinking score.

To build a predictive model classifying students by their interaction types with generative AI, we applied binary logistic regression with a target variable representing critical-thinking level: High CM (1) for students with a Watson–Glaser total ≥ 18 and Non-High CM (0) for all others (≤ 17). Predictors comprised behavioral indicators of AI interaction: the frequency of verifying AI-provided information, the degree of doubt about its accuracy, and the willingness to consult instructors/peers. Features were standardized; model training used L2 regularization to minimize overfitting. Predictor informativeness was assessed via model coefficients and odds ratios (OR), interpreted on the standardized scale. Model quality was evaluated using ROC–AUC. All computations were performed in Python.

7

Figure 1: Frequency of AI Use in the Educational Process

8

Figure 2: Distribution of Watson–Glaser Test Scores

9

Figure 3: Correlation between Watson–Glaser Subtests and Total Score

10

Figure 4: Correlations between Questionnaire Items and Total Watson–Glaser Score

11

Figure 5: PCA projection of behavioural predictors and total CM

12

Figure 6: ROC Curve: Logistic Regression (High vs Medium+Low CM).

13

Figure 7: Odds ratios for behavioural predictors obtained from logistic regression model

​

Conclusions

The results suggest that the influence of generative AI on the development of critical thinking is non-linear and cannot be reduced to mere access to the technology. The decisive factor is the behavioral mode of interaction in which a student engages with AI as an information source. Our data show that the strongest links with high levels of critical thinking are verification-oriented actions – fact-checking, consulting alternative sources, and social verification through consultation. This implies that critical thinking is less an “internal trait” than a procedural activity manifested in concrete operations on information.

​

At the same time, the mere presence of doubt without subsequent steps to verify it does not serve as a marker of critical judgment. Such “untransformed uncertainty” reflects cognitive instability rather than cognitive complexity. This leads to an important conclusion: developing critical thinking in digital environments is not about cultivating more doubt, but about expanding the repertoire of control, verification, and analytical operations.

​

Accordingly, pedagogical interventions should not focus on restricting AI use or diminishing its role in learning, but on managing informational uncertainty. Building skills in verification, alternative interpretation, cross-checks, and social fact-checking can convert the “AI risk” into a resource for strengthening students’ cognitive autonomy.

Guided by the identified behavioral profiles, instructional efforts for class 0 (Non-High CM) students should prioritize transforming intuitive doubts into consistent verification actions. Each instance of uncertainty about an AI response should culminate in a brief fact-check using an independent source – ideally, two; the habit of “double-checking” should become a required step in completing assignments.

For class 1 (High CM) students, the priority is to scale and stabilize already established practices. Formalize your verification procedures – write down fact-checking rules, source-selection criteria, and minimum evidentiary requirements – and apply them consistently across courses. Where feasible, turn individual checks into collaborative ones: organize brief reciprocal reviews in shared documents and record conclusions. For complex topics, raise the evidentiary bar – combine textbooks, peer-reviewed publications, and professional databases – and use AI as a counter-arguer explicitly tasked with probing weaknesses in your own reasoning. Institutionalizing these practices makes critical thinking not only an individual competence but also a collective norm within the learning community, thereby increasing error resilience and improving the quality of academic decisions.

​

Thanks for Your Attention!