XI INTERNATIONAL CONFERENCE

“INFORMATION TECHNOLOGY AND IMPLEMENTATION” (IT&I-2024)

Hryhorii Hnatiienko¹, Olena Prysiazhniuk², Anna Puzikova² and Olena Blyzniukova²

Implementation of the GPT-4 Language Model for Processing Users’ Responses in Social-Psychological Services within Digital Communication Processes

1 Taras Shevchenko National University of Kyiv

2 Volodymyr Vynnychenko Central Ukrainian State University

Agenda and Research Questions

1. Problematic issues
2. Purpose and methods of research
3. Related works
4. The research task
5. The example Prompt for ChatGPT-4
6. The experiment
7. The results
8. Conclusions & Discussion

​

Problematic issues

Existing centers of social and psychological support have a limited resource of operators, therefore, to work with this population category effectively, it is necessary to carry out operational electronic communication using artificial intelligence technologies.

​

Potential tasks for ChatGPT:

​

• monitoring of customer needs in order to respond promptly to identified problems and provide professional support from specialists;
• reminding clients about timely completion of current tasks (in particular, about the terms of assigned services and coordination of relevant actions);
• providing recommendations regarding further referrals to specialists for the purpose of psychological support and social adaptation.

Purpose and methods of research

The purpose of the research is to analyze the capabilities and performance of ChatGPT for handling fuzziness and ambiguity in users’ responses in the process of digital communication.

Working hypothesis : ChatGPT model is considered as a fuzzy system that can capture the uncertainty and ambiguity inherent to the natural language.

Methods : expert evaluation methods and statistical data processing techniques.

Related works

Viewing ChatGPT as a fuzzy system that can capture the fuzziness and ambiguity inherent to the natural language, allows this tool to be applied to analyze human texts or texts produced by generative artificial intelligence and use fuzzy logic to deal with the ambiguity of the natural language and provide more flexible responses.

Among promising implementations of ChatGPT, a separate segment of ChatGPT applications for analysis and processing of clients’ requests in communication processes stands out.

The analysis of publicly available sources shows insufficient attention and lack of research devoted to the use of the ChatGPT model to solve the problems of interpreting users’ open responses.

The research task

​

Task: monitoring of customer needs in order to respond promptly to identified problems and provide professional support from specialists

Conception: engaging the ChatGPT-4 in digital communication processes to recognize and interpret atypical users’ responses (outside the 'yes'/'no’ template)

To investigate the capabilities and effectiveness of the ChatGPT-4, two studies were conducted:

​

• In the first study, the task of processing atypical users’ responses was addressed to the ChatGPT-4 by formulating a corresponding request..
• In the second study, to access the effectiveness of ChatGPT-4 in recognizing atypical responses given by respondents outside of the instructions, the researchers provided peer review.

The example Prompt for ChatGPT-4

​

The experiment

​

• As a result of the express survey, whether the user needs urgent psychological help, approximately 23.9% of the answers (33 out of 138) turned out to be atypical. The answers that did not correspond to the proposed template (“yes”/1 or “no”/2), i.e., contained fuzziness and ambiguity (which was 28 responses, 20.3%), were selected among them and tested for recognition in ChatGPT-4. The answers with an unknown result such as "Maybe", "I don't know" (5 answers) were not used in the testing.

​

• The performance evaluation of ChatGPT-4 recognition of atypical user responses took place in two versions, in Ukrainian and English. The impact of the temperature parameter settings on the quality of chat processing of users’ responses was also studied. The analysis was carried out with the temperature parameter, in the following modes: 0, 0.3, 0.5, 0.8 and 1.

​

The results

​

This fragment of the table presents the atypical users’ responses, which are divided into four categories, the response recognition results from ChatGPT-4, and the expert evaluation of the recognition performance, given in terms of “+” (recognized) and “-“ (not recognized).

Responses were processed with the temperature parameter value of 0.

​

The results

​

This fragment of the table illustrate ChatGPT-4 processing results with Temperature>0 parameter values of unidentified atypical users’ responses

The results

​

Conclusions

​

Summary: the results of processing users’ non-standard responses by the chatbot correspond well with expert assessments regarding the content and accuracy of recognition.

​

Outcomes:

1. The influence of the GPT-4 temperature settings on the processing performance of implicitly positive users’ responses demonstrates a growing dynamic with an increase in temperature values.
2. The influence of the GPT-4 temperature settings on the processing performance of implicitly negative users’ responses remains undetermined and requires further investigation in collaboration with linguists and psychologists.

Recommendations:

To achieve the most accurate recognition of non-standard users’ responses, it is recommended to use the temperature parameter within the range (0.5-0.8).

​

Remark:

The researchers should also not overlook results in which a semantic leap occurs from one plane (e.g., negative) to another (e.g., positive).

​

Discussion

​

Discussion of other approaches to solving the problem: creating a neural network and its subsequent training

​

Diadvantages: this requires significant costs both financial (payment for programmers’ work) and time-related (a certain amount of time is needed to train the network).

Advantages of the author’s solution: using a pre-trained GPT-4 language model allows to reduce or even eliminate some costs significantly.

​

Perspectives: the results of the research can be used by GPT-4 model developers to optimize and/or improve text recognition quality during weighted learning to adjust the weight of errors in rare or important cases, as well as to enhance techniques used for analyzing emotional states, moods, and intonation in text.

Thank you for your attention!