ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
Simple Methodology
2
In this review of ChatGPT's capabilities for producing an initial bibliography on specific topics, a straightforward methodology was employed to analyze the results. The following steps were undertaken:
3
4
Steps
5
• Execution of 10 prompts consecutively within the same "conversation" to ensure consistency in using the same model (ChatGPT 3.5 [free] using the model available on May 23). These prompts can be found within each workbook (titled "Prompt 1", "Prompt 2", etc.)
6
• The responses generated by ChatGPT were examined, and 10 citations were extracted and transferred to Google Sheets. Each prompt's citations were organized in separate worksheets.
7
• Each citation was carefully scrutinized to determine its authenticity, classifying it as either "Yes" (real), "No" (critical errors), or "Mixed" (concerns but seemingly real).
8
• For citations identified as "fake" or "mixed," a secondary search was conducted to verify their validity. This involved directly accessing the referenced journal and cross-referencing the provided date, volume, issue, and pagination with the citation.
9
• The analysis solely focused on identifying whether the articles generated by ChatGPT were real, fake, or mixed. It did not assess the content or relevance of the articles to the given prompts.
10
11
By employing this methodology, a comprehensive evaluation of ChatGPT's citation accuracy was conducted, specifically focusing on the authenticity of the generated articles.
12
13
Future Research
14
In the future, we will employ a comparable methodology to adopt a more informal approach with ChatGPT, departing from a strictly "scholarly" perspective. Further investigation could be conducted in future research to analyze the actual citations in relation to their relevance to the given prompts.
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100