Document Type
Original Research
Disciplines
Higher Education | Medicine and Health Sciences
Abstract
Background. Use of language-learning model-based artificial intelligence (AI) chatbots in literature reviews has not been explored. The present study seeks to evaluate and compare the quality of reference lists and scientific discussion provided by ChatGPT (Chat Generative Pre-Trained Transformer) (4o vs. 4) and Google Gemini.
Methods. ChatGPT (4o-mini, paid vs. 4o, free) and Google Gemini were given prompts to identify, discuss results of, and provide related references to nine specific randomized controlled trials in urologic practice. Reference lists and associated discussion were graded using modified DISCERN criteria. Hallucination rates for unverifiable references were calculated; GPT-2 Output Detector and Sapling AI Detector were used to assess detection of AI-generated text. Mean ± SD scores were calculated and compared with independent-samples T-tests.
Results. The target trial was identified in 11%, 22%, and 22% of prompts by ChatGPT 4o-mini, Google Gemini, and ChatGPT 4o, respectively. Hallucinations (out of ten) were high for ChatGPT4o and ChatGPT 4 (7.7 and 6.3, respectively); in comparison, Google Gemini was significantly less likely to hallucinate (0.17, p < 0.005). Modified DISCERN scores were poor for all three chatbots (range: 26.6 to 29.7). AI output was detected in 60-72% of responses via GPT-2 and Sapling AI.
Conclusion. AI chatbots were unable to identify the target article in most instances. Only Google Gemini appropriately refused to provide related articles and/or answer open-ended prompts if the target article could not be identified. Detection of AI-generated text was high for all chatbots, and caution should be used when using these AI resources in scientific writing or literature review.
DOI
10.32873/unmc.dc.gmerj.8.2.003
Keywords
Artificial intelligence; large language model; medical education; urology
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Recommended Citation
Huynh, L. M., Helm, M., Staab, N., Fadel, A., Walsh, J., Glassman, J., , Deibert, C. M. Battle of the Bots: Comparison of Artificial Intelligence Chatbots in a Literature Review of Randomized Controlled Trials in Urology. Graduate Medical Education Research Journal. 2026 Oct 30; 8(2).
https://digitalcommons.unmc.edu/gmerj/vol8/iss2/3
Table 2.docx (16 kB)
Table 3.docx (17 kB)
Figure 1.jpg (1507 kB)
Supplemental Materials.docx (24 kB)
