P3656 - Chat GPT vs Traditional Search Engines: A New Era in Diverticulitis Information Retrieval

Mark Tawfik, DO¹, Angelica Rozenfeld, BA², Chloe Lahoud, MD¹, Sherif Andrawes, MD^1
1Staten Island University Hospital, Northwell Health, Staten Island, NY; ²Lake Erie College of Osteopathic Medicine, Staten Island, NY

Introduction: Artificial Intelligence has revolutionized the way people access and retrieve information. ChatGPT represents a significant advancement in natural language processing and has the potential to enhance patient health literacy. Diverticulitis, a common gastrointestinal disease, often drives patients to seek information about their condition online. This study investigates the accuracy and helpfulness of ChatGPT compared to traditional search engines for medical queries related to diverticulitis.

Methods: We used ChatGPT 3.5 and compared its performance with that of the widely used conventional search engine, Google. We identified the top six autocomplete suggestions from Google searches: diverticulitis symptoms, diverticulitis diet, diverticulitis treatment, diverticulitis pain, diverticulitis surgery, and diverticulitis causes. Responses from ChatGPT 3.5 and Google were evaluated for accuracy and helpfulness using a five-point Likert scale by three independent evaluators. Statistical analysis was performed using Python.

Results: Chat GPT received higher accuracy ratings of 4.83, 4.83, and 4.67, compared to traditional search engines with ratings of 4.16, 4.67, and 4, respectively. The corresponding p-values were 0.068, 0.528, and 0.052, indicating statistical significance. Additionally, Chat GPT received higher helpfulness ratings of 4.5, 4.67, and 4.5, compared to traditional search engines with ratings of 3.5, 3.5, and 3.67. The p-values for helpfulness were 0.046, 0.012, and 0.084, showing significance. Evaluator 2 found no significant difference in accuracy, but Evaluators 1 and 3 rated Chat GPT higher. All evaluators noted significant differences in Chat GPT's helpfulness.

Discussion: Our study highlights the nuanced distinctions between ChatGPT and traditional search engines in addressing diverticulitis-related queries. While both platforms offered accurate information, ChatGPT received a superior helpfulness score and demonstrated the ability to deliver tailored responses with minimal user input. However, additional research is required to validate its efficacy across diverse medical topics and ensure reliability.


Disclosures:


Mark Tawfik, DO¹, Angelica Rozenfeld, BA², Chloe Lahoud, MD¹, Sherif Andrawes, MD¹. P3656 - Chat GPT vs Traditional Search Engines: A New Era in Diverticulitis Information Retrieval, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.