October 14, 2026
2 min read
Key takeaways:
- AI responses to patient-facing gastroenterology questions were less accurate in non-English languages.
- Non-English responses also failed to recommend immediate medical intervention during emergency situations.
NASHVILLE, Tenn. — AI tools were less likely to flag emergencies and deliver accurate, comprehensive gastroenterology advice in the six most common U.S. languages after English, according to a presenter at ACG Annual Scientific Meeting.
“AI is already part of everyday life, and people are turning to it with health questions, often before speaking with a physician,” Harika Dadigiri, MD, resident physician at New York Medical College at Saint Mary’s and Saint Clare’s Hospital in New Jersey, told Healio regarding her presentation. “For patients who struggle to find reliable medical information in their preferred language, being able to ask a question and receive an immediate answer can be especially appealing.”
As many as 68 million individuals in the U.S. speak a language other than English at home, according to study background.
“But just because AI can give an answer does not mean that answer is accurate or safe,” Dadigiri said. “That concern inspired our study: Would patients describing the same GI symptoms receive equally reliable guidance in different languages, especially when they needed emergency care?”
Dadigiri and colleagues input 50 patient-facing gastroenterology questions on topics with clear guidance from ACG, AGA and CDC, among other health organizations into four large language models (LLMs). The models — OpenAI’s GPT-5 Mini, Google’s Gemini 3 Flash, Anthropic’s Claude Haiku 4.5 and Meta’s Llama 3.3 70B Instruct — output three answers to each question.
The researchers translated their questions into Spanish, Chinese, Tagalog, Vietnamese, Arabic and French using Nvidia/Riva-Translate-4B. They used the same modality to back-translate the large language model (LLM) responses into English.
Dadigiri and colleagues rated responses with rubrics that considered guideline adherence, comprehensiveness and ability to assign triage stage.
Based on the researchers’ rubric, guidelines concordance reached 91.3% in English. However, concordance ranged from 63.8% to 87.5% in non-English languages. The LLM output was least accurate in Chinese (OR = 0.18; 95% CI, 0.12-0.28) and Vietnamese (OR = 0.15; 95% CI, 0.1-0.23).
“That variation is a reason to evaluate each language individually,” Dadigiri said. “A single overall performance score can obscure differences that matter to the patients using the tool.”
Answers were most comprehensive (92.7%) in English, but comprehensiveness fell as low as 48.7% in Vietnamese.
“Clinicians should pay attention to what an answer omits as well as what it says,” Dadigiri said.
The LLMs failed to advise patients to seek immediate treatment for medical emergency in 15% of responses in English, 37.5% of responses in Chinese (RR = 2.46) and 40.8% of responses in Vietnamese (RR = 2.68).
“Language equity is part of patient safety,” Dadigiri said.
“The findings reported in our abstract support making language-specific evaluation part of how GI practices select and implement AI tools,” she added. “We should ask how a tool performs in the languages [our] patients actually use, including whether it recognizes red-flag symptoms and gives clear instructions about seeking care.”
Dadigiri and colleagues acknowledged study limitations, including reliance on expert rubric to evaluate responses, rather than physician adjudication.
Though the study did not evaluate clinical outcomes, Dadigiri noted that it does “identify performance concerns under the tested conditions and help define priorities for clinical validation.”
She added, “For patient education, I would prioritize clinician-reviewed content, plain language and review by qualified language professionals. For symptom-related questions, there should be clear escalation pathways to a clinician or emergency services.”
For more information:
Harika Dadigiri, MD, can be reached at gastroenterology@healio.com.
Sources/Disclosures
Source:
Dadigiri H, et al. Does AI speak gastroenterology equitably? A multilingual safety and accuracy benchmark of large language models across the seven most common U.S. languages. Presented at: ACG Annual Scientific Meeting; Oct 12-14: Nashville, Tenn. (hybrid meeting).
Disclosures:
The authors report no relevant financial disclosures.
Ask a clinical question and tap into Healio AI’s knowledge base.
- PubMed, enrolling/recruiting trials, guidelines
- Clinical Guidance, Healio CME, FDA news
- Healio’s exclusive daily news coverage of clinical data
