November 20, 2026

4 min read

Key takeaways:

  • AI is quickly integrating into various areas of medicine, but it is not perfect.
  • So, keeping a human in the loop is a critical part of this integration, an expert said.

ORLANDO — Healthcare providers need to continue exercising judgment like it is a muscle, in order for AI to reach its full potential in medicine, according to a speaker here.

Harvey Castro, MD, MBA, an emergency medicine physician and AI in healthcare speaker, was part of a panel discussion on what happens when a clinician disagrees with AI at this year’s AIMed conference.



PC1126Castro_Graphic_01_WEB



Healio spoke with Castro to learn more about how AI is developing in medicine, what healthcare providers should consider during its use and more.

Healio: Why is this an important conversation right now?

Castro: For most of my career in the emergency department, the hard calls were made by me, my team and the patient. Now there is another voice in the room. AI tools are moving from offering suggestions to taking actions inside hospital systems, and every clinician is going to face a moment when the machine says one thing and their judgment says another.

At AIMed26 in Orlando, I opened the session with a composite emergency department case and then chaired the panel “What happens when AI and the clinician disagree?” The question I kept asking was this: When a system keeps going after a doctor would stop, who protects the patient? We need that answer now, before these tools are everywhere.

Healio: What steps should clinicians take when using AI to ensure it will be as accurate as possible?

Castro: Treat AI like a smart colleague who has never met your patient. Its note can be useful, and it can be wrong. Before acting on it, check it against the person in front of you, the history, the exam and the chart.

AI saves real time when it drafts, summarizes or pulls information together. The minutes you should never give back are the ones spent checking anything that would change a diagnosis, a medication or whether a patient goes home. If you can’t explain why the recommendation fits this patient, keep thinking. That check is part of the care, and it is what keeps the patient safe.

Healio: What should be done when a clinician disagrees with AI’s recommendations?

Castro: The clinician owns the decision, and the patient is counting on that. When I disagree with a recommendation, I don’t follow it because the screen sounds confident. I write down what the tool suggested, why I disagreed and what I did instead.

Hospitals should welcome that disagreement. If a system only measures whether doctors followed the model, people will stop speaking up, and patients lose one of their best safeguards. My rule is human plus AI. The physician keeps the moral compass, while the machine finds patterns and analyzes them at scale.

Healio: Does that response change if the AI is an integrated part of the clinician’s healthcare system? Should inaccuracies be reported somewhere?

Castro: Yes. An integrated tool is part of how the hospital delivers care, so its mistakes belong in the same safety process as any other clinical system. Report them to the clinical informatics team or chief medical informatics officer, to quality and patient safety, and to the vendor, with enough detail that someone can find the cause and fix it.

Every report helps the next patient. If a hospital has no clear place to report a bad recommendation, that is a governance gap its leaders need to close.

Healio: What are your thoughts on the potential of deskilling here?

Castro: I worry about it, especially for young doctors and nurses who will train alongside a model that always sounds sure of itself. If we only teach people how to use a tool and never how to question it, we will raise clinicians who miss a confident error.

Medical education should keep the exam, the differential and the bedside at the center. I like one habit in particular: Before you look at the AI output, ask yourself what you would have done. Senior clinicians can help by thinking out loud when they disagree with a model, and by admitting it when the model turns out to be right. Judgment is a muscle, and our patients need us to keep it strong.

Healio: Patients are increasingly presenting with information from their own AI systems . How should clinicians handle that situation?

Castro: With respect and curiosity. A patient who brings in an AI answer is trying to take part in their own care, and that is a good thing. Thank them, look at it together, and then go back to their story, their symptoms and your exam.

If the information is wrong or unsafe, explain why in plain language and walk them through what you recommend instead. Patients don’t need us to win the argument. They need help making a safe decision, and they need to leave feeling heard.

Healio: What is the take-home message?

Castro: AI should make care more human.

At its best, technology gives clinicians more time to listen, to explain and to comfort. When a model and a clinician disagree, the right question is what helps this patient, and who is accountable if we get it wrong.

Healio: Is there anything else you would like to add?

Castro: I am optimistic. Used well, AI can help a tired clinician catch something at 3 a.m., bring expert knowledge to communities that have little access to specialists, and give families clearer answers. Getting there takes humility. The tools need to show their reasoning, and we need to keep our judgment sharp.

Editor’s note: These comments are educational and reflect the AIMed26 discussion. They are not medical advice for any individual patient or an endorsement of any product.

For more information:

Harvey Castro, MD, MBA, can be reached at https://www.harveycastromd.com and on X, at @HarveyCastroMD.



Source link