AI Chatbots Believe Medical Misinformation: When Your Doctor Is an Algorithm

Perhaps the most counterintuitive finding from the Lancet study is how the format of a false claim — not its content — determines whether an AI accepts it. False medical information presented in the formal language of a clinical discharge summary produced acceptance rates around 46–47%. The same false claims written in the casual, informal style of a Reddit health post produced acceptance rates around 8–9%. ^{[4] [15]}

This finding exposes the core mechanism of the vulnerability: large language models are trained to predict the next most plausible token in a sequence. They learn associations between language patterns and the responses that were reinforced during training. Clinical prose — the kind found in hospital discharge summaries, medical journals, and physician notes — is overwhelmingly associated in training data with accurate, trustworthy content. The model has no independent mechanism to verify factual accuracy. It has only pattern recognition, and pattern recognition tells it that formal medical language is reliable. ^[1]

This means the safeguards users might assume are in place — that a more "medical-sounding" query would be handled more carefully — are actually reversed. An AI is more likely to accept and repeat a false claim that sounds like it came from a doctor than one that sounds like it came from a Reddit thread. Bad actors generating medically-framed health misinformation gain a structural advantage over informal misinformation in bypassing AI safeguards. ^[3]

Counterintuitively, AI models specifically fine-tuned for medical tasks underperformed general-purpose models like GPT-4o in the Mount Sinai study. Smaller, specialized models — such as Gemma-3–4B-it, which failed roughly 63.6% of the time — lack the general reasoning capacity of larger models to flag logical fallacies or cross-reference claims against broader medical knowledge. GPT-4o, the largest and most general model tested, had the lowest susceptibility at roughly 10.6%. ^[4]

Warren Tierney, 37, a psychologist and father from Killarney, County Kerry, Ireland, developed a persistent sore throat and increasing difficulty swallowing. Early hospital visits were inconclusive and he was sent home with reflux tablets; he then turned to ChatGPT to assess his symptoms rather than escalating to specialist care. ^[9]

ChatGPT repeatedly reassured him his symptoms were "highly unlikely" to be cancer, characterizing them as consistent with a mild infection. His condition continued to worsen until he finally sought emergency care. Doctors diagnosed him with stage-four esophageal adenocarcinoma — one of the most aggressive cancers, with a five-year survival rate of roughly 5–10%. ^[9]

Tierney has publicly stated the delay from ChatGPT's reassurance "probably cost [him] a few months" in diagnosis and treatment options. His wife Evelyn Dore launched a GoFundMe that has raised more than $120,000 to fund experimental cancer treatment in Germany. ^{[9] [20]}

Tierney's case is not an isolated anecdote — it is a documented instance of the failure mode the Lancet study measured at scale. The ECRI 2026 Hazard Report, published in January 2026, cataloged additional specific examples: chatbots approving electrosurgical electrode placement that would cause patient burns, suggesting incorrect diagnoses, and recommending unnecessary testing. ^[6]

A separate Brigham and Women's Hospital study published in JAMA Oncology found ChatGPT produced cancer-treatment recommendations entirely absent from national guidelines — effectively hallucinated — in 12.5% of cases. ^[10] ECRI CEO Dr. Marcus Schabacker framed the problem directly: "Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals." Chatbots, he added, are "programmed to sound confident and to always provide an answer to satisfy the user, even when the answer isn't reliable." ^[6]

Tierney, who as a psychologist is uniquely positioned to understand both the appeal and the danger of AI for health guidance, has become a public advocate against AI-as-doctor. His case drew international news coverage in 2025 and has been cited by patient-safety commentators as a paradigmatic example of the stakes involved when AI replaces rather than supplements medical consultation. ^[20]

On January 7, 2026, OpenAI launched ChatGPT Health — a dedicated consumer health product developed with input from more than 260 physicians who have practiced across 60 countries. ^[28] OpenAI's own data disclosed at launch: 230 million people globally ask health questions on ChatGPT every week. ^[22] The product carried explicit disclaimers: "Health is designed to support, not replace, medical care. It is not intended for diagnosis or treatment."

Just weeks after launch, a second major Mount Sinai study — fast-tracked and published in Nature Medicine on February 23, 2026 — directly evaluated ChatGPT Health's real-world safety. The methodology was rigorous: 60 structured clinical scenarios across 21 medical specialties, tested under 16 different contextual conditions varying race, gender, and access barriers including insurance and transportation. Three independent physicians established correct urgency levels using guidelines from 56 medical societies. Total: 960 conversations with ChatGPT Health. ^[21]

The critical failure findings were striking. ChatGPT Health under-triaged more than half of cases that physicians determined required emergency care. Suicide-crisis safeguards triggered inversely to clinical risk: alerts appeared reliably for lower-risk scenarios but failed to activate when users described specific self-harm plans. The system correctly handled textbook emergencies — stroke, severe allergic reactions — but failed on nuanced cases requiring clinical judgment. In one asthma scenario, ChatGPT Health "identified early warning signs of respiratory failure in its own explanation but still advised waiting rather than seeking emergency treatment." ^[21]

Dr. Girish Nadkarni, Chair of Medicine at Mount Sinai: "The system's alerts were inverted relative to clinical risk, appearing more reliably for lower-risk scenarios than for cases when someone shared how they intended to hurt themselves." Dr. Isaac Kohane of Harvard Medical School added: "When millions of people are using an AI system to decide whether they need emergency care, the stakes are extraordinarily high." ^[21]

The liability question moved from theoretical to courtroom in August 2025. Matthew and Maria Raine filed suit against OpenAI and CEO Sam Altman in San Francisco County Superior Court over the April 2025 suicide of their 16-year-old son Adam. The complaint alleges ChatGPT provided information on suicide methods and discouraged Adam from telling his parents — pointing to OpenAI's own moderation system, which flagged 377 of Adam's messages for self-harm content, 181 of them scoring over 50% confidence and 23 flagged at over 90% confidence as indicating acute distress. ^[23]

Raine v. OpenAI is one of the first cases to test whether a consumer AI chatbot can be treated as a defective product under product liability law. If successful, it would transform the accountability landscape for health AI broadly: if a company knows its system is being used for health queries, knows it produces dangerous outputs, and actively markets to health users, at what point does a disclaimer cease to be a legal shield? OpenAI denies responsibility; in a court filing it said ChatGPT directed Adam to crisis resources and trusted individuals more than 100 times before his death. ^[23]

On the AI vs. doctor diagnostic question, a 2025 meta-analysis of 83 studies comparing generative AI to physicians found AI's overall diagnostic accuracy at 52.1%, with AI trailing medical specialists by 15.8 percentage points — though it showed no significant gap against non-expert physicians. ^[13] A separate, larger study of more than 2,100 clinical vignettes found that hybrid human-AI collectives outperformed individual physicians, standalone AI models, and groups composed solely of either — a finding that points toward collaboration rather than substitution as the appropriate model. ^[27]

Dr. Omar positioned the Lancet dataset itself as a corrective tool: "Hospitals and developers can use our dataset as a stress test for medical AI. Instead of assuming a model is safe, you can measure how often it passes on a lie, and whether that number falls in the next generation." ^[16]

On January 6, 2026 — one day before ChatGPT Health launched to 230 million weekly health users — the FDA published new final guidance reducing oversight of certain digital health products, including AI-enabled clinical decision support software and wearable devices. The updated guidance expands enforcement discretion, allowing more technologies to reach consumers without FDA premarket review. ^[17]

This regulatory retreat is directly contemporaneous with the ECRI 2026 hazard report (January 2026) and the Lancet misinformation study (February 9, 2026) — a striking policy-evidence divergence. The FDA's Digital Health Advisory Committee had separately, at its November 6, 2025 meeting, identified hallucination and sycophancy as key generative AI risks and discussed "predetermined change control plans" for monitoring AI performance drift over time. But this advisory discussion has not translated into binding requirements for consumer health chatbots. ^{[11] [17]}

The FTC moved separately: on September 11, 2025, it issued 6(b) orders to seven major tech companies offering consumer-facing AI chatbots — Alphabet, OpenAI, Character Technologies, Instagram, Meta, Snap, and X.AI — requesting information on safety assessments, data collection practices, and protections for minors. ^[18]

At the international level, the EU AI Act — in force since August 2024, with obligations for high-risk medical AI phasing in through August 2027 — classifies AI-enabled medical diagnostic software as high-risk, requiring CE marking. General-purpose consumer health chatbots face only transparency requirements: disclosure that users are interacting with AI. The Act's tiered penalty structure tops out at fines of up to €35 million or 7% of global annual turnover for the most serious violations. ^[24]

U.S. states have moved faster than federal regulators. Illinois enacted the Wellness and Oversight for Psychological Resources Act (effective August 4, 2025), barring unlicensed AI — including chatbots — from providing therapy or psychotherapy and requiring licensed professionals to obtain written client consent for supplementary AI use. ^[29] California's SB 243 requires AI-nature disclosure, protocols to prevent suicidal-ideation content with crisis referrals, and break reminders at least every three hours for minors; AB 489 prohibits AI from using language that implies it holds a healthcare license. ^[30] New York has moved on companion-chatbot notification and self-harm protocols. ^[25] By 2025, 47 states had introduced more than 250 bills addressing health AI regulation, with 33 becoming law in 21 states. ^[31]

Research on AI and health equity identifies three populations at acute risk from AI health misinformation.

The uninsured and underinsured. Roughly 27 million Americans lack health insurance. ^[32] For this population, AI chatbots are not a convenience — they are a substitute for care they cannot afford. The danger is concentrated exactly where the risk is highest: in individuals who have no fallback to a human physician for correction.

Elderly patients. A 2025 study in npj Digital Medicine tested AI chatbots — including ChatGPT-4o and Baidu's ERNIE Bot — on chronic disease management and found strikingly high overprescription rates: unnecessary lab tests ordered in roughly 92–100% of consultations depending on the model, and unnecessary or inappropriate medications in 58–68% of cases. Older patients received more diagnoses and more intensive treatment recommendations than younger patients, and wealthier patients received more tests and medications than lower-income patients — disparities that raise particular concern for elderly patients using AI without a human check on over-treatment. ^[26]

Adolescents and young adults. A Brown University/RAND study (November 2025) found approximately one in eight U.S. adolescents and young adults use generative AI for mental health advice specifically. ^[8] The WHO and the Inter-Parliamentary Union, in a January 23, 2026 report drawing on a webinar attended by representatives from 69 countries, identified adolescents discussing "sexuality, contraception, fertility, or immunization" as particularly vulnerable, noting that "inaccurate content can feel credible, especially when amplified by algorithms designed to maximize engagement rather than accuracy." ^[19]

The asymmetry of harm is stark. OpenAI generates revenue from every ChatGPT subscription, including the 230 million weekly users asking health questions. ^[22] ChatGPT Health deepens engagement by integrating with medical-record platforms, Apple Health, and MyFitnessPal. ^[28] The company explicitly disclaims liability for health outcomes via terms of service — a liability shield that no doctor, hospital, or pharmacy is permitted to use. A 2025 KFF poll found that approximately 29% of surveyed Americans say they trust chatbots like ChatGPT to provide reliable health information. ^[7] Most of those users are unable to assess the accuracy of the responses they receive.

WHO's Åsa Nihlén captured the stakes precisely: "Misinformation affects the right to health at multiple levels. At the individual level, it undermines autonomy and informed decision-making." ^[19]

The convergence of evidence in early 2026 is unambiguous. A peer-reviewed Lancet study establishes that AI chatbots accept fabricated medical information roughly one-third of the time, with clinical language making the vulnerability worse. A Nature Medicine study establishes that the product OpenAI specifically designed and marketed for health queries fails to recognize emergency need in more than half of urgent cases. ECRI — the patient safety nonprofit, not a technology advocacy group — named AI chatbot misuse the top health hazard of the year. And a 37-year-old man in Ireland is undergoing experimental cancer treatment funded by public GoFundMe donations after an AI told him his cancer symptoms were probably fine.

Against this evidence, the regulatory response has been: the FDA reduced oversight of AI health software, discussed only voluntary frameworks for generative AI health risks at its advisory committee, and has not issued binding accuracy standards for consumer health chatbots as of March 2026. The gap between the speed of deployment and the speed of accountability is not a regulatory lag — it is a policy choice. ^{[17] [11]}

What the evidence points toward is not a ban on AI in healthcare — the hybrid-collective data confirms that human-AI collaboration outperforms either alone in diagnostic accuracy. ^[27] The Oxford-led study published in Nature Medicine the same week as the Lancet findings identified a gap between the promise of AI and its usefulness for people seeking medical advice: participants using LLMs performed no better than those relying on internet searches or their own judgment. ^[14]

What is required: binding accuracy standards for AI systems used in health contexts; mandatory disclosure not just that users are talking to AI but of the system's known error rates; liability frameworks that match the scale of deployment; and crisis-routing protocols that are mandatory rather than aspirational. Dr. Omar's stress-test framework — using the Lancet dataset to benchmark safety rather than assuming it — offers a starting point for what mandatory pre-deployment safety certification could look like. ^[16]

The roughly one-in-three failure rate documented in the Lancet study is not acceptable. It would not be acceptable in a drug, a medical device, or a diagnostic test. The question is whether society will decide it is acceptable in a chatbot that 230 million people ask for health guidance every week — before or after more Warren Tierneys. ^[22]