Explainable AI in Healthcare: A Clinical Use Case for Governing Agentic AI in Medical Settings

Key Takeaways

• Explainable AI in healthcare has three requirements. The clinician at the point of care needs case-level reasons, confidence, and limits within seconds. The regulator needs system-level evidence of subgroup performance, change control, and logs. The patient needs a plain-language reason and a way to appeal.
• Agentic AI in healthcare now acts. Utah lets AI renew prescriptions, and Nolla Health’s pilot lets AI issue initial prescriptions for topical acne treatments under phased physician oversight.
• Four workflows may carry specific failure modes: automation bias in diagnostic support, biased inputs in clinical triage, algorithmic denials in prior authorization, and batch-speed denials in claims review.
• A plausible explanation can increase overtrust. In a 2023 JAMA study, explanations did not protect clinicians from a biased model that cut their diagnostic accuracy by 11.3 percentage points.
• The rules ask for overlapping evidence. The EU AI Act, EU MDR, FDA’s CDS and PCCP guidance, HIPAA, CMS payer rules, and ISO/IEC 42001 can be served by one governance program.
• The governance pattern has five steps: inventory, pre-deployment evaluation across patient subgroups, runtime guardrails on data and actions, full traceability, and continuous drift monitoring.

Utah now lets AI write prescriptions. In January 2026, it became the first state to let an AI system take part in prescribing decisions, when its Office of Artificial Intelligence Policy signed a regulatory mitigation agreement with Doctronic for autonomous renewals of chronic-condition prescriptions. On October 5, 2026, MobiHealthNews reported that Nolla Health had launched a Utah pilot in which its AI goes a step further and issues initial prescriptions for topical acne treatments.

The design of Nolla’s pilot reads like a governance specification. The AI chooses only from predefined, physician-approved topical protocols, and oral medications and isotretinoin are excluded. When it cannot reliably choose a treatment, it routes the patient to a licensed physician. Two Utah-licensed physicians review prescriptions for at least the first 100 patients, and oversight then steps down in stages to a monthly retrospective review of at least 10% of enrollment volume. As Nolla’s chief medical adviser put it, “The AI can’t improvise.”

Not all clinicians are comfortable with the pace.

In April 2026, the Utah Medical Licensing Board called for the Doctronic pilot to be suspended, saying it had not been consulted before launch.

That friction is where AI in healthcare now sits. Agentic AI in healthcare is moving from recommendation to action: ordering tests, drafting documentation, routing patients, and now prescribing. Once a system acts, each action needs an explanation that someone can check. Explainable AI in healthcare answers to three audiences at once: the clinician at the point of care, the regulator reviewing a submission, and the patient asking why. Each needs a different explanation, at a different depth, delivered on a different surface.

This use case is written for chief medical information officers, clinical AI governance committees, hospital compliance and privacy officers, and payer risk leaders. It covers four named workflows (diagnostic support, clinical triage, prior authorization, and claims review), maps them to the EU AI Act, EU MDR, FDA guidance, HIPAA, and ISO standards, and closes with evaluation criteria and a governance pattern you can apply to agentic deployments.

Why Agentic AI and Deep Learning in Healthcare Raise the Bar for Explainability

Agentic AI and deep learning in healthcare carry explainability requirements that most other sectors never face. Five features of clinical work explain the difference.

1. Errors land on patients, and many cannot be reversed. A wrongly declined loan can be appealed. A missed sepsis case cannot. When University of Michigan researchers externally validated the Epic Sepsis Model across 38,455 hospitalizations, they measured an AUC of 0.63 against vendor-reported figures of 0.76 to 0.83. The model missed 67% of patients who developed sepsis while alerting on 18% of all hospitalized patients.
2. Explainability decides whether software counts as a medical device. In the US, clinical decision support (CDS) software stays outside device regulation only if a healthcare professional can independently review the basis for its recommendations. FDA kept that test, known as Criterion 4, when it revised its CDS guidance on January 6, 2026. Few other industries draw a regulatory boundary along explainability.
3. The person in the loop holds a license and a duty of care. The clinician who accepts a recommendation owns the outcome. An explanation has to give that clinician enough evidence to overrule the model when it is wrong.
4. Deep learning explanations are approximations. A saliency map on a chest X-ray or an attribution score on a free-text note estimates what the model weighed, which is different from showing it. Ghassemi, Oakden-Rayner, and Beam argued in The Lancet Digital Health that current explainability methods fail too often to justify decisions about individual patients, and called for rigorous internal and external validation instead.
5. Averages hide subgroup harm. A care-management algorithm studied by Obermeyer and colleagues predicted cost as a proxy for need. At the same risk score, Black patients were sicker, with 26% more chronic illnesses at the 97th percentile. Black patients made up 17.7% of those automatically flagged for extra care. Without the bias, the researchers calculated, they would have made up 46.5%. An aggregate accuracy score would never have surfaced that gap.

The last point matters most for governance. Explainable AI in healthcare has to work at the level of patient subgroups, and it has to keep working after deployment as populations, coding practices, and care pathways shift.

Explainability Has Three Sides: Clinician, Regulator, Patient

Explainable AI in healthcare serves three audiences who ask different questions on different timelines. One explanation cannot serve all three.

	Clinician at the point of care	Regulator reviewing a submission	Patient asking why
Their question	Should I act on this, for this patient, now?	Is this system safe and effective, for whom, and will it stay that way?	Why did this happen to me, and what can I do about it?
Depth	Case level: the few factors behind this output, confidence, missing data, known weak spots for patients like this one	System level: intended use, data provenance, subgroup performance, validation, change control, logs	Plain language: the main reasons, the role AI played, who decided, how to appeal or reach a human
Surface	Inside the EHR or worklist, at the moment of decision	Technical documentation, marketing submission, audit trail	Portal message, denial letter, app, or a conversation with the care team
Time available	Seconds	Weeks to months	Minutes, often under stress
Requirements it answers	FDA CDS Criterion 4; EU AI Act Articles 13 and 14	EU AI Act technical documentation; EU MDR; FDA submissions and PCCPs; ISO/IEC 42001 audits	EU AI Act Articles 50 and 86; CMS prior authorization denial reasons

The Clinician 

Needs an explanation that supports judgment without replacing it. Article 14 of the EU AI Act requires that the people overseeing a high-risk system can interpret its output correctly, stay aware of “the possible tendency of automatically relying or over-relying on the output,” and choose to disregard or override it. That calls for explanations that show uncertainty and limits as prominently as reasons. A clinician who sees “low confidence: this model underperforms for patients over 80” makes a better call than one who sees a confident heatmap.

The Regulator 

Reads at system level. Article 13 requires instructions for use that describe a high-risk system’s “performance regarding specific persons or groups of persons” and the technical measures that help deployers interpret its outputs. Reviewers want evidence: who the model was tested on, how it performed in each subgroup, what has changed since clearance, and a log that ties each output to a model version and its inputs.

The Patient 

Gets the shortest explanation and has a strong legal claim to it. Under Article 86, a person affected by a decision based on an Annex III high-risk system can request “clear and meaningful explanations of the role of the AI system in the decision-making procedure” and the main elements of the decision. In the US, the CMS Interoperability and Prior Authorization Final Rule (CMS-0057-F) requires Medicare Advantage, Medicaid, CHIP, and federal exchange plans to give a specific reason when they deny a prior authorization request. A feature-attribution plot meets neither standard.

A common failure is to build one artifact, usually a feature-attribution chart designed by data scientists, and hand it to all three audiences. A governed program produces three outputs from the same evidence: a trace of what the model or agent saw, what it did, and which policy allowed it.

Four Workflows Where AI Solutions in Healthcare Now Act

AI solutions in healthcare are taking on actions in four workflows: diagnostic support, clinical triage, prior authorization, and claims review. Each has a documented failure mode, and each needs all three explanations.

1. Diagnostic Support

What the agent does: It pre-reads imaging, lab panels, or ECGs, drafts preliminary findings, and queues follow-up orders, such as a repeat CT for an incidental lung nodule. Ambient documentation agents turn the encounter into a draft note.

Failure mode – automation bias: In a study published in Radiology, 27 radiologists read 50 mammograms with AI-suggested BI-RADS categories. Very experienced readers (more than 15 years on average) assigned the correct category 82% of the time when the AI was right and 45.5% of the time when it suggested an incorrect category. Inexperienced readers fell below 20%. Documentation agents add a second failure mode: transcription hallucination. Researchers studying OpenAI’s Whisper found roughly 1% of transcriptions contained hallucinated phrases, and 38% of those hallucinations included explicit harms.

The three explanations:

• Clinician: the image regions or values behind the finding, the model’s confidence, and whether this patient belongs to a subgroup where the model is weaker.
• Regulator: the validated intended use, reader-study results, and performance across sites, scanner vendors, and patient demographics.
• Patient: that AI assisted the reading, that a radiologist made the call, and whom to ask about it.

2. Clinical Triage

What the agent does: It scores emergency department arrivals, sorts patient portal messages by urgency, and routes patients to self-care advice, a virtual visit, urgent care or the emergency department, sometimes booking the slot itself.

Failure mode – biased inputs and undertriage: Triage models inherit the bias of their inputs. Sjoding and colleagues reported in NEJM that among patients whose pulse oximeters read 92% to 96%, arterial blood gas tests revealed occult hypoxemia in 11.7% of Black patients and 3.6% of white patients. A triage agent that trusts the oximeter will undertriage the patients the device misreads. Agents that read free-text messages carry a second risk, because instructions hidden in the content they process can redirect them. The EU AI Act names “emergency healthcare patient triage systems” as high-risk in Annex III, point 5(d).

The three explanations:

• Clinician: the acuity drivers, any input flagged as unreliable, and the route the agent chose along with the alternatives it rejected.
• Regulator: undertriage and overtriage rates by age, sex, race, ethnicity, and preferred language, plus evidence of the human oversight measures Article 14 requires.
• Patient: why they were sent where they were, and a direct path to a person if they disagree.

3. Prior Authorization

What the agent does: On the provider side, agents assemble clinical documentation and submit requests. On the payer side, agents check requests against coverage criteria, auto-approve clear cases, and route the rest to clinical reviewers.

Failure mode – algorithmic denial without individual review: A Senate Permanent Subcommittee on Investigations report found that UnitedHealthcare’s prior authorization denial rate for post-acute care rose from 10.9% in 2020 to 22.7% in 2022 as insurers automated their processes. In Estate of Lokken v. UnitedHealth Group, plaintiffs allege that the nH Predict tool supplanted physicians’ judgment on post-acute care. UnitedHealthcare calls it “a care-support tool.” In March 2026, a federal magistrate judge ordered broad discovery, including internal AI review board materials.

The rules are tightening. CMS has told Medicare Advantage plans that coverage decisions must rest on the individual patient’s circumstances and that an algorithm’s length-of-stay prediction “cannot be used as the predicate to terminate coverage.” California’s SB 1120 bars AI tools from denying, delaying or modifying care based on medical necessity, and the California Department of Insurance requires those tools to be open to inspection and reviewed periodically. CMS’s own WISeR model has used AI in Traditional Medicare prior authorization in six states since January 2026, and it survived a Senate repeal vote in July 2026.

The three explanations:

• Clinical reviewer: the coverage criterion applied, the documentation the agent matched against it, and what is missing.
• Regulator: decision logs showing that a licensed professional made each adverse medical-necessity determination, plus approval and denial rates by service and member subgroup.
• Patient and provider: the specific denial reason CMS-0057-F requires, and how to appeal.

4. Claims Review

What the agent does: It checks coding and documentation, flags payment integrity issues and patterns of fraud, waste, and abuse, and drafts denial or adjustment letters.

Failure mode – denial at batch speed: ProPublica reported that Cigna’s PXDX system let medical directors deny more than 300,000 claims over two months in 2022, spending an average of 1.2 seconds on each. A signature applied in 1.2 seconds, without opening the patient’s file, is not meaningful oversight.

The three explanations:

• Reviewer: the rule or pattern that flagged the claim, the supporting evidence, and comparable claims.
• Auditor or regulator: review time and override rate per reviewer, which exposes rubber-stamping.
• Patient: a plain-language reason and the appeal route.

What Are the Challenges of Implementing Agentic AI in Medical Settings?

The challenges of implementing agentic AI in medical settings come from the move from advice to action. Seven matter most to clinical AI governance committees.

1. Scoping what an agent may do. An agent with write access to the EHR can place orders, message patients, and reschedule procedures. A misbehaving or compromised agent can reach systems and data it was never meant to touch. Every tool needs explicit permission, limits on its parameters, and a rule for which actions require sign-off.
2. Keeping protected health information (PHI) out of the wrong places. PHI can end up in a model’s context window, a third-party API call, a prompt log, or a trace. Under HIPAA, each copy is a record you must safeguard, and the minimum necessary standard limits what agents should see for most non-treatment purposes.
3. Blocking content that hijacks the agent. Agents read referral faxes, scanned records, portal messages, and web pages. Instructions embedded in that content can redirect an agent that has the authority to act.
4. Explaining multi-step chains. Feature-attribution methods explain a single model output. An agent’s outcome comes from many LLM calls, tool calls, and retrieved documents, so explaining it requires a trace of every step, the data each step used, and the policy that allowed it.
5. Placing humans where they add value. Sign-off on every action recreates alert fatigue and turns review into rubber-stamping, as the 1.2-second claim reviews show. Sign-off on none removes oversight. Risk-tiered oversight sends routine, reversible actions through automated controls and reserves human review for high-risk or irreversible ones. Article 26 of the EU AI Act requires that the people assigned to oversight have “the necessary competence, training and authority.”
6. Controlling change. Foundation model vendors ship new versions, teams edit prompts, and retrieval sources change. Each can alter behavior that was validated. FDA’s predetermined change control plan (PCCP) framework lets device manufacturers pre-specify modifications. Health systems need equivalent version control and re-evaluation triggers for everything else.
7. Detecting drift after go-live. Performance shifts with case mix. When COVID-19 arrived, total daily alerts from a sepsis model rose 43% across 24 hospitals while patient census fell 35%. Drift often hits subgroups first, where aggregate dashboards miss it. Article 26 also requires deployers of high-risk systems to monitor operation, suspend use when they identify a risk, and keep automatically generated logs for at least six months.

Mapping Explainable AI in Healthcare to the Rules That Apply

The frameworks below overlap, and most ask for the same underlying evidence: what the system was designed to do, how it performs for each patient group, who oversees it, and a record of what it did.

Framework	Where it applies in these workflows	What it asks for on explainability and oversight	Main audience
EU AI Act, Annex III	Point 5(a): public bodies deciding eligibility for healthcare services. 5(c): risk assessment and pricing in health insurance. 5(d): emergency patient triage	Instructions for use with subgroup performance (Art. 13); human oversight that guards against automation bias (Art. 14); deployer monitoring and log retention of at least six months (Art. 26); AI disclosure (Art. 50); a right to explanation (Art. 86)	Regulator, clinician, patient
EU AI Act Art. 6(1) with EU MDR	AI medical devices that need notified body review, which under MDR Rule 11 covers most diagnostic and therapeutic decision software (class IIa and above)	High-risk AI Act duties added to MDR technical documentation and quality systems. MDCG 2025-6 asks manufacturers to define who can override or stop the system and how automation bias is controlled	Regulator, clinician
FDA CDS guidance (January 2026)	Clinician-facing decision support in the US	Software stays non-device only if the clinician can independently review the basis for each recommendation (Criterion 4); time-critical tools do not qualify	Clinician
FDA PCCP final guidance (December 2024) and AI-enabled device draft guidance (January 2025)	AI-enabled medical devices in the US	Pre-specified model changes with a validation protocol; performance across subgroups such as sex, age, race and ethnicity; transparency in labeling, with an optional model card; post-market monitoring	Regulator
HIPAA Privacy and Security Rules	Any workflow that touches PHI, including prompts, traces, and explanation logs	Minimum necessary access, access and audit controls, and business associate agreements with AI vendors that handle PHI	Regulator, patient
CMS payer rules and state law	Medicare Advantage, Medicaid, CHIP and exchange plans; California-regulated insurers	Coverage decisions based on individual circumstances; specific denial reasons; public prior authorization metrics; licensed professionals making medical-necessity calls (SB 1120)	Patient, regulator
ISO standards	Any organization governing clinical AI	ISO/IEC 42001 for a certifiable AI management system; ISO/IEC 23894 for AI risk management; ISO/IEC TS 6254:2025 for matching explainability methods to each stakeholder’s objectives; ISO 14971 for device risk management	All three

ISO/IEC TS 6254 deserves a closer look from governance committees. It frames explainability as a set of objectives that differ by stakeholder, which is the three-audience model in standard form.

The Explainability Trap: Plausible Explanations Can Increase Overtrust

In clinical AI, a plausible explanation can raise trust in a wrong answer. The evidence is specific.

• Explanations did not protect clinicians from a biased model. In a JAMA study published in December 2023, 457 hospitalist physicians, nurse practitioners, and physician assistants diagnosed acute respiratory failure with baseline accuracy of about 73%. An accurate model raised accuracy by 2.9 percentage points, or 4.4 points with explanations. A systematically biased model cut accuracy by 11.3 points, and image-based explanations did not help clinicians recover.
• Explanations raised acceptance whether the AI was right or wrong. Bansal and colleagues found that explanations “increased the chance that humans will accept the AI’s recommendation, regardless of its correctness.”
• Language model rationales can be unfaithful. Turpin and colleagues showed that hidden biases in a prompt can steer a model’s answer while its step-by-step explanation never mentions them, and that models write convincing justifications for wrong answers. For an agent drafting a prior authorization letter or a discharge summary, the stated rationale may not match what drove the action.

Regulators have named the problem. Article 14(4)(b) of the EU AI Act requires that overseers “remain aware of the possible tendency of automatically relying or over-relying on the output,” and MDCG 2025-6 asks device manufacturers to state how automation bias is controlled.

Five design choices reduce the risk.

1. Lead with confidence and known limits. Put uncertainty, missing inputs, and subgroup caveats above the reasons, where the clinician sees them first.
2. Test explanation faithfulness before deployment. Check that the features an explanation cites change the output when perturbed, and that an LLM’s stated rationale matches its behavior across varied inputs.
3. Use cognitive forcing for high-stakes, irreversible decisions. Asking clinicians for their own assessment before showing the AI output reduced overreliance compared with simple explainable AI designs. Participants liked those designs least, so reserve them for decisions where the cost of error is highest.
4. Measure reliance in production. Track agreement rate, override rate, and review time per reviewer and per patient subgroup. Agreement approaching 100% alongside falling review times is a warning sign.
5. Seed known errors. Route a small number of cases with known-wrong AI outputs to reviewers and measure how many they catch. This red-teams the human and AI together, which is the unit that makes the decision.

Evaluation Criteria for Clinical AI Explainability

Use these eleven criteria when a clinical AI governance committee reviews a model, an agent or a vendor. Each one names the evidence that answers it, so the review produces documentation as well as a decision.

Audience	Criterion	Question to ask	Evidence to request
Clinician	Decision relevance	Does the explanation show the few factors behind this output, in clinical terms, inside the workflow?	Usability testing with the intended users in the EHR or worklist
Clinician	Uncertainty and limits	Does each output show confidence, missing inputs, and whether the patient belongs to a weaker subgroup?	Calibration by subgroup; out-of-distribution flags
Clinician	Faithfulness	Do the cited factors drive the output?	Perturbation tests; rationale-consistency tests for LLM outputs
Clinician	Reliance impact	Does the explanation improve accuracy when the AI is wrong, as well as when it is right?	A reader study that includes deliberately incorrect AI outputs
Regulator	Subgroup performance	How do sensitivity, specificity and calibration vary by age, sex, race, ethnicity, language, payer and site?	Pre-deployment fairness and bias evaluation report
Regulator	Traceability	Can every output or action be tied to the model version, prompt version, inputs, policy decision, and human reviewer?	A sample audit trail for one real case
Regulator	Change control	What triggers re-evaluation, and what must pass before a change ships?	PCCP or internal change protocol; regression evaluation suite
Patient	Plain-language reason	Can the system state the specific reason, the role AI played, and the appeal route in language a patient understands?	Sample notices with a readability check

A system that fails the reliance-impact or subgroup-performance rows is not ready for a clinical pilot, however good its explanations look.

A Governance Pattern for Agentic AI in Healthcare

The Nolla pilot already shows the shape of good agentic governance: a fixed list of permitted actions, escalation to a physician when the AI is unsure, and human review that steps down only as evidence builds. The pattern below generalizes that design to any clinical or payer workflow. Every agent action is controlled before it happens, recorded as it happens, and explainable long after.

1. Inventory and classify. Register every model and agent with a purpose, an owner, and a risk tier mapped to the rules above: Annex III point, MDR class, CDS status, CMS scope. Lumenova AI’s AI inventory and agentic AI registry give each system a named owner, and shadow AI discovery holds any agent seen in traffic but missing from the registry until someone reviews it.
2. Evaluate before deployment. Lumenova AI runs pre-deployment evaluations with deep-dive fairness, bias, explainability, and robustness assessments across patient subgroups. More than 200 built-in quantitative and qualitative metrics are translated into plain-language clinical and business impact, so a CMIO and a compliance officer can read the same result. Evaluations combine deterministic checks, semantic quality judgment, full review of an agent’s reasoning and tool use, and adversarial red teaming. Clinician annotations become reusable evaluation data instead of notes lost in spreadsheets.
3. Constrain action at runtime. Every LLM call, MCP tool call, and agent hop passes through one enforcement point. Automated guardrails detect and mask restricted patient data before it is submitted to an AI system, and flag prompt injection hidden in the content an agent reads. Policy as code decides which tools an agent may call and with which parameters before the call is made. Routine, reversible actions proceed automatically, and high-risk actions trigger human-in-the-loop review. Because the platform can be self-hosted inside your own perimeter, prompts and completions never leave your network.
4. Record every decision. Each request becomes a trace, broken into the steps the agent took. Each enforcement decision is stored with the policy that matched, the outcome, the reason, and the full policy input, with parameters redacted according to policy. This record is the regulator’s evidence, and it is also where a plain-language answer to a patient’s “why?” comes from.
5. Observe and re-evaluate. Continuous observability tracks performance by patient subgroup, detects drift after deployment, and alerts owners when a system crosses a threshold. Evaluators keep running on live traffic, and findings feed back into step 2.

Each step produces evidence for a specific audience. Steps 2 and 3 give clinicians validated explanations with confidence and limits at the point of care. Steps 1, 2, and 4 give regulators and auditors system-level documentation. Step 4 gives patients a specific, checkable reason. One evidence base serves all three audiences.

Govern Clinical AI Before It Acts

Explainable AI in healthcare works when each audience gets the explanation it needs: clinicians get reasons and limits they can act on, regulators get evidence that holds up in review, and patients get a reason they can understand and appeal. Getting all three right lets you scale clinical AI safely, protect patient trust as agents take on more of the care pathway, and reduce the legal exposure that comes from decisions no one can explain. 

Agentic AI in healthcare is already renewing and issuing prescriptions in Utah, routing patients and drafting coverage decisions. If your organization is moving clinical or payer AI from recommendation to action, start by measuring where your governance stands today. 

• Take the Agentic AI Risk & Governance Assessment to benchmark your controls, oversight, and traceability for agentic deployments.
• Book a discovery call with Lumenova AI to see how pre-deployment evaluations, guardrails, and continuous observability apply to your diagnostic, triage, prior authorization, and claims workflows.

Frequently Asked Questions About Explainable AI in Healthcare

---

Related topics: Explainable AIHealth & Life Sciences