November 17, 2026
Explainable AI in the Agentic Era: Trace-Based Explainability, Context Attribution, and the Limits of Asking a Model Why

Contents
Key Takeaways
- Classical explainable AI methods such as SHAP and LIME assume a fixed feature vector mapped to one score. Agents have neither, so those old methods cannot explain an agent’s decision end to end.
- An agent’s computation is observable while it runs and lost if nobody records it. Agent explainability is therefore an instrumentation problem, with a ceiling set at design time by what you log.
- The execution trace is the primary explanation. Context attribution, counterfactual replay, and policy decision records answer the questions a trace alone cannot.
- An agent’s account of its own reasoning is a plausible narrative, not evidence of its causal path. In an audit file, traces and policy records outrank it.
- Real agentic decisions need compositional explanation: model-level methods for the models an agent calls, plus execution-level evidence of what it did with their outputs.
- EU AI Act Article 86, Regulation B, and Colorado’s SB 26-189 all ask for the main reasons behind an individual decision, which an agent can give only if its decision provenance was captured.
Why Classical Explainable AI Breaks on Agents
→ Ask a validator to explain a credit scorecard, and they reach for SHAP values, LIME, or a surrogate model. Those tools share one assumption: a fixed vector of input features mapped to a single output score. Attribution works in this scenario because there is a stable space to attribute over and one number to attribute to.
An AI agent meets neither condition. It takes unstructured input, retrieves context, calls tools, reads their results, and emits a sequence of actions. The inputs differ on every run, and the output is a chain of steps rather than a score.
→ Run SHAP on the language model inside an agent, and you get token-level attributions for one generation step. That says little about why the agent called the fraud model before the coverage engine, or why it drafted a denial. Explainable AI agents need a toolkit built around what the agent did, because the question an auditor asks has not changed: why did it do that, and what proves it?
For Agents, Explainable AI Starts as an Instrumentation Problem
For a predictive model, the computation is opaque, so any explanation is reconstructed after the fact. In that setting, explainability is an approximation problem, judged by how close SHAP or a counterfactual gets to what the model did.
An agent works the other way round. Every plan, tool call, parameter, result, and handoff passes through code you control, in plain view, while it happens. If nobody records it, it is discarded when the run ends.
That moves the work to design time. An agent’s explanation ceiling is set by what you chose to log, and no post-hoc method recovers a retrieval result or tool response that was never captured. When a complaint or an examiner arrives months later, the trace either exists or it does not.
The EU AI Act requires high-risk systems to log events automatically (Article 12) and deployers to keep those logs for at least six months (Article 26(6)). ISO/IEC 42001 control A.6.2.8 asks organizations to decide when event logs are recorded, at minimum while the system is in use. Our guide on what to instrument in an AI agent covers span design in detail.
Explainable AI for Agents: Six Layers of Evidence
The techniques that work for agents stack in layers. Each answers a different question and has a different claim to faithfulness.
| Layer | Question it answers | Faithfulness | Classical LLM counterpart |
| 1. Execution trace | What did the agent do, in what order, with what inputs and results? | Exact, if captured | None: a record replaces the approximation |
| 2. Context attribution | Which retrieved chunk or tool result supported which claim? | Strong with ablation, weaker with citations alone | Feature attribution, applied to context |
| 3. Counterfactual replay | Would the action change if one input changed? | Strong, measured by re-execution | Counterfactual explanations |
| 4. Guardrail and policy records | Which rule allowed, blocked, or escalated an action? | Exact, because the rule is deterministic | Rule documentation |
| 5. Self-explanation and reasoning traces | What does the model say it was doing? | Weak as evidence of the causal path | None |
| 6. Aggregate behavioural explanation | How does the agent behave across many sessions? | Descriptive | Global explanations, drift monitoring |
1. The Execution Trace Is the Explanation
A trace records each request as a tree of spans: the plan, each tool call with its parameters, each result, each model generation, and the final action. The trace is the computation itself, so trace-based explanation needs no approximation. Its only gaps are what you failed to record.
2. Context Attribution
A trace shows what entered the context window. This does not cover which piece of context drove which claim. Citation grounding and groundedness scoring check that a source agrees with a claim. Context ablation checks contribution: remove a chunk, regenerate, and see whether the claim survives.
Full leave-one-out ablation is expensive on long contexts. ContextCite, from MIT researchers, makes it cheaper by fitting a sparse linear surrogate over a small number of random context ablations.
3. Counterfactual Replay
The counterfactual idea transfers to agents more cleanly than any other classical method. A deployed scorecard often cannot be re-run on altered history, but an agent can be re-executed. Counterfactual replay re-runs the agent with one perturbation, such as a reworded prompt, a removed document, or a mocked tool response, and measures whether the action changes.
Language model outputs vary between runs, so replay each variant several times and compare action distributions. Pin the model version, prompt template, and tool mocks so the perturbation is the only change. The same reasoning underpins counterfactual explanations for predictive models.
4. Guardrail and Policy Decision Records
When a policy engine allows, blocks, or escalates a tool call, it applies a deterministic rule. A record of the matched policy, the decision, and the reason is perfectly faithful, because the rule is the cause. It explains the constraint even when the model behind the request stays opaque.
5. Self-Explanation and Reasoning Traces
Agents can narrate their reasoning, and reasoning models expose a chain of thought. Both help with debugging and drafting customer letters, but they are weak evidence of why the agent acted.
6. Aggregate Behavioral Explanation
Some questions concern a population of decisions rather than one. Action distributions show how often the agent escalates, denies, or calls each tool, by segment and over time. Topic clustering groups sessions by what users asked, so you can see which request types lead to adverse outcomes.
The Limits of Asking a Model Why
The most tempting explanation method is also the least reliable, which brings us to our first problem. Ask an agent why it denied a claim, and it will give a fluent, plausible answer. That answer is new output from the same model, and nothing guarantees it describes the causal path behind the decision.
Two studies on chain-of-thought faithfulness show the gap. In a NeurIPS 2023 study, Turpin et al. biased models by reordering few-shot examples so the right answer was always “(A)”. The models followed the bias, wrote reasoning that almost never mentioned it, and lost up to 36% accuracy across 13 tasks.
Anthropic’s 2025 study of reasoning models planted hints in prompts and checked whether the chain of thought admitted using them. The reveal rate was often below 20%. Unfaithful reasoning also tended to be longer than faithful reasoning, so length and detail are no signal of honesty.
The second problem sits with the human reviewer. In a CHI 2021 study, Bansal et al. found that AI explanations increased the chance people accepted a recommendation, whether or not it was correct. A convincing explanation can raise automation bias rather than reduce it, which defeats the purpose of human review.
NIST’s four principles of explainable AI treat explanation accuracy, meaning the explanation reflects the system’s actual process, as its own requirement. Self-reported reasoning fails that test too often to stand alone, so trace and policy records outrank it in an audit file. Keep the model’s narrative as a labelled draft and check each claim against the trace.
Compositional Explanation: A Claims-Handling Agent, Step by Step
A real agentic decision is explainable only when a model-level explanation composes with an execution-level one. Take a claims-handling agent processing a water-damage claim. It retrieves the policy wording, calls a fraud score, calls a coverage-rules engine, and drafts a denial.
| Step | Component | What must be explained | Method | Evidence in the file |
| 1 | Policy retrieval | Which clause the denial rests on | Context attribution with ablation | Retrieved chunks with document versions, attribution scores, the clause cited |
| 2 | Fraud score | Why the score came back high | Classical counterfactual or feature attribution on the fraud model | Feature values, score, threshold, the smallest change that would clear it |
| 3 | Coverage-rules engine | Why the loss falls outside cover | Deterministic rule record | Rule ID, inputs, outcome |
| 4 | Denial draft | Whether the letter matches what drove the decision | Counterfactual replay against the trace | Replay results, final letter, reviewer sign-off |
The answer the policyholder is owed spans steps 1 to 3. Suppose the letter cites an exclusion for gradual seepage. Replay the run with the fraud tool mocked to return a low score.
If the agent still denies, the exclusion drove the decision. If it now approves, the fraud score was a principal reason, and a letter citing only the exclusion misstates why the claim was denied.
Only trace plus replay can tell those two cases apart. The agent’s own narrative cannot, for the reasons above. Nor can SHAP on the fraud model, which explains the score but not what the agent did with it.
How to Explain an Agentic Decision to a Regulator
Regulators ask the same two questions in different words: what role did the system play, and what were the main reasons for the outcome? Every framework below assumes you can reconstruct a single decision long after it happened.
| Framework | What it asks for | What the agent record must supply |
| EU AI Act Article 86 | For decisions based on Annex III high-risk systems with legal or similarly significant adverse effects, “clear and meaningful explanations” of the system’s role and the main elements of the decision. Article 13 adds that deployers must be able to interpret the system’s output. | The trace, the components that drove the outcome, and the clause or rule relied on |
| Regulation B under ECOA (12 CFR 1002.9) | A statement of the specific, principal reasons for adverse action, whatever technology made the decision. | Replay evidence showing which components were the principal reasons |
| Colorado SB 26-189 | From 1 January 2027, a plain-language description of the system’s role within 30 days of an adverse consequential decision, including insurance and lending, plus meaningful human review on request | A per-decision record a reviewer can read, with reviewer notes kept alongside |
| Fed, OCC and FDIC SR 26-2 | Effective challenge by objective experts across the model lifecycle | A trace and a replay harness, so a validator can test a non-deterministic system |
| NIST AI RMF and NIST IR 8312 | Explainable and interpretable systems, with explanation accuracy as its own principle | Evidence that explanations match the recorded process |
| ISO/IEC 42001 | Annex A control A.6.2.8: event logs, at minimum while the system is in use | Defined, retained event logs per decision |
Three timing points matter. Under the EU AI Omnibus, Annex III high-risk obligations now apply from 2 December 2027. The CFPB withdrew its 2022 and 2023 circulars on algorithmic adverse action notices in May 2025, but the Regulation B text they interpreted is unchanged. Colorado’s law takes effect on 1 January 2027, though xAI and the US Justice Department are expected to challenge it, as they did with its predecessor.
SR 26-2, which replaced SR 11-7 in April 2026, places generative and agentic AI outside its scope. It still expects firms to govern those tools through broader risk management, and the fraud or credit models an agent calls stay in scope. Expect validators to challenge the agent around those models too, and an agent with no trace gives them nothing to examine.
Agent Decision Provenance: What the Record Must Contain
For a decision to be explainable months later, its record needs these fields, captured at runtime:
- Identity and versions: agent ID, model and version, prompt template version and configuration.
- Inputs: the request, the case or customer reference, and timestamps.
- The full trace: the plan, every tool call with redacted parameters, every result, and every retrieval with chunk IDs and document versions.
- Policy decisions: for every guardrail check, the matched policy, the decision, and the reason.
- Component explanations: each score the agent consumed, with its model-level explanation.
- Context attribution: which sources support which claims in the output.
- Final action and human review: reviewer, decision, notes, and timestamp.
- Retention and redaction: the policy applied to the record itself.

Self-reported reasoning can sit in the file, labelled as the model’s narrative. It should never be the evidence of record.
Lumenova AI’s AI Observability records each agent request as an OpenTelemetry trace broken into spans, with tool names, parameters, and results, and masks sensitive fields before traces are sent. The Constraint Trace Explorer records which policy matched, the decision, and the reason for each check, so guardrail evidence sits beside the trace.
Agent detail pages add dashboards and topic clustering for behavioural explanation across sessions. AI Evaluations score full agent workflows from their traces, a different method from scoring final outputs. Human annotations turn reviewer judgement into reusable evidence, and the platform reports how closely automated judges agree with your reviewers.
Can Your Agent Answer “Why Did It Do That?”
If you own model risk, validation, or compliance for agents in production, test yourself with one exercise. Pick last month’s highest-stakes agent decision and try to rebuild it from what you logged: the trace, the policy checks, the sources behind each claim, and the reasons a customer would be owed.
If that exercise exposes gaps, request a demo of Lumenova AI. We will walk through one of your agent workflows against the record spec above. You will see where trace, policy, and evaluation evidence would sit before an examiner or an Article 86 request finds the gaps.
Frequently Asked Questions About Explainable AI for Agents
AI agent explainability is the ability to show why an agent took a specific action, using evidence of what it did. It covers the execution trace, which context supported each claim, how the action changes under counterfactual replay, and which policies constrained it. Classical explainable AI, by contrast, attributes one score to a fixed set of input features.
Not as evidence. Studies of chain-of-thought faithfulness show models often leave out the factors that drove their answers. Fluent explanations also raise acceptance of AI advice whether it is right or wrong, so treat stated reasoning as a hypothesis to check against the trace and replay results.
State the role the agent played, the components that drove the outcome, and the evidence for each: the trace, policy records, context attribution for cited sources, and model-level explanations for any scores. Then show replay results confirming which factors were principal reasons. That maps to Article 86’s “main elements of the decision” and Regulation B’s principal reasons.
Pull the decision record: identity and versions, inputs, the full trace, policy decisions, component explanations, and human review. Run a counterfactual replay of the agent, changing one input at a time, to test which inputs mattered. Then compare the agent’s stated rationale and any customer letter with what the record shows. Without a trace captured at runtime, an audit can describe outcomes but not causes.
Article 86 covers decisions a deployer takes based on output from an Annex III high-risk AI system, where the decision has legal or similarly significant adverse effects. Architecture does not matter, so an agent used for an Annex III purpose such as creditworthiness assessment is likely in scope. Under the AI Omnibus, Annex III high-risk obligations apply from 2 December 2027.