Explainability Is Not the Same as Transparency, Interpretability, or Accountability

An explanation does not by itself establish that the model is accurate, fair, causal, lawful, secure, or appropriate.

A model can be explainable and still use an impermissible factor. A model can be accurate on average and still perform poorly for a subgroup. A clear explanation can faithfully describe an unacceptable policy.

Four Principles of Explainable AI

NIST identifies four broad principles for explainable AI:

1. Explanation: The system supplies evidence, support, or reasons for its outputs.
2. Meaningful: The explanation is understandable to the intended user.
3. Explanation accuracy: The explanation correctly reflects the process used to generate the output.
4. Knowledge limits: The system operates only within conditions for which it was designed or provides appropriate notice when confidence is insufficient.

Why all four matter

A natural-language sentence may be meaningful but unfaithful. A detailed feature plot may be faithful but incomprehensible to a customer. A valid local explanation may still be misleading when the input is outside the model’s intended population.

Different Audiences Need Different Explanations

The original article correctly identified that one explanation does not fit every audience. The implementation should define the purpose before selecting the format.

Do not expose everything to everyone

Explanations may contain personal information, confidential data, security-sensitive rules, fraud controls, trade secrets, or details that enable gaming.

Use audience-specific access, redaction, aggregation, and disclosure rules while preserving meaningful review and recourse.

Types of Explanation

Local and global explanations answer different questions

A customer may need reasons for one decision. A model owner needs evidence about overall model behaviour and whether the local explanation is representative or unusual.

Do not infer a general policy from one local explanation or treat global feature importance as the reason for an individual decision.

The Limits of SHAP, LIME, and Counterfactual Explanations

SHAP, LIME, and counterfactuals are useful methods. They are not universal truth generators.

SHAP

SHAP uses concepts from Shapley values to assign feature contributions to a model output. The result depends on the model, explainer, background or reference data, feature dependence assumptions, and implementation.

A SHAP value generally indicates contribution relative to a baseline. It does not prove that changing the feature will cause the outcome to change in the real world.

LIME

LIME creates an interpretable approximation around an individual prediction by perturbing inputs and fitting a local surrogate. Results can be sensitive to sampling, representation, neighbourhood definition, and random variation.

The surrogate explains its approximation of the black box near that case—not the complete model.

Counterfactuals

A counterfactual identifies changes that could alter the model output. A useful counterfactual should be:

• Actionable
• Plausible
• Consistent with business and legal rules
• Stable
• Free from protected or immutable changes
• Honest about uncertainty

“Increase income” or “change employment history” may not be immediately actionable. A changed input may also cause other real-world variables to change, invalidating the simple counterfactual.

Visual saliency

A highlighted image region can show model sensitivity without proving that the region contains the clinically, mechanically, or legally relevant reason.

Interpretable Models vs. Post-Hoc Explanations

The original article treated an accuracy-explainability trade-off as a general rule. That trade-off is not universal.

In some problems, a decision tree, scorecard, rule list, generalized additive model, monotonic model, or other interpretable approach can perform sufficiently well or competitively.

Research by Cynthia Rudin argues that high-stakes decisions should use inherently interpretable models where practical rather than relying on explanations of black-box models.

Model-selection questions

• Does the complex model produce a material improvement?
• Is the improvement measured on representative data?
• Does it improve the business or human decision?
• Can the intended users understand and challenge it?
• Can the organization validate and monitor it?
• Can the required explanation be produced reliably?
• Can compensating controls address residual opacity?
• Would a narrower or more interpretable model meet the need?

OSFI’s financial-sector work notes that explainability should be considered from model design and that high-impact uses may justify selecting an explainable model rather than a higher-performing opaque model.

The Market Already Has Explainability Tools

The original article stated that the market did not offer usable explainability tools. That is no longer accurate.

Current examples

• Amazon SageMaker Clarify provides model explainability, bias analysis, and monitoring capabilities.
• Azure Machine Learning’s Responsible AI dashboard includes interpretability, feature importance, error analysis, counterfactual what-if analysis, fairness metrics, and related components.
• SHAP is an open-source framework for feature-attribution explanations.
• LIME remains a widely used model-agnostic local explanation method.
• Model-governance platforms and open-source libraries provide model cards, lineage, validation, monitoring, and fairness analysis.

The market is fragmented

Existing tools may be tied to a cloud, model type, dataset, workflow, programming environment, or internal technical audience. They do not automatically create:

• A legally sufficient customer notice
• A faithful natural-language reason
• Cross-platform decision lineage
• A human-review workflow
• A complaint or appeal process
• Audience-specific communication
• Independent validation
• A complete audit package

Product lifecycle is a risk

Google Cloud deprecated Vertex Explainable AI in March 2026. That does not mean explainability disappeared from Google products, but it shows why an enterprise should not make a critical governance process dependent on one vendor feature without an exit plan.

What Explainable AI as a Service Should Provide

A credible service should begin with an explanation requirements assessment—not an API connection.

Core service components

• AI and model inventory
• Decision and use-case mapping
• Risk classification
• Audience and explanation-purpose matrix
• Model and data access assessment
• Selection of explanation methods
• Reason-code design
• Model cards and technical documentation
• Decision lineage and audit records
• Explanation APIs and user interfaces
• Human review and approval
• Customer notice and service workflows
• Correction, complaint, and recourse
• Explanation fidelity and stability evaluation
• Model, data, and explanation monitoring
• Third-party model governance

Service delivery options

The service should clearly state which models and decisions it can explain and which it cannot.

Model Explanation Is Not the Same as Decision Rationale

A business decision may combine:

• Applicant or customer information
• Model score
• Eligibility rules
• Policy exclusions
• Thresholds
• Manual documents
• Fraud or security controls
• Human judgement
• Exception or override

A feature-attribution explanation only describes the model component. The final decision notice should describe the specific factors that actually determined the outcome.

Example decision record

1. Model version 4.2 produced risk score 0.73
2. Validated reason factors were identified
3. Business threshold for this product was 0.68
4. An eligibility rule also failed because a required document was absent
5. A reviewer confirmed the document was not received
6. The final decision was recorded with approved reason codes
7. The customer received the applicable correction or review path

The explanation service should not say that the model “denied” the application when the final outcome resulted from a rule or human decision.

Natural-Language Explanations Must Be Controlled

The original article proposed real-time natural-language explanations as if they could be generated directly from model percentages. That can create false precision and unsupported causality.

A feature contribution such as “40%” may be misunderstood as a percentage of the reason, legal weight, or causal responsibility. The actual meaning depends on the explanation method and model scale.

Safer explanation pipeline

1. Capture the actual decision inputs and versions
2. Compute the approved explanation method
3. Validate fidelity and stability
4. Map technical factors to controlled business concepts
5. Check whether the factor is permitted and appropriate to disclose
6. Generate wording from templates or tightly controlled structured data
7. Apply human review where consequence is high
8. Store the exact explanation delivered

Do not let an LLM invent the rationale

A language model may make an explanation sound clear while adding an unsupported fact or overstating the model’s confidence.

Use generative AI for wording only when:

• The underlying reasons are structured and validated
• The model cannot add new reasons
• Output is checked against an allowlist or schema
• Important notices receive appropriate review
• The original factors remain available for audit

Use plain language without hiding uncertainty

The explanation should state when the system cannot identify a stable reason or when information may be incomplete. A refusal to provide a misleading explanation is safer than a confident fabrication.

High-Impact Decisions Need More Than Explainability

Credit, insurance, healthcare, employment, fraud, security, legal, and public-service decisions can affect rights, access, health, finances, work, and reputation.

Credit

In the United States, CFPB guidance states that creditors must provide specific and accurate reasons for adverse actions even when complex algorithms are used. Technology complexity is not an excuse for generic or inaccurate reasons.

The applicable requirements in Canada and other jurisdictions differ. Organizations should identify the exact credit, consumer, privacy, human-rights, and provincial requirements that apply.

Insurance

Explainability may be needed across pricing, underwriting, claims, fraud, and service decisions. OSFI’s model-risk guidance requires risk-based lifecycle governance for federally regulated financial institutions and recognizes that explainability requirements vary with purpose, autonomy, regulation, and customer impact.

Healthcare

Model transparency should reflect the needs of clinicians, patients, operators, regulators, and other users. FDA guiding principles for machine learning-enabled medical devices emphasize communicating information that can affect patient risks and outcomes in the context of the intended user and workflow.

Employment

Hiring, promotion, performance, scheduling, discipline, and termination systems require employment, human-rights, privacy, accessibility, and fairness review. An explanation does not cure discriminatory criteria or an invalid assessment.

Fraud and security

Detailed explanations can expose detection logic or allow gaming. Provide enough information for accountable review and appropriate recourse without disclosing sensitive controls unnecessarily.

The Regulatory Landscape Is Real—but Not Uniform

The original article referred to the EU AI Act, FTC guidance, and Canada’s AIDA framework as though they created one universal explainability requirement. That is inaccurate.

European Union

The EU AI Act entered into force on August 1, 2024 and became broadly applicable on August 2, 2026, with staged and amended timelines. Following the 2026 AI Omnibus, rules for certain Annex III high-risk areas are scheduled for December 2, 2027, while high-risk systems embedded in regulated products have a longer transition to August 2, 2028.

The Act includes transparency, documentation, logging, human-oversight, and information requirements for applicable systems. It also includes a right to explanation in specified circumstances where the output of an Annex III high-risk system is used for a decision producing legal or similarly significant effects.

That is not a universal requirement to publish model internals or provide a feature-attribution chart for every AI output.

Canada

The Artificial Intelligence and Data Act was proposed as part of Bill C-27 during the parliamentary session that ended January 6, 2025. It did not become a general Canadian AI statute.

In June 2026, the Government of Canada introduced new proposed privacy legislation that includes increased transparency around automated decision systems. Its final obligations depend on the legislative process.

Existing privacy, human-rights, consumer-protection, sector, contract, professional, and common-law duties may already apply. Federal departments are also subject to the Government of Canada’s Directive on Automated Decision-Making, which includes impact assessment, transparency, quality, recourse, and public-reporting requirements.

United States

Sector-specific laws and regulators apply. CFPB guidance on adverse credit actions is one clear example: complex algorithms do not remove the requirement for specific and accurate reasons under applicable U.S. credit law.

Use-case legal mapping

For each system, identify:

• Countries and provinces or states
• Industry
• Decision type
• Provider, deployer, user, and affected person
• Personal and sensitive information
• Level of automation
• Legal or significant effect
• Required notice, reason, human review, complaint, or appeal
• Documentation and retention

Third-Party and Proprietary Models

An organization remains responsible for how it uses a model even when the model comes from a vendor.

Third-party models may limit access to:

• Training data
• Feature engineering
• Architecture
• Weights
• Reason generation
• Validation evidence
• Model updates
• Subprocessors

Vendor due diligence

• Intended use and prohibited use
• Model version and update process
• Input and output specifications
• Explanation methods available
• Validation and performance evidence
• Population and data limitations
• Security and privacy
• Monitoring and incident notification
• Audit rights
• Change, deprecation, continuity, and exit

Black-box vendor controls

Where internal explainability is limited, alternative controls may include:

• Narrowing the intended use
• Using advisory rather than autonomous output
• Independent benchmark testing
• Outcome and subgroup monitoring
• Input restrictions
• Human approval
• Parallel or fallback models
• Conservative thresholds
• Contractual evidence requirements

OSFI’s 2027 model-risk guideline expects federally regulated institutions to include third-party models in model identification, review, validation, and monitoring proportionate to risk.

Explainability for Generative AI and Agents

Traditional feature-attribution methods do not fully explain why a generative model produced a paragraph, tool call, plan, or multi-step action.

Use traceability instead of pretending to expose an internal rationale

For a generative system, capture:

• Model and version
• System and developer instructions
• User input
• Retrieved documents and versions
• Tool calls and arguments
• Permissions
• Structured intermediate state where appropriate
• Output validation
• Human review
• Final action

Retrieval citations are not enough

A citation can show which source was retrieved. It does not prove that the output interpreted the source correctly, included all relevant information, or followed the governing policy.

Agent explanations

An agent should be able to report:

• Which approved goal it was assigned
• Which tools it used
• Which records it accessed
• Which actions it proposed or executed
• Which approvals it received
• Which policy or rule limited the action
• What failed or remained uncertain

This is an operational trace, not proof that a generated self-explanation faithfully represents internal model computation.

A Practical XAI as a Service Architecture

Security and privacy boundaries

The explanation service may access sensitive inputs, model outputs, fraud controls, employment data, health information, or financial records. Apply:

• Least privilege
• Role- and record-level access
• Encryption
• Secure APIs
• Data minimization
• Retention controls
• Tenant separation
• Audit logging
• Secure deletion
• Incident response

A Twelve-Stage Explainability Implementation Roadmap

Production Readiness Gates

Measure Whether the Explanation Works

Do not measure success by the number of feature charts generated. Measure whether the explanation changes model validation, user decisions, customer understanding, complaint handling, error correction, or governance.

Common Risks and Recommended Controls

FAQs About Explainable AI as a Service