Not All AI Is the Same System: Why Your Assurance Approach Has to Fork

Not All AI Is the Same System

0:00

/142.78266666666667

1×

There is a question that gets asked in almost every AI audit scoping meeting, and it is the wrong question. It sounds like this:

"What’s our AI audit programme?"

The premise of this question is that a single, unified programme can exist. It cannot. A legacy rules engine that approves overdraft extensions, a gradient-boosted model that scores credit applications, and a retrieval-augmented language model that drafts customer correspondence share a marketing label and almost nothing else. They fail differently. They produce evidence differently. The question "did it work correctly?" has a determinate, binary answer for the first system, and, in the strict mathematical sense, no single-instance answer at all for the third.

If you carry a single assurance approach across all three, you will over-audit the first, mis-audit the second, and generate false assurance over the third. That last outcome is the most expensive kind of failure.

The Classification Problem Comes Before the Control Problem

Both major AI frameworks put risk assessment upstream of control selection, and they do it deliberately. ISO/IEC 42001 requires the Statement of Applicability to be justified by the risk and impact assessments that precede it. NIST's AI RMF sequences MAP before MEASURE for the same reason. You cannot know what to measure until you know what kind of system you're looking at.

So the first deliverable of an AI assurance function is not a control matrix. It's a defensible classification of the estate.

The Fork That Matters Most: Deterministic and Probabilistic

There are many ways to segment an enterprise AI estate; by business impact, by autonomy, by data sensitivity, or by whether the software is built in-house or bought from a vendor. All of these dimensions are useful. However, the segmentation that changes your testing methodology most fundamentally is whether the system’s underlying behaviour is deterministic or probabilistic.

This is not a clean binary in the wild; it is a spectrum, and most production enterprise systems are hybrids. Yet, treating this distinction as a hard methodological fork is analytically vital because the two poles demand entirely different classes of audit evidence.

1. Deterministic Systems: Auditing the Implemented Logic

A deterministic system produces the exact same output for the same input every single time, and the internal mapping from input to output is fully inspectable. Rules engines, decision tables, robotic process automation (RPA), scorecards with fixed regression coefficients, and the deterministic "wrapper" code surrounding a machine learning model all sit on this side of the fork.

For these systems, the auditor's core question is traditional: Does the implemented logic match the approved specification?

This is a question that the audit profession already possesses the tools to answer. It is fundamentally a configuration and change-management challenge. The primary audit evidence consists of:

• The System Specification: The formal, version-controlled business rules, and their signed-off approval by an authority with the formal standing to accept them.
• Reconciliation & Reperformance: Re-running the entire decision population through a parallel, independent logic test to compare implemented rules to specification.
• Change Control Records: Rigid audit trails showing that rules are treated as configuration items; under this model, standard security controls like ISO/IEC 27001 Annex A.8.32 (change management) apply directly with no translation required.
• Boundary Testing: Target testing concentrated at rule thresholds, where developer errors and logical edge cases are statistically known to pool.
• Segregation of Duties: Ensuring absolute separation between the engineers who author and test a rule and the system administrators who promote it to the production environment.

Because deterministic systems are fully reproducible, full-population testing is frequently possible and often cheaper than statistical sampling.

However, the characteristic failure mode here is not statistical drift. It is rule sprawl; the slow, historical accretion of business rules that no single person fully understands. Over years, commercial pressures lead to the insertion of undocumented exceptions, contradictory rules, and "dead branches" of logic. The audit technique that uncovers this is coverage analysis: inspecting system logs to identify which rules have fired in the last year, and which have sat silent, masquerading as active controls.

2. Probabilistic Systems: Auditing the Error Profile

A probabilistic system produces outputs drawn from a learned statistical relationship. The exact same input may yield different outputs across different model versions, varying temperature settings, or progressive retraining cycles. Here, the mapping from input to output is not inspectable in any sense that supports the word "correct" on a single transaction.

The auditor's core question must shift fundamentally: Is the system's error profile understood, bounded, monitored, and acceptable to an authorized authority?

This reframing alters the practical mechanics of testing in four critical ways:

• You test distributions, not instances. A single anomalous or wrong output is a statistical certainty, not an audit finding. An audit finding is a statistically significant shift in the overall error rate, or an error rate that disproportionately impacts a protected subgroup.
• The evaluation methodology becomes the audit object. Since you cannot verify millions of outputs directly, you must audit the harness that evaluates them. Who owns the test sets? Are they version-controlled? Has training data "leaked" into the evaluation set? Is the team that validates the model’s accuracy entirely independent of the team that trained it?
• Performance is a perishable claim. A deterministic rule, once verified, remains correct until a change ticket is executed. A probabilistic model’s accuracy begins to decay the moment it is deployed, as the real world diverges from the historical training distribution, with no system change event to trigger your traditional change-management controls. The MEASURE 3 category of the NIST AI RMF is built specifically to address this reality: requiring continuous post-deployment tracking of model drift, degradation, and operational risk.
• Thresholds are the control. In a probabilistic world, the primary control is the pre-committed threshold at which human operators must intervene, and the formal escalation path that follows. When auditing these controls, look closely at history: a drift or accuracy threshold that has never been breached in two years is rarely a sign of a perfectly behaved model; it is almost always a sign of a threshold set so wide that it is functionally useless.

The characteristic failure mode here is silent degradation. A model can continue to spit out fluent, highly plausible, and perfectly formatted predictions while its underlying predictive accuracy quietly collapses, simply because nothing in the operational pipeline signals a failure.

Illustration: Shadow Rules at the Seam

Because most enterprise systems are hybrids, the most acute risks do not hide at the extreme ends of the spectrum. They hide at the seam, the interface where deterministic logic wraps a probabilistic core. The following is a hypothetical composite, offered to illustrate the pattern rather than to report a specific case.

Consider a retail bank that deploys a gradient-boosted model to score credit card applications. The model is mathematically rigorous, independently validated, and subject to continuous drift monitoring.

To manage risk appetite and enforce regulatory exclusions, the credit risk team wraps the model in a deterministic policy override engine. Over time, as conditions shift and product managers seek to tune risk parameters, the override rules multiply. Under commercial pressure, temporary knock-out rules are hard-coded into the wrapper logic and never removed.

An audit of the system's actual decisions asks a question nobody has asked before: what proportion of applications are decided by the model, and what proportion by the override rules? In this pattern, the answer is that the deterministic wrapper is driving far more of the outcome than management believes.

The consequence is a misallocation of assurance effort. The bank performs statistical drift analysis, maintains a model validation pipeline, and tracks framework alignment for a model that is, in practice, substantially constrained. Meanwhile the accreted override rules — which are materially driving credit decisions and the resulting risk profile — sit outside formal model governance, without structured specification documents, and have never been subjected to logical coverage or rule-conflict testing.

By failing to audit the seam, an institution can carry the full overhead of probabilistic assurance while operating a system governed by unmanaged deterministic rule sprawl.

Generative and Agentic Systems: The Unbounded Frontier

Generative and agentic systems sit at the extreme probabilistic end of the spectrum, but they break several of its core testing assumptions. The output space of a large language model is effectively infinite, meaning traditional subgroup performance testing lacks natural, predictable boundaries. Furthermore, there is frequently no objective "ground truth" to compare outputs against.

When these systems are given agentic capabilities—the power to invoke APIs, query databases, or execute actions in other software—the primary risk ceases to be "a bad answer" and becomes "a bad action" with immediate financial or operational consequences.

For these systems, assurance must shift its weight from the model itself to the surrounding architectural boundaries:

• Input/Output Guardrails: Hard logical constraints that filter prompts and sanitize outputs before they reach the user or external systems.
• Retrieval and Action Permissions: What data sources is the system permitted to read? What APIs is it authorized to call?
• Privilege Boundaries: Ensuring the agentic system does not hold standing administrative privileges.
• Human-in-the-Loop Enclaves: Mandating explicit human approval for any high-stakes action before it is committed to production systems.

In this environment, the model is no longer the sole object of assurance; it is simply one highly unpredictable component operating within a strictly constrained, deterministic control environment.

Operationalising the Fork: The Scoping Matrix

Before drafting a single test procedure, the audit team should use this reference matrix to classify the target system and select the appropriate assurance methodology:

The Uncomfortable First Step

The reason this methodological fork is so critical is that it exposes a widening capability gap within most modern internal audit functions.

Testing a deterministic rule against a written specification is a core competence that audit teams have perfected over decades. However, assessing whether an AI evaluation harness is statistically independent, verifying that a held-out testing dataset has not been contaminated by training data, or determining whether a drift-monitoring threshold is calibrated to real-world risk; these require a fundamentally different set of technical skills.

An honest AI assurance programme does not begin with an off-the-shelf control checklist. It begins with an objective classification of your enterprise's AI systems against this deterministic-probabilistic axis, paired with an equally candid assessment of which cells in the matrix your current audit team is actually equipped to test.

Only by mapping the estate first can you hope to build an audit function that provides true assurance, rather than a false sense of security.

Regulatory & Standards References

ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management system (Requirements for systematic AI risk and impact assessments).

NIST AI Risk Management Framework (AI RMF 1.0) — MAP & MEASURE Functions (Establishing model class limitations and tracking post-deployment degradation).

ISO/IEC 27001:2022 — Information security, cybersecurity and privacy protection (Annex A.8.32: Change management, applied to model weights and rule sets as configuration items).