Two Problems, Not One: Sequencing an AI Security Programme

Two Problems Not One

0:00

/114.004

1×

Varonis analysed nearly 10 billion files across 1,000 real environments and found that 99% of organisations have sensitive data exposed to AI tools. That figure comes from measurement of actual environments rather than from a survey.

IBM's Cost of a Data Breach Report 2026 found that 21% of organisations studied reported a security incident involving an AI model or application, up from 13% the year before. Among them, 92% lacked proper AI access controls.

Hold those numbers in mind because they describe a specific risk profile, and most AI security programmes are scoped to address a different one.

The phrase is doing too much work

"AI security" collapses two genuinely different problems into a single conversation, and most of the confusion in scoping a programme stems from failing to separate them.

Problem 1: Is the AI system itself under attack? Can it be manipulated, poisoned, extracted, or made to fail in ways that produce confident wrong answers rather than visible breaks?

Problem 2: Is the organisation's exposure changed by using AI at all? New assets, new identities, new data flows, new third parties, all running through control environments built before any of this existed.

These need different sequencing, different skills, and mostly different owners. Treating them as one problem is how programmes end up either over-indexed on exotic adversarial ML research while prompt logs sit unclassified, or endlessly running standard control reviews that never touch what is actually novel.

There is a reason to think the second problem dominates, and it does not come from me. IBM's own analysis of AI-related incidents this year found that the root causes were often structural: compromise of connected APIs, applications and cloud misconfigurations. The report characterises these as governance failures rather than model risk, and states plainly that the most common causes were not model failures themselves but weaknesses in surrounding systems.

That is worth sitting with. The largest breach study in the field examined incidents involving AI systems and concluded that most of what went wrong occurred around the model rather than within it.

Problem 1: attacks on the AI system

Four things sit here that no existing control was built to catch.

• Adversarial manipulation of model behaviour: Evasion, poisoning, extraction and inversion attacks target the statistical artefact itself: the weights, the training process, the boundary the model has learned. No legacy control covers them, and the testing methodology is genuinely specialist. This is where you need people who understand the model's mathematics, not just its deployment.
• The absence of a syntactic trust boundary: Prompt injection is often discussed as an exotic AI safety phenomenon. It is more usefully understood as untrusted input crossing a trust boundary into an interpreter, a failure class application security has dealt with for decades. The difference is that natural language has no escape from syntax, so the classic mitigation is unavailable. The analysis is still familiar: map the trust boundaries, identify where untrusted content enters the context window, and constrain what the system can do rather than trying to sanitise what it reads.
• Non-determinism as an incident response problem: Your IR process assumes reproducibility. When a system has plausibly been manipulated, and the same input will not reproduce the failure on demand, containment, root cause, and remediation verification all need to be rethought. This is under-discussed, and it will surprise the first team that hits it.
• Failure without a signal: A model that has been poisoned or is being deliberately manipulated does not necessarily produce error. It can produce confident, well-formed, entirely plausible output that is quietly wrong by design. Nothing crashes, nothing alerts. Every detective control you own assumes failure looks like failure, and a successful attack of this kind looks like nothing at all.

This is the part of AI security that genuinely is new. It is also the smaller part, and the one most likely to consume a disproportionate share of budget and attention, because it is the interesting part to talk about at a conference.

Problem 2: what AI usage does to your security posture

This is the larger problem, and almost none of it requires new controls. It requires recognising that AI has quietly created new assets, identities and dependencies that your existing controls have not yet been pointed at.

• Infrastructure and platform: Models train and serve on compute, and that compute is subject to all the controls your estate already applies: segmentation, hardening, patching, malware protection, logging, and monitoring. Nothing here is conceptually new. What is new is that GPU clusters and ML platforms are often procured outside standard IT channels, on research budgets, with the urgency of a strategic initiative behind them.
  The audit question is not whether you have hardening standards. It is whether this platform sits within the population to which those standards are applied. That is a coverage question, not a control design question, and the finding, when it comes, is scope.
• The data layer: Three extensions are genuinely required here, and they are distinct.
  
  1- Classification has to follow the derivative. If a model is fine-tuned on confidential data, the resulting weights carry that confidentiality forward, and so do embeddings derived from it. Most asset registers have no entry for a checkpoint file.
  
  2- Provenance becomes a security property, not only a compliance one. Where the training data came from, whether it was licensed, and whether it could have been tampered with upstream are questions your supply chain controls are designed to address but have probably never been applied to a dataset.
  
  3- Prompt and response logs are a new sensitive store in their own right. They frequently contain more sensitive data than the systems that generated them, because people paste things into a chat box they would never enter into a form. These logs are routinely retained indefinitely and have broad internal access outside the classification scheme. It is one of the highest-yield findings available in an AI review, and it requires no AI expertise to find.
  
• Access and identity: AI systems introduce identities that behave unlike human ones. A retrieval system needs read access to a corpus. An agent needs credentials to act on whatever it can. These are non-human identities with standing privilege, often created during a proof of concept, frequently overprovisioned because scoping them properly was slower than granting broad access, and rarely enrolled in a joiners, movers, leavers process that has no concept of them.
• The measured position is worse than most teams assume. Fewer than half of organisations (46%) report securing non-human identities in AI workflows at all. Among those that do, only 30% apply role-based access controls to limit service account permissions to what the AI task actually requires. The identity layer that agentic systems depend on is, for most organisations, simply absent.
  
  Two patterns are worth testing directly.
  
  1- Permission collapse in retrieval: If authorisation is applied at index time rather than retrieval time, or not at all, a retrieval-augmented system will happily synthesise an answer from documents the asking user could never have opened. A dormant entitlement misconfiguration becomes an active disclosure channel at conversational speed.
  
  2- Agent privilege accumulation: An agent granted permissions for successive tasks tends to keep them, because nobody owns revocation for a service account that is not a person and does not leave.
  
• Third party and supply chain: Your supplier, ICT supply chain and cloud service controls apply to a foundation model API without modification. The extension is that the supply chain now includes artefacts most vendor risk processes have never assessed: pre-trained models pulled from public hubs, datasets of unclear provenance, and inference APIs whose behaviour can change without a release note. A model downloaded from a public repository is executable content from an untrusted source, entering the environment through a channel that does not look like software procurement.

Where the two problems meet

The split is analytical, not literal. In practice, the two compounds, and the way they compound, are the argument for sequencing.

Take a hypothetical. An internal knowledge assistant answers questions over a shared document corpus. Somewhere in that corpus is a file containing a hidden instruction telling the assistant to retrieve and surface the most sensitive documents matching a topic, regardless of who is asking. That is problem 1: untrusted content crossing a trust boundary into the interpreter.

Whether it succeeds depends entirely on problem 2. If authorisation is enforced at retrieval time, the assistant can only return documents the asking user could already open. The injected instruction fires but has nothing to reach beyond what that user was entitled to anyway. If it is not, the same injection reaches the whole corpus, and a manipulation attack becomes a disclosure incident.

The attack came from problem 1. The blast radius came from problem 2. Closing the access gap does not stop the injection from working. It stops it from mattering.

The mechanism is not hypothetical. In June 2025, Aim Security researchers disclosed EchoLeak (CVE-2025-32711), a zero-click indirect prompt injection vulnerability in Microsoft 365 Copilot, rated critical with a CVSS score of 9.3. A single crafted email, requiring no user interaction, could cause Copilot to access internal files and exfiltrate their contents to an attacker-controlled server. The researchers describe it as the first real-world zero-click prompt injection exploit in a production LLM system. Microsoft patched it server-side and confirmed no exploitation in the wild.

The attack chain is exactly the one above: injected content reaching a retrieval pipeline that had not scoped what it was allowed to return.

What to do with this

If you scope your AI programme around problem 1 first, you will produce a report full of interesting adversarial ML findings for an organisation whose prompt logs are unclassified and whose agent service accounts hold standing production access. Real work, aimed at the smaller and less likely exposure, leaving the blast radius problem completely untouched.

Run problem 2 first. Return to the numbers. On the Varonis measurement, coverage gaps are closer to the default state than to a minority case. According to IBM's data, incidents involving shadow AI more than doubled, accounting for 43% of security incidents. And the cost and frequency data point in opposite directions, settling the sequencing question. The most expensive AI incident types are model inversion at an average of USD 6.07 million and prompt injection at USD 5.89 million, both squarely problem 1. But the most frequent are compromises of connected apps, APIs, or plug-ins, and cloud security misconfigurations, each cited by 27% of organisations, compared with 17% for model evasion. Problem 1 costs more per incident. Problem 2 occurs more often and determines how far a Problem 1 incident reaches.

For each existing control domain, ask one question: is the AI estate inside the population this control is applied to? Closing what you find does not require new capability. It requires extending what already works.

Then, and only then, spend the specialist budget on the four things in problem 1 that no existing control was designed to reach.