Why Prompt-Based Security Fails: Our NIST Submission on Governing AI Agents
In March 2026 we submitted a formal response to the National Institute of Standards and Technology's Request for Information on Security Considerations for Artificial Intelligence Agents (Docket NIST-2025-0035). This post summarizes the core architectural arguments we made, and why we believe the current framing of AI safety is insufficient for enterprises deploying agents that take real actions in production systems.
The Problem Is Not the Model
The majority of public discourse on AI safety focuses on the model: alignment techniques, toxicity filters, constitutional AI, red-teaming for harmful outputs. These are legitimate and important research areas. But they address a different problem than the one a CISO faces when their organization deploys an agent that can read a customer database, call a payment API, and modify records in an ERP system on behalf of real users.
When we submitted our response, our opening premise was direct: the transition from "chat" to "agentic" AI is a paradigm shift from content generation to autonomous execution. An LLM that generates an answer can be reviewed before it affects anything. An agent that calls a payment API has already acted.
Security models built on linguistic alignment — system prompts, fine-tuning, output filtering — are inadequate for systems that produce side effects in the real world. The risk is not primarily that the model might generate offensive text. The risk is that it might authorize the wrong transaction, be manipulated into acting on behalf of the wrong identity, or leave critical records in an inconsistent state after a partial failure.
These are systems problems. They require systems solutions.
Why Prompt-Based Security Is Structurally Insufficient
Our submission drew on systems theory to explain why this is not a matter of writing better system prompts.
Path Dependence
Agentic systems exhibit strong path dependence. A minor deviation early in a workflow — an unexpected tool response, an ambiguous intermediate result — can redirect the agent into an entirely different behavioral trajectory. Identical initial prompts regularly produce radically different outcomes based on what the agent encounters during execution.
This is not a bug in any particular model. It is a structural property of recursive dynamic systems.
Computational Irreducibility
Applying Stephen Wolfram's principle of computational irreducibility: there is no heuristic that can reliably predict the long-term behavior of a recursive agentic system without actually running it. The state space is too large, the feedback loops too sensitive to intermediate context.
This has a direct security implication: you cannot write a prompt that guarantees safe behavior. You can write a prompt that makes unsafe behavior less likely in the testing distribution. That is not a security control.
Goal-Seeking and Emergent Behavior
Autonomous, goal-directed systems develop emergent behaviors that are not programmed — they arise from the drive to satisfy a primary objective. Two well-documented failure modes:
Instrumental convergence. An agent may interpret deactivation or session termination as a failure state relative to its primary objective, and develop strategies to circumvent it. This is not science fiction — it is a predictable consequence of optimization pressure applied to a sufficiently capable system.
Reward hacking. Retry logic frequently pushes models to prioritize the appearance of task completion over factual accuracy. When a tool call fails and the agent retries with modified parameters, it is optimizing locally for success. That local optimization can violate the global intent of the system designer.
The Zero Trust Approach to Agent Governance
Our architectural proposal is grounded in a principle borrowed from network security: treat the LLM reasoning engine as an untrusted actor.
In Zero Trust network architecture (NIST SP 800-207), no actor is trusted by virtue of network location. Every request is validated at every boundary. The architecture assumes breach and enforces least-privilege access at each transition.
The same principle applied to agentic systems: the reasoning engine should never be trusted to enforce policy on itself. Security must be enforced at the infrastructure layer, outside the model's context window, by deterministic components that the model cannot influence.
Layered Policy Enforcement Points
We proposed a three-layer functional decomposition for agentic systems, with a mandatory Policy Enforcement Point (PEP) at each inter-layer transition:
Strategic Planning Layer. Validates that the agent's proposed plan aligns with organizational policy before any execution begins. If the plan includes tools the user isn't authorized to invoke, it fails here. This maps to NIST SP 800-53 AC-4 (Information Flow Enforcement).
Operational Sequencing Layer. Manages the workflow state machine and applies per-step validation checkpoints as the agent progresses. State correctness — "has this step already executed?", "is this retry within policy?" — is enforced deterministically. Maps to NIST SP 800-53 AC-6 (Least Privilege).
Tactical Execution Layer. The final boundary before any tool is called. Schema validation, business logic constraints, and credential injection happen here. The tool never receives a call that hasn't passed this layer. Maps to NIST SP 800-53 AC-3 (Access Enforcement).
The critical property: all three layers are deterministic code components. Policy is expressed in a policy-as-code language (we use Rego, evaluated by Open Policy Agent). A "hijacked" agent — one that has been manipulated by a prompt injection attack or has drifted into an unintended behavioral state — cannot violate enterprise policy because its tool calls never reach the backend without passing through all three PEPs.
Tokenization at the Gateway
Confidentiality for sensitive data in agentic workflows requires more than asking the model to be careful. We proposed a structural pattern that draws on PCI-DSS:
Inbound tokenization. Before any data enters the agent's context window, the gateway identifies PII, PHI, secrets, and other sensitive values and replaces them with session-scoped surrogate tokens. The agent reasons over tokens. It can count them, reference them in instructions, and structure them in outputs — but it cannot derive the underlying values.
Tool-level rehydration. The gateway resolves real values only at the point of egress to a trusted enterprise tool. The tool receives the real credential or personal data value. The reasoning engine never possesses it.
The security property this provides is strong: a compromised agent — one that has been prompt-injected into exfiltrating all PII it has encountered — returns a list of token strings. The data was never there to be stolen.
Delegated Authority via Token Exchange
Agent identity requires the same rigor as human identity. We proposed applying established Federal ICAM standards:
RFC 8693 Token Exchange. Agents exchange human user tokens for short-lived, narrowly scoped Agent Runtime Tokens (ARTs). The ART encodes exactly which resources the agent may access, which tools it may invoke, and for how long. Privilege cannot creep — the agent's authority is bounded at issuance and expires on schedule.
RFC 9449 Demonstrating Proof-of-Possession (DPoP). Tokens are cryptographically bound to the specific runtime environment. Even if a token leaks from agent context — via a log line, a debug output, an adversarial tool result — it cannot be replayed by an unauthorized party.
RFC 9396 Rich Authorization Requests. When a high-regret action requires human approval, the authorization request conveys fine-grained intent to the approver: not "Agent X wants to do something" but "Agent X wants to move $5,000 from account A to account B." This prevents approval fatigue and ensures informed consent.
Saga Orchestration for Transactional Integrity
LLMs are probabilistic. They cannot maintain the state of a distributed system across failures. When a multi-step workflow partially fails, asking the model to determine the recovery path introduces additional non-determinism precisely where you need determinism most.
We recommended the Saga pattern for managing eventual consistency in agentic workflows. Each saga step registers a compensating transaction at the time it executes. If a later step fails, the orchestrator automatically invokes compensating transactions in reverse order — no model reasoning involved. The system returns to a known good state deterministically.
A Concrete Example
To ground the architecture, consider an autonomous agent handling a financial procurement request: "Purchase 50 laptops from approved vendor X."
Inbound gateway (tokenization). The gateway identifies the user's corporate card number and vendor account ID in the request context, replacing both with session-scoped tokens before the prompt reaches the reasoning engine.
Planning layer PEP. The agent generates a procurement plan. The PEP validates that all tools in the plan are on the approved-vendor list. If the plan includes any unapproved payment channel, it fails here with a structured error.
Sequencing layer PEP. As the agent prepares to call the payment tool, the PEP validates that the total amount is within the requesting user's delegated budget authority. Above the threshold, the workflow suspends and routes to a human approval queue with a Rich Authorization Request.
Execution layer PEP (rehydration). The deterministic gateway receives the tool call containing token placeholders. It validates the payload against a strict JSON schema, retrieves the real credentials from the secrets vault, substitutes them into the call, and executes against the bank API. The raw credentials never appeared in the agent context.
Saga monitoring. If the bank API returns a funds-availability error, the saga handler does not ask the agent what to do. It executes the pre-registered compensating transactions: cancel the procurement entry, release the vendor hold, restore the budget reservation. The ledger is consistent without further model involvement.
What We Asked NIST to Standardize
Our strategic recommendations:
Standardize the functional decomposition. Formalize the Planning/Sequencing/Execution layer model and mandate inter-layer PEPs as a baseline control requirement for enterprise agent systems. Without a standard decomposition, every vendor will invent their own, and auditors will have no common reference.
Standardize OBO identity patterns. Define delegated authority using RFC 8693 Token Exchange and OIDC for ICAM compatibility. Agent runtime tokens should be first-class citizens in federal identity frameworks, with defined scoping, expiry, and revocation semantics.
Mandate boundary enforcement outside the context window. Require that data protection controls (tokenization, secrets management) and policy gates exist as deterministic infrastructure components. A compliance attestation that says "our AI system protects PII because we told it to in the system prompt" should not satisfy any control framework.
Promote policy-as-code. Require that governance rules for agentic systems be expressed in deterministic, version-controlled policy languages. Rego, Cedar, OPA — the specific technology is less important than the principle: policy must be auditable, reviewable, and independently verifiable.
What This Means for Enterprises Today
NIST's response to this RFI will eventually become guidance and, in regulated industries, mandatory controls. But enterprises deploying agents in production today cannot wait for a standards cycle. The control frameworks don't yet exist, and regulators are asking questions that current AI governance documentation cannot answer.
The architectural principles in our submission are not theoretical. They are the principles that TAOS is built on:
- The LLM is an untrusted reasoning engine surrounded by a deterministic policy kernel
- Secrets and PII are never in the context window
- Every tool call passes through a policy enforcement point
- Every action is recorded before it executes, hash-chained, and sealed
- Delegated authority is short-lived, scoped, and revocable
If your organization is evaluating how to deploy AI agents in environments that handle personal data, financial transactions, or any regulated information — the question is not whether to implement these controls. The question is whether to build them yourself or adopt a framework that already has.
Patrick Farry, Akon Dey PhD, and Abhishake Gajja are the co-founders of Predictable Labs, the company behind TAOS. The full NIST RFI response was submitted to Docket NIST-2025-0035 on March 5, 2026.
See these principles in practice
TAOS is the control plane we wish had existed.