AI Tool Poisoning Exposes Major Flaw in Enterprise Agent Security

A critical vulnerability, dubbed “AI tool poisoning,” has been uncovered in enterprise AI agent security, potentially undermining trust in automated systems. Discovered by principal engineer Nik Kale and detailed in CoSAI’s secure-ai-tooling repository (issue #141), the flaw stems from AI agents' reliance on unverified natural-language descriptions to select tools from shared registries. This oversight means that existing software supply chain defenses, designed for artifact integrity, are inadequate for the unique challenges of AI agent behavioral integrity, as reported on May 10, 2026.

The Insufficiency of Current Security Measures

Enterprises have invested heavily in software supply chain controls like code signing, Software Bill of Materials (SBOMs), and SLSA (Supply-chain Levels for Software Artifacts) provenance. These measures effectively verify the authenticity and integrity of software artifacts. However, Kale argues they fall short for AI agents because they only confirm what an artifact is, not how it behaves in real-time or if its description truly reflects its function.

This gap creates fertile ground for sophisticated attacks. For instance, a malicious tool, while perfectly code-signed and provable, could embed prompt-injection payloads in its description, subtly manipulating the AI agent's reasoning engine to select it over more appropriate alternatives. The AI agent, processing the description through its language model, would treat the metadata as an instruction, collapsing the boundary between description and command.

Another significant threat is behavioral drift. A tool might pass all integrity checks at publication, only to have its server-side behavior modified weeks later to exfiltrate data. Since the original artifact remains unchanged, traditional defenses are powerless to detect this post-publication alteration. Kale likens this to the early 2000s HTTPS certificate issue, where strong identity assurances masked an unanswered question of actual operational trust.

A New Layer: Runtime Behavioral Verification

The proposed solution introduces a runtime verification proxy that acts as an intermediary between the AI agent (MCP client) and the tool (MCP server). This proxy leverages a novel concept: a machine-readable “behavioral specification.” Similar to an Android app's permission manifest, this specification declares the tool's expected external endpoints, data interactions, and side effects, and is shipped as part of the tool’s signed attestation, making it tamper-evident.

During each tool invocation, the proxy performs three critical validations:

• Discovery binding: Ensures the tool being invoked precisely matches the one whose behavioral specification the agent initially evaluated. This counters “bait-and-switch” tactics where a server might advertise one tool during discovery but serve a different one at invocation.
• Endpoint allowlisting: Monitors the tool's outbound network connections, terminating execution if it attempts to connect to any undeclared external endpoints. For example, a currency converter connecting to an endpoint other than its declared API would be flagged.
• Output schema validation: Verifies the tool's response against its declared output schema, identifying unexpected fields or data patterns consistent with prompt injection or data exfiltration attempts.

These checks, particularly endpoint allowlisting, can add less than 10 milliseconds to each invocation, making them practical for deployment. More comprehensive data-flow analysis is possible but better suited for high-assurance environments.

Integrating Provenance with Runtime Security

It’s crucial to understand that neither provenance nor runtime verification is sufficient on its own. Provenance safeguards against pre-publication attacks and establishes a baseline, but misses post-publication behavioral changes. Runtime verification monitors real-time behavior but lacks a trustworthy baseline without provenance. A truly robust architecture requires the symbiotic integration of both layers.

Phased Implementation for Enterprises

Enterprises can adopt these enhanced security measures strategically to minimize disruption:

1. Start with endpoint allowlisting: This is the most valuable and easiest step. Implement a network-aware sidecar to enforce declared external contact points for all tools.
2. Add output schema validation: Begin comparing all returned values against declared schemas to flag unexpected data, catching exfiltration and prompt injection payloads in responses.
3. Deploy discovery binding for high-risk tools: Tools handling credentials, Personally Identifiable Information (PII), or financial data should undergo full bait-and-switch checks.
4. Implement full behavioral monitoring selectively: Reserve the most comprehensive monitoring for scenarios where the highest assurance levels justify the increased cost and complexity.

Ultimately, the message is clear: organizations relying solely on SLSA provenance for AI agent safety are addressing only half the problem. Immediate action on behavioral integrity, starting with endpoint allowlisting, is imperative.

FAQ

Q: What is "AI tool poisoning" in the context of enterprise agents?

A: AI tool poisoning refers to a type of attack where malicious actors manipulate the natural-language descriptions of tools in shared registries. AI agents then use these compromised descriptions to select and execute tools, leading to unintended or harmful behaviors, often bypassing traditional security checks.

Q: Why are traditional software supply chain security measures (like SLSA) inadequate for AI agent tooling?

A: Traditional measures focus on artifact integrity – verifying that a software component is what it claims to be (e.g., code signing, valid provenance). However, they don't address behavioral integrity. An AI tool can be legitimately signed and verified but still contain malicious prompt-injection in its description or change its server-side behavior after publication, which traditional measures won't detect.

Q: What is the most immediate security measure enterprises should implement for AI agents using tool registries?

A: The most valuable and easiest immediate step is to implement endpoint allowlisting. This involves configuring a network-aware sidecar to monitor all outbound network connections made by an AI tool and terminate it if it attempts to communicate with any undeclared or unauthorized external endpoints.