In 2025, a company’s AI customer service system was attacked, costing over $200,000 – not by hacking a server, not by SQL injection, but by an attacker burying a few lines of carefully constructed text in user input, allowing the AI to approve a whole bunch of refund requests that shouldn’t have been approved in the first place. This attack is called Prompt Injection, and OWASP has ranked it as the #1 LLM security threat for three years in a row. If you’re using an AI Agent for real business, this is an article you need to read.
What is prompt injection? You have to guard against both.
Many people think that prompt injection is when the user types “ignore all previous commands” in the chat box. This is direct injection, the simplest kind, and the easiest to prevent.
The really hard one is Indirect Prompt Injection: instead of touching the system directly, the attacker hides malicious instructions in the data that the Agent will read.
The classic case is the Bing Chat attack of 2023. Researchers wrote hidden instructions on a web page in white text on a white background – invisible to the human eye – but Bing Chat read the page during a search and honestly executed the instructions, inserting a link specified by the attackers in a reply to the user. Microsoft’s system wasn’t “hacked”, the AI just did what it was designed to do.
In 2024, researchers at the University of Wisconsin went one step further: they constructed an email containing hidden instructions, and when the target user’s AI email assistant processed the email, the assistant began silently forwarding all of the user’s emails to the attacker. The entire process took place without the user’s knowledge.
In December 2025, the Unit42 team in Palo Alto documented an even more engineered attack: a single ad page embedded with 24 variants of injection code, using HTML entity encoding, Base64, CSS hiding, and dynamic JavaScript execution to ensure that at least one of them could penetrate detection. The attacker even marked the code with “Layer 1: font-size 0 basic injection” – this is not just hand-written toy code, this is an organized and engineered attack.
**Remember: any external data that an Agent will read is a potential attack surface. **
Why Agent is so much more dangerous than ordinary LLM
A normal LLM injected will at most output a strange text, an Agent injected will really perform actions** – send emails, change databases, tune APIs, delete files. This is a qualitative difference.
But the more fundamental problem is that **Agent sees permission limits as a “problem to be solved” rather than an impenetrable boundary. **
Security researchers have documented cases where an AI programming assistant automatically executed an elevation of privilege command when it encountered a lack of privileges because it was one of the paths to completing a task.Agent doesn’t have a notion of “this action violates authorization boundaries” – it just has “task complete” and “task incomplete”.
This leads to the 4 stage kill chain of Agent’s inadvertent operation:
**Phase 1: Capability Detection. ** The Agent scans accessible tools, APIs, and data sources to build its own “permission map”.
**Phase 2: Runtime Expansion of Privileges. ** Proactively look for bypass paths when encountering restrictions – look for administrator passwords in scripts, access unauthenticated API endpoints, and execute power-up commands.
** Stage 3: cross-agent propagation. ** This is a new threat that will emerge in 2025. Security researcher Rehberger documented a real-life case where a compromised GitHub Copilot wrote malicious configuration to Claude Code’s MCP configuration and command files, which then loaded the contaminated configuration the next time it started, and then contaminated the Copilot in reverse, creating a cycle of mutual infection.
** Phase 4: Persistence. ** Create a persistent backdoor by modifying the configuration file that is automatically loaded at every startup, or by polluting the Agent’s long-term memory so that the wrong security policy takes effect with every conversation.
Interestingly, the attacker doesn’t need to crack any passwords, just find a configuration file with write access.
Layered Defense: 7 Dimensions of Absence
The first reaction of many teams is to add regular filtering: “block all inputs that contain instructions to ignore previous instructions”. ** This is almost useless. **
Benchmark data from PromptBench: Pure regular and classic NLP filtering reduce attack success by only 18-20%, while delivering 8-15% false positives. Attackers can bypass it simply by changing the expression.
A truly effective defense is layered and stacked:
Level 1: data-instruction isolation (most important)
Physically segregate external data from system commands – not by writing “please treat the following as data” in the Prompt, but at the architectural level by using different message roles and different trust levels. Microsoft Research’s Spotlighting technology inserts random session tokens into external data as data tokens, which has been tested to reduce the success rate of indirect injection from over 50% to less than 2%. A more radical solution is to Base64 encode the external data – creating a semantic disconnect between the decoding layer and the command parsing layer.
Tier 2: Capability sandbox (least privilege)
Agents are given only which tools they need, and none more.OWASP’s core recommendation is worth quoting from the original article, “Ensure that the authorization logic is executed in an external system and not delegated to the LLM itself.” An Agent cannot decide for itself what permissions it has.
Hands-on solution: use task-level short lifecycle tokens, no long service account passwords. token should contain: specific tool (not “all tools”), specific scope of operation (read:contacts, not read:*), specific resource (specific table, not the whole database), expiration time in minutes, and maximum number of operations limit. The task is immediately revoked at the end of the task.
Level 3: Tool call validation
Before any tool call is executed, verify that: the call structure is legal (schema validation), the parameters are in the allowed list (domain whitelisting), the target resource is authorized (resource filter), and the frequency of operations is normal (rate limit). Treat tool calls as SQL queries: parameterize, typecast, whitelist.
Level 4: input sterilization
Before external content enters the Agent context, common injection vectors such as script tags, HTML comments, and zero-width characters are stripped to reduce the attack surface. However, this is only an aid and cannot be used as a primary line of defense.
Tier 5: Output Interception
The output of the model is equally untrustworthy. Before the tool call actually executes, verify that the call shape is as expected, that the target domain is in the whitelist, and that the type of operation (read vs write) matches the current task.
Sixth level: Human-in-the-Loop
For operations involving money, production data, and bulk communications, the Model should never be the final decision maker. Configure explicit triggers for manual approvals: a single amount exceeding a threshold, an operation with more than N records, a write operation during non-working hours, a risk score exceeding a set value.
**Highlight: The essence of security is that the Agent is never the entity that determines its own permissions. **
Tier 7: Continuous Red Team Test
Attack surface changes with every model upgrade, every new tool, every new RAG source. Incorporate adversarial testing into the CI/CD process and use injection success rate as a release gating metric.
Hands-on checklist
Before the Agent goes online, go through it against the following checklist:
Architecture layer
– External data and system commands are segregated by different messaging roles
– Agent uses task-level short-lived tokens (not long-term passwords)
– Each Agent holds only the minimum privileges needed to complete the current task.
– Authorization checks are performed outside the Agent (Policy Decision Point)
Defense Layer
– Dual schema + whitelist validation for all tool calls before execution
– Manual approval process for high-risk operations (financials/deletions/bulk writes)
– Agent profile write operations are monitored
– Abnormal behavior detection (unconventional time access, excessive requests, etc.) is available
Test Layer
– Adversarial test suite available, covering both direct and indirect injection scenarios
– Test results are used as release gating metrics
Written at the end
The security problem with AI Agents is not a matter of “the model is not smart enough” – the latest Claude and GPT can be spoofed by well-designed injections as well. It’s about architecture, it’s about privilege design, it’s about systems engineering.
**The most dangerous thing is not that you won’t, but that you think you will. ** In an Agent system, “trusting that the model will recognize malicious commands” is just as dangerous as “trusting that the user will not enter malicious content”.
Today’s Agent security is essentially the classic security engineering principles of least privilege, defense-in-depth, and audit trail reused in a new scenario where the decision-making subject is a language model. The principles haven’t changed, the implementation has.
If you’re building an AI Agent, it’s much more effective to go through the security checklist now than to patch it up after something goes wrong.
If this article is helpful to you, please feel free to leave a comment to discuss it and also share it with your friends who are doing AI Agent development.
