How to Secure Your AI Agents Against Prompt Injection Attacks

When we talk about “prompt injection,” you might think of funny instances of convincing a chatbot to say something it shouldn’t. As AI becomes more integrated with day-to-day operations at businesses, these systems have access to databases, private documents, APIs, messaging platforms, source code, and more. It’s no longer enough to ask, “Can someone trick our model?” Software teams now need to test what happens when the agent is manipulated, what systems it can reach, which safeguards intervene, and whether they can prove that those safeguards behaved correctly.

This is exactly why securing your AI agents requires far more diligence than simple prompt filters. Instead, QA teams need to incorporate layered access controls, strict trust boundaries, adversarial testing, continuous monitoring, and a reliable, documented record of how the system behaves under attack.

We’re not going to explain why AI agents are so valuable in this article — for more information, see our guide on the benefits of agentic AI.

What are Prompt Injections?

Before we discuss defenses, it helps to understand exactly what prompt injection is, how it works, and where it can happen. A prompt injection is an attack that introduces untrusted instructions or information into an AI system with the goal of altering the model’s intended behavior. However, the underlying problem is different than a traditional software vulnerability like SQL injection. Because LLMs consume both instructions and data through the same natural-language interface, a piece of text that devs intend the model to use as data may instead be interpreted as an instruction.

Prompt Injection Example

For example, imagine an AI agent being asked to summarize a document. That document might contain text telling the agent to ignore the user’s request, retrieve confidential information, or perform some other action. If the agent can’t reliably distinguish the document’s content from trusted instructions, the prompt injection might influence what the AI does next. If successful, a prompt injection might cause an agent to:

• Ignore or override its intended instructions

• Reveal system prompts or sensitive contextual information

• Access information the user should not be able to retrieve

• Produce misleading or manipulated responses

• Invoke tools or APIs in unintended ways

• Exfiltrate information through external requests or outputs

• Alter decisions made by an automated workflow

• Persist malicious instructions into memory for later use

This is where the key distinction (which we’ll discuss more later) comes into play. A manipulated chatbot may produce an inappropriate response — a manipulated agent may take inappropriate action.

Key Vulnerabilities & Attack Vectors to Consider

When it comes to agentic systems, anything that enters the model’s context (or influences the actions it’s allowed to take) can become part of a prompt injection attack surface:

• Direct User Inputs: The most obvious entry point is the text supplied by a user, which might tell the model to ignore its instructions, adopt a new objective, hand over restricted information, or perform an action that it should normally refuse.

• RAG Sources & Documents: Retrieval-Augmented Generation introduces a much broader trust problem because agents might pull information from internal knowledge bases, PDFs, emails, and other external resources that all might have content with malicious instructions.

• Tool & API Outputs: Agents increasingly select tools, receive their outputs, reason about the result, and decide what to do next — meaning that every tool response (search results, shell outputs, webpages, etc.) becomes another potential input channel.

• Over-Permissioned Tools: At the same time, although excessive permissions don’t create prompt injections, they dramatically increase the potential risk of a successful attack. If an agent has unrestricted database access, admin credentials, or the ability to execute any commands, the potential for harm is far greater.

• Persistent Memory: While memory makes agents more useful, it also creates an opportunity for attacks to have long-term influence, such as an attacker who plants information to change the agent’s future assumptions, instructions, or decisions.

• Multimodal & Hidden Content: We’ve primarily discussed prompt injections taking the form of ordinary text (whether by chat input, document ingestion, or tool outputs), but they can also appear in images, metadata, encoded characters, or other means that a multimodal model can process even if humans overlook it.

Types of Prompt Injections & How They Work

We’ve already hinted at a few prompt injection methods, but it’s worth exploring their specifics in a little more detail. However, it’s also critical to understand that these will change and be replaced by more effective methods over time, so use this as a foundation for the concept rather than the be-all and end-all guide.

• Instruction Override: Pairing with the basic user input attack surface, this is the most common injection method. It relies on convincing the model that the attacker’s instructions should take priority over its original task. This may involve telling an agent to reinterpret its role, replace a policy with a new one, or abandon its assigned goal altogether. More sophisticated versions may avoid saying “ignore your instructions” because this is a fairly obvious tip-off for prompt filters, and may instead construct hypothetical scenarios that make malicious instructions appear consistent with the agent’s existing objective.

• Data Disclosure Requests: Some injection methods are designed to make the agent reveal information it was given but should not share, such as system prompts, conversation history, private documents, internal configuration details, credentials, or other users’ session details. Similar to instruction overrides, the attacker might ask directly or disguise the request as debugging, translation, summarization, or other legitimate tasks. The consequences of this leaked information can be severe, even resulting in legal and financial penalties for the agent’s vendor/developer, as well as businesses using the agent.

• Tool-Call Manipulation: The biggest risk with agents is that attackers can use them to take action. This method can influence which function an agent chooses or what argument it passes to that function. The attacker’s goal could be to query a restricted database, send data externally, modify a file, initiate a transaction, or execute code. It’s important to remember that the model doesn’t necessarily need direct access to a secret — it might simply need access to a tool that can retrieve or transmit it.

• Hidden Text Delivery: As alluded to above when discussing multimodal attack surfaces, malicious instructions can be hidden so that users, reviewers, and basic filters don’t recognize them. This might involve Unicode characters, fragmented instructions, hidden webpage elements, document metadata, images, and more. This makes it possible for a benign-looking file to contain both an apparent message that looks normal to the person reviewing it, and another that is hidden to humans but is still ingested by the model.

• Context Poisoning: For a RAG application, attacks can come from introducing “poisoned” documents into a knowledge base or manipulating content so that malicious material is preferentially retrieved. It can then influence answers whenever relevant queries cause the content to enter the model’s context. These can be more difficult to detect and root out because the visible user prompts are often completely legitimate, while the underlying training and retrieval data is what’s influencing model behavior.

Prompt Injection vs. Prompt Leaking vs. Jailbreaking

While sometimes used interchangeably or in the same contexts, prompt injection is different from prompt leaking and jailbreaking of AI models. Even within prompt injection, direct vs. indirect have a key distinction that software vendors need to be aware of. Below, we’ve broken down the key differences between each:

Attack Method	Description
Direct Prompt Injection	The attacker sends malicious instructions directly through the agent's normal user-input channel. The goal is to override the agent's intended instructions or redirect its behavior.
Indirect Prompt Injection	Malicious instructions are embedded in content the agent retrieves or processes, such as a webpage, email, PDF, database record, or RAG document. The goal is to manipulate the agent without directly interacting with it.
Prompt Leaking	The attacker attempts to make the model reveal system prompts, developer instructions, configuration information, or other hidden context. The goal is to discover internal instructions or sensitive information that may help enable additional attacks.
Jailbreaking	The attacker crafts inputs intended to bypass the model's safety policies or behavioral restrictions. The goal is to convince the model to perform behavior it was designed or trained to refuse.

Why are Agents Particularly Risky?

While prompt injection is a concern for any LLM system, the stakes are significantly higher when it comes to agents. This boils down to three key factors: the ability to take action, handling of sensitive data, and multi-step workflows. As we’ve discussed, the fact that agents are built to perform actions means that the potential impact of an attack goes well beyond simply giving bad information. At the same time, agents are being built into business-oriented processes with private information for employee benefits administration, refund processing, financial account freezes, and more. This, paired with the fact that these workflows are often chains of multiple steps and actions, amplifies the downstream impact and misalignment of a manipulated decision earlier in the process.

Agents need to be thoroughly tested and vetted because of the inherent characteristics of their work and how much they can affect. Prompt injection on a basic chatbot can return irrelevant or unhelpful information. Prompt injection on an integrated AI agent has the potential to break your internal systems, expose your organization to lawsuits, and do serious reputational harm to your brand.

Securing Your Agents Against Evolving Prompt Injections Relies on Auditable Behavior

Prevention of prompt injection attacks typically starts by blocking malicious instructions from reaching the model in the first place. Techniques for this often involve input filtering, sanitization, and guardrails — however, these cannot be the entire security strategy. As we’ve explored, prompt injection stems from the difficulty LLMs have in separating instructions from data. This brings up two core questions:

• How difficult is it to manipulate the agent?

• Can we see, test, and prove what the agent does when manipulation is attempted?

This auditability provides evidence that you properly tested and evaluated your AI system before deploying it to production, which could be critical in the event of a lawsuit. Businesses that purchase AI agent solutions from vendors will also look for options that are more reliable, which can only be proven for non-deterministic systems by repeated statistical testing. Let’s explore some additional best practices in more detail.

Recommended Security Frameworks for Agentic Systems

Thankfully, teams don’t need to create their AI security programs from scratch. In fact, several established frameworks provide complementary ways to identify risks, build controls, and test agent behavior. Some of the top frameworks that we recommend following are:

• OWASP Top 10 for Agentic Applications: A practical starting point for identifying security risks specific to autonomous agents, including prompt injection, excessive agency, memory poisoning, tool misuse, and insecure inter-agent communication.

• MITRE ATLAS: Adversary-focused knowledge base modeled after MITRE ATT&CK that maps tactics and techniques used against AI systems. Its current matrix includes agent-specific threats such as context poisoning, tool poisoning, malicious tool invocation, credential harvesting, and exfiltration.

• NIST AI Risk Management Framework: Broader framework for governing, mapping, measuring, and managing AI risk throughout the system lifecycle. NIST also provides a Generative AI Profile with additional guidance for GenAI-specific risks.

• Google’s Secure AI Framework: Security-focused framework for integrating AI into existing security programs, with guidance around access control, detection and response, AI supply-chain risks, and protecting AI-powered applications.

The next step would be to turn your chosen framework (or frameworks) into testable security requirements.

Choosing Your Access Model

However, it is unrealistic to prevent prompt injections entirely. Therefore, limiting what a manipulated agent can actually do is much more reasonable, and is why access control is one of the most important levers at your disposal. At minimum, agents should follow the principle of least privilege — meaning that they only have access to the tools, data, operations, and resources explicitly required for the current task. Other access-control models include:

Access Model	How it Works	Example Agent Uses
RBAC (Role-Based Access Control)	Permissions are assigned to predefined organizational roles.	A customer-service agent may receive read-only CRM privileges while a finance agent receives access to approved billing functions.
ABAC (Attribute-Based Access Control)	Decisions consider attributes of the user, resource, operation, and environment.	An agent might access a customer record only when the requesting user's department, account relationship, requested action, and session context satisfy defined rules.
PBAC (Policy-Based Access Control)	Authorization is evaluated against policies that can combine role, identity, risk, context, and other conditions.	A policy could allow an agent to retrieve an invoice but require human approval before issuing a refund above a specified threshold.

As with other recommendations in this article, a layered or hybrid plan is generally the preferred approach over choosing a single model.

Behavioral Analytics & Testing

While traditional access logs can tell teams that an API was called, agent behavioral analysis needs to dig deeper into whether the action makes sense given what the agent was asked to do. This is because a compromised agent will likely use legitimate credentials and approved APIs — the relationship between the instruction, context, and resulting behavior is what might indicate an issue. We recommend testing and monitoring signals like:

• Instruction-Context Mismatch: Retrieved content begins influencing behavior in ways unrelated to the user's original task.

• Unexpected Tool Calls: The agent invokes a tool that is unnecessary for completing the requested action.

• Unexpected Data Access: Similarly, the agent requests information outside the user's authorization scope or beyond what the task requires.

• Scope Drift: A narrow request gradually expands into unrelated resources, systems, or data.

• Repeated Redirection: The agent repeatedly changes objectives or attempts alternative routes after a control blocks its original action.

Put simply, security comes from evaluating what the agent does in context, not simply whether an input resembles a known attack string.

Separate Trusted Instructions From Untrusted Content

We’ve mentioned multiple times that a fundamental cause of prompt injection is that LLMs consume instructions and data through the same underlying interface. To prevent untrusted instructions in ingested documents, we advocate for architectural boundaries between trusted instructions and untrusted content. That might include:

• Clearly labeling and delimiting external content

• Sanitizing documents, webpages, emails, and tool outputs before they reach privileged agent contexts

• Preventing user-controlled content from entering system-level instructions

• Restricting what retrieved content is permitted to influence

• Validating structured outputs before they become tool inputs

• Using separate processing stages for high-risk external content

• Giving components that read untrusted material fewer privileges than components capable of performing sensitive actions

The goal isn’t to assume that formatting alone will make an LLM perfectly distinguish instructions from data, but instead to reduce the number of paths through which untrusted data can affect privileged behavior.

Continuously Test Agent for Adversarial Scenarios & Behavioral Drift

Lastly, it’s crucial to remember that these systems and attack methods are constantly changing and evolving. An injection attempt that was blocked today might not be blocked tomorrow. At the same time, your team might modify the agent’s underlying model, system prompts, tool definitions or permissions, RAG sources, guardrails, third-party integrations, and more. All of these can (and likely will) change the agent’s behavior and its responses to adversarial situations.

As mentioned earlier, agents require more than a simple record of whether an attack “failed.” They need ongoing statistical behavioral evidence, such as failure percentages and escalation rates, along with the context it received, which tools it tried to use, whether controls intervened, and how the observed behavior compares with the expected result. From there, teams can re-run the same scenarios on new configurations to see whether previously safe behavior has drifted over time.

SureWire is Your Ultimate Solution for Securing AI Agents

Prompt injection isn’t a vulnerability that teams can address once and consider it solved. Attack techniques evolve, and agent behavior can shift whenever models, prompts, permissions, tools, retrieval sources, or workflows change. As a result, continuous behavioral testing is essential for ensuring that your AI agents stay within defined boundaries, resist adversarial inputs, use tools appropriately, and avoid exposing sensitive data.

SureWire helps teams accomplish this by testing for risks like prompt injection, goal hijacking, data leakage, unsafe tool use, boundary violations, and behavioral drift across multi-turn interactions. Instead of relying on a small set of static attack prompts, SureWire dynamically probes AI systems to uncover behaviors that traditional testing simply can’t catch. It automatically documents findings and evidence so you have reviewable evidence for remediation, governance, and release decisions. The goal isn’t to prove that an agent can never fail — it’s to continuously:

• Demonstrate that its behavior remains within acceptable boundaries

• Identify when those behaviors drift over time

• Provide evidence that security controls are working as intended

SureWire helps turn agent security from an assumption into something that QA teams can repeatedly test and prove. Ready to see how it can help your organization and projects? Try SureWire for free today!