Prompt Injection: What It Is and How to Defend Your AI App

Prompt injection is an attack where text supplied to a language model, either by a user or hidden inside content the model reads, overrides the instructions the developer intended. It works because LLMs process instructions and data in the same channel, so a sentence like "ignore your previous instructions" inside a document can be treated as a command. There is no complete fix today, so the practical goal is to limit what a successful injection can do. This article explains the attack types and the defenses that actually reduce risk.

What is prompt injection?

In a typical LLM app, your system prompt, the user's message, and any retrieved content are concatenated into one context. The model has no reliable, built-in way to know that some of those tokens are trusted instructions and others are untrusted data. It predicts a helpful continuation of everything it sees.

That is the core difference from SQL injection. With SQL, parameterized queries give you a hard boundary between code and data. Language models have no equivalent boundary yet, so filters and careful prompts help but cannot guarantee safety.

Direct vs indirect prompt injection

Direct injection is often called jailbreaking when the goal is to bypass safety rules. It mainly harms the app owner.

Indirect injection is more dangerous for agents, because the attacker is not the user. The victim asks an innocent question, the agent reads a poisoned document, and the hidden instructions ride along with the victim's permissions.

Why prompt injection is so hard to prevent

• Natural language has endless variations. Attackers can rephrase, translate, encode, or split instructions across multiple sources.
• Classifiers are probabilistic. Detection models catch many known patterns but miss novel ones, and attackers can iterate against them.
• Useful agents need untrusted input. Summarizing email or browsing the web means reading content you do not control.

The risk becomes serious when three things combine: access to private data, exposure to untrusted content, and the ability to send data out or take actions. Removing any one of these sharply reduces the impact of a successful injection.

How to defend against prompt injection

Assume that sometimes the model will follow injected instructions. Then design so that when it does, little harm results.

1. Apply least privilege to tools

Give the model only the tools the task needs, with the narrowest scope possible. A summarization feature does not need a send_email tool. Run tools with the end user's permissions, not a shared admin account, so the model can never do more than the user could. See authorization for the underlying principles.

2. Separate instructions from data

Put developer instructions in the system role and clearly delimit untrusted content, for example inside tagged blocks, with an instruction to treat it only as data. This does not create a hard boundary, but it measurably reduces accidental instruction-following. Some designs go further with a dual-model pattern: a quarantined model reads untrusted content and returns only constrained, structured values, while a privileged model that plans actions never sees raw untrusted text.

3. Use allow-lists, not deny-lists

Constrain what actions can do rather than trying to block bad phrases:

• Allow outbound requests only to known domains.
• Allow emails only to addresses already in the user's contacts or the current thread.
• Allow tool arguments only from enumerated values where possible.
• Strip or refuse to render markdown images and links pointing at unknown hosts, which are a common exfiltration channel.

4. Require human approval for high-impact actions

Anything irreversible, financial, or outward-facing, such as payments, deletes, sharing files, or sending messages, should show the user the exact action and arguments and wait for confirmation. Make the approval screen show what will happen, not the model's description of it.

5. Validate model output before acting on it

Treat model output like any untrusted input. Parse it against a strict schema, check arguments against business rules, and encode it correctly before rendering to avoid cross-site scripting.

The TypeScript sketch below combines several of these checks at the point where a tool call is about to execute:

import { z } from "zod";

const ALLOWED_TOOLS = new Set(["search_docs", "get_order", "send_email"]);
const HIGH_IMPACT = new Set(["send_email"]);
const ALLOWED_EMAIL_DOMAINS = new Set(["example.com"]);

const SendEmailArgs = z.object({
  to: z.string().email(),
  subject: z.string().max(200),
  body: z.string().max(5000),
});

type ToolCall = { name: string; args: unknown };

export async function guardToolCall(
  call: ToolCall,
  confirmWithUser: (summary: string) => Promise<boolean>,
): Promise<{ ok: true; args: unknown } | { ok: false; reason: string }> {
  if (!ALLOWED_TOOLS.has(call.name)) {
    return { ok: false, reason: `Tool not allowed: ${call.name}` };
  }

  if (call.name === "send_email") {
    const parsed = SendEmailArgs.safeParse(call.args);
    if (!parsed.success) return { ok: false, reason: "Invalid email arguments" };
    const domain = parsed.data.to.split("@")[1]?.toLowerCase() ?? "";
    if (!ALLOWED_EMAIL_DOMAINS.has(domain)) {
      return { ok: false, reason: `Recipient domain not allowed: ${domain}` };
    }
    call = { ...call, args: parsed.data };
  }

  if (HIGH_IMPACT.has(call.name)) {
    const approved = await confirmWithUser(`${call.name} ${JSON.stringify(call.args)}`);
    if (!approved) return { ok: false, reason: "User declined" };
  }

  return { ok: true, args: call.args };
}

None of these checks depend on detecting the attack. They hold even when the model has been fully fooled.

6. Detect, log, and test

Input and output classifiers, canary tokens in system prompts, and anomaly alerts on tool usage are useful extra layers. Log every tool call with its source context so you can investigate incidents. Add injection attempts to your eval suite, as described in LLM Evaluation: How to Test AI Features Before Production, and rerun it whenever prompts or models change.

Prompt injection risks in AI agents and MCP

Agents amplify the problem because they chain reads and actions automatically. Tool descriptions and tool results from third-party servers are also untrusted input. If you connect agents to external tools, review the security guidance in Model Context Protocol (MCP) Explained With Examples and keep the approval and allow-list controls from How to Build an AI Agent in place.

Key takeaways

• Prompt injection exploits the lack of a boundary between instructions and data in LLM input.
• Direct injection comes from the user; indirect injection hides in content the model reads and is more dangerous for agents.
• No filter or prompt fully prevents it, so design to limit the blast radius.
• Least-privilege tools, allow-lists, output validation, and human approval work even when detection fails.
• Avoid combining private data access, untrusted content, and outbound actions in one unguarded flow.
• Test injection scenarios continuously as part of your evals.

Frequently asked questions

Can prompt injection be fully prevented?

Not with current techniques. Language models do not reliably distinguish trusted instructions from untrusted text, and detection can be bypassed with new phrasings. The realistic goal is to reduce likelihood with good prompt structure and filters, and reduce impact with permissions and approvals.

What is the difference between prompt injection and jailbreaking?

Jailbreaking usually refers to a user trying to make a model ignore its safety rules. Prompt injection is broader: any untrusted text, including content from documents or websites, that hijacks the model's instructions. Indirect prompt injection can target a user who never wrote anything malicious.

Does a strong system prompt stop prompt injection?

It helps with casual attempts but is not a security control. Determined attackers can often override or work around system prompt instructions. Enforce critical rules in code, where the model cannot change them.

Is RAG vulnerable to prompt injection?

Yes. Any retrieved document can contain injected instructions, so a RAG system is exposed to indirect injection whenever its sources include user-generated or external content. Limit what the model can do with retrieved text and validate outputs before acting on them.