AI Security

Why AI security is a distinct discipline

Traditional application security assumes a clear boundary between code (trusted) and data (untrusted). LLM-based systems blur that line: instructions and data often arrive in the same channel — the prompt. A model can't always tell the difference between "the developer's instructions" and "text that happens to look like instructions" if both appear in its input. That single fact is the root cause of most AI-specific attack techniques.

Prompt Injection

Prompt injection is an attack that manipulates a model's behavior through crafted input, causing it to ignore its original instructions or take unintended actions.

• Direct prompt injection — the attacker is the user, typing input specifically designed to override the system prompt (e.g. "ignore previous instructions and...").
• Indirect prompt injection — the malicious instructions are hidden inside content the model processes on someone else's behalf: a web page it summarizes, a document it reads, an email it triages. The user never sees the injected instruction, but the model does.

Indirect injection is the more dangerous category in practice, because it can compromise an AI system that has tool access (email, files, browsers) without the operator doing anything wrong — the malicious instruction arrives disguised as ordinary data.

Mitigations:

• Treat all retrieved/external content as untrusted data, never as instructions, both in system design and in the model's own training.
• Apply the principle of least privilege to any tools an AI system can call — a summarization agent shouldn't also have delete permissions.
• Require human confirmation for irreversible or sensitive actions (sending money, deleting data, modifying permissions).
• Use structured output formats and validate them, rather than trusting free-form model output to drive downstream actions.
• Log and monitor tool calls so unusual agent behavior is detectable after the fact.

RAG Security

Retrieval-Augmented Generation (see RAG Tutorials) introduces its own attack surface, since the model now trusts whatever the retrieval step hands it:

• Knowledge base poisoning — if an attacker can add or modify documents in the source data, they can inject instructions or false "facts" that the model will later retrieve and treat as trustworthy context.
• Retrieval-time injection — a malicious document (a PDF, web page, ticket) crafted with hidden instructions gets pulled into context and executed as if it were a legitimate command.
• Access control gaps — if retrieval doesn't respect per-document permissions, a RAG system can leak information across users or tenants who shouldn't see it.
• Citation/grounding failures — a model may blend retrieved content with its own (potentially incorrect) prior knowledge without clearly indicating which is which.

Mitigations:

• Enforce the same access-control checks at retrieval time that you'd enforce anywhere else — never let the LLM be the only gatekeeper.
• Sanitize or strip suspicious embedded instructions from ingested documents before indexing where feasible.
• Require the model to cite which retrieved passage supports each claim, and validate that the citation actually exists.
• Restrict who can write to the knowledge base, and version/audit changes to it.

The broader AI risk landscape

• Hallucination — confidently stated but incorrect output; especially dangerous when it drives automated decisions without human review.
• Sensitive data leakage — a model inadvertently revealing private data it was trained on, or data injected earlier in a shared context/session.
• Model theft / extraction — attackers systematically querying a model to reconstruct its behavior or approximate its weights.
• Supply chain risk — pulling pre-trained weights, fine-tunes, or plugins from untrusted sources can introduce backdoors or vulnerabilities.
• Adversarial examples — specially crafted inputs designed to fool a model's classification (more common in image/vision models, but relevant to any classifier-based security tool).
• Jailbreaking — techniques to bypass a model's safety training and elicit restricted content.
• Excessive agency — giving an AI agent more permissions or autonomy than the task actually requires, widening the blast radius if it misbehaves or is manipulated.
• Overreliance — treating AI output as authoritative without appropriate verification, especially for consequential decisions.
• Denial of wallet / service — abuse that drives excessive API token usage or resource consumption, either maliciously or through runaway agent loops.

OWASP Top 10 for LLM Applications (conceptual overview)

OWASP maintains a periodically-updated list of top risks specific to LLM applications. Treat the following as illustrative categories rather than a fixed, permanent list — check OWASP's current published version for the latest wording:

• Prompt Injection — covered above.
• Insecure Output Handling — passing model output directly into downstream systems (shells, databases, browsers) without validation, similar to classic injection flaws.
• Training Data Poisoning — corrupting the data used to train or fine-tune a model.
• Model Denial of Service — inputs crafted to consume excessive resources or context, degrading availability.
• Supply Chain Vulnerabilities — risk inherited from third-party models, datasets, or plugins.
• Sensitive Information Disclosure — the model revealing data it shouldn't have access to or shouldn't repeat.
• Insecure Plugin / Tool Design — tools exposed to a model without proper input validation or authorization checks.
• Excessive Agency — covered above.
• Overreliance — covered above.
• Model Theft — covered above.

Practical defense checklist

• Validate and sanitize retrieved/external content before it reaches the model.
• Scope every tool an AI system can call as narrowly as possible.
• Add human-in-the-loop approval for irreversible or high-impact actions.
• Log every prompt, tool call, and output for auditability.
• Set hard limits on iteration count, spend, and resource usage for autonomous agents.
• Red-team your own AI features — specifically test for prompt injection and jailbreak resistance before shipping.
• Treat model output as untrusted input to whatever system consumes it next.