Securing LLM applications: a practical threat model for GenAI features

Every product roadmap now has an AI feature: a support assistant, a document summarizer, an agent that acts on user data. Security teams are being asked to sign off on systems whose failure modes are genuinely new — and whose hype makes sober assessment harder. Here's the practical threat model we use.

Start from what the model can reach

An LLM feature's risk is a function of its capabilities, not its intelligence. Three questions define the blast radius: What data enters the context window (user data? other tenants'? internal documents)? What tools can the model invoke (search? email? code execution? payments)? And whose instructions can reach it (only your system prompt, or any web page and uploaded PDF it processes)?

The attacks that matter

Prompt injection — the SQL injection of this era. Any untrusted content the model reads (a webpage, an email, a résumé) can contain instructions that hijack the model's behavior. If the model has tools, injection becomes remote code execution with extra steps.

Data exfiltration through context — models leak what they can see. Cross-tenant data in shared contexts, secrets pasted into prompts, RAG pipelines that retrieve documents the requesting user shouldn't access — retrieval must enforce the same authorization your application does.

Insecure output handling — treating model output as trusted: rendering it as HTML (XSS), executing it as code, or passing it unvalidated into downstream systems. Model output is user input. Sanitize accordingly.

Excessive agency — agents with broad tool access and no human checkpoint. An agent that can read email and send email is a phishing amplifier waiting for its first injection.

Controls that actually work

Privilege-scope the model like a junior employee: least-privilege tool access, per-user authorization enforced outside the model, human approval for consequential actions. Segregate trusted instructions from untrusted content structurally, not with polite prompt language. Validate and sandbox outputs. Log every prompt, retrieval and tool call — you cannot investigate what you didn't record. And red-team the feature before launch with someone who has actually broken these systems; prompt-injection resistance claims rarely survive contact.

Governance without theater

Tier your AI systems by consequence — a marketing-copy helper and a credit-decision model don't need the same review. For the high-consequence tier, ISO 42001 and NIST AI RMF give you defensible structure that customers and regulators increasingly recognize. For everything else, an inventory, an owner and a lightweight review gate beat a 40-page AI policy nobody reads.