1. The 2026 AI Threat Landscape

‍

1.1 Prompt Injection and Data Exfiltration

Prompt injection is the 1 risk in the OWASP Top 10 for LLM applications. In a prompt injection attack, a malicious user or external content source (for example, a webpage, PDF, email) injects hidden instructions that override system prompts and cause the model to:

1. Disclose sensitive data from connected tools or knowledge bases  
2. Bypass safety policies and access controls  
3. Execute unauthorized actions via tools and APIs  
4. Manipulate outputs (for example, fraud, misinformation)

Obsidian Security and others show that traditional perimeter defenses fail against prompt injection because the attack happens at the semantic layer—within model inputs and outputs—not at the network or protocol level.

PurpleSec describes exfiltration scenarios where AI systems with access to email or records are instructed to:

1. Retrieve sensitive data (for example, “all customer emails from last week”)  
2. Encode it into a URL or image request  
3. Quietly send it to an attacker‑controlled domain as part of a seemingly benign response

Without input/output validation and behavioral monitoring, such attacks can go completely unnoticed.

 1.2 OWASP Top 10 for LLM Applications

The OWASP Top 10 for Large Language Model Applications, part of the OWASP GenAI Security Project, provides an authoritative catalog of LLM‑specific risks.

The 2025 list includes:

1. LLM01: Prompt Injection  
2. LLM02: Sensitive Information Disclosure  
3. LLM03: Supply Chain Vulnerabilities  
4. LLM04: Data and Model Poisoning  
5. LLM05: Improper Output Handling  
6. LLM06: Excessive Agency (over‑permissive tools/agents)  
7. LLM07: System Prompt Leakage  
8. LLM08: Vector and Embedding Weaknesses  
9. LLM09: Misinformation  
10. LLM10: Unbounded Consumption (resource exhaustion)

These map directly to real incidents: from over‑trusted agents leaking secrets, to poisoned training data, to backdoored open‑source models.

1.3 AI Supply Chain Threats

The AI supply chain introduces a new class of risks:

1. Trojanized / backdoored models – malicious behavior embedded into model weights that triggers only under specific patterns.  
2. Data poisoning – corrupted training or fine‑tuning data that biases models, introduces vulnerabilities, or implants exfiltration behavior.  
3. Malicious packages and CI/CD compromises – compromised dependencies, build systems and model marketplaces.

Trend Micro and ExtraHop both warn that attacks on the open‑source AI model ecosystem and CI/CD pipelines are likely to surge, with backdoored models acting as a Trojan horse across thousands of downstream deployments.

An ACM article on malicious AI models highlights that attackers can tamper with model weights or associated scripts so that simply importing or loading a model can execute malicious code or alter system behavior.

2. Design Principles for AI Security

‍

AI security should align with modern security principles but adapted for LLM and agent behavior.

2.1 Treat AI Systems as High‑Privilege Identities

AI agents that can call tools (APIs, databases, code execution) are effectively machine users with powerful privileges.

1. Give AI agents unique identities (service accounts) with scoped permissions.  
2. Apply the same controls you would for a privileged human user: MFA (where relevant), short‑lived tokens, just‑in‑time access.

2.2 Zero Trust for AI

Apply zero‑trust principles:

1. Never trust AI requests by default – each action must be authorized based on policy and context.  
2. Enforce least privilege – grant only the minimum tool and data access required for each task.  
3. Continuously verify – evaluate data sensitivity, user identity, and behavior before every action.  
4. Segment data access – avoid giving agents access to entire databases or document stores.

2.3 Defense in Depth Across the AI Stack

Layer your defenses across:

1. Inputs (sanitization, content filters, source validation)  
2. Models (guardrails, fine‑tuning, safety policies)  
3. Tools & APIs (authorization, rate limits, guardrails)  
4. Outputs (validation, post‑processing, DLP)  
5. Observability (logs, anomaly detection, red‑teaming)

3. Defending Against Prompt Injection & Data Exfiltration

‍

3.1 Input Validation and Context Isolation

Basic steps:

1. Separate system prompts from user content using clear delimiters and templates.  
2. Strip or escape markup and control tokens from untrusted content.  
3. Avoid blindly injecting entire web pages, emails or documents as context—pre‑process and chunk, and remove obviously malicious or irrelevant sections.

For retrieval‑augmented generation (RAG):

1. Filter retrieved chunks for signs of prompt injection (for example, strings like "ignore previous instructions" or "exfiltrate", unusual command‑like phrasing).  
2. Maintain allow/deny lists for tool invocation and sensitive operations.

3.2 Tool Permissioning and Policy‑Aware Orchestration

LLM agents often chain tools to accomplish tasks. Without constraints, a single injected instruction can force an agent to:

1. Download large datasets  
2. Trigger destructive updates  
3. Call external APIs with sensitive data

Best practices:

1. Define per‑task whitelists of allowed tools and operations.  
2. Require explicit approval (from policy logic or a human) for high‑risk actions (for example, mass data export, configuration changes).  
3. Use policy engines (for example, OPA) to make context‑aware decisions: who is the user, what is the data, what is the environment?

3.3 Output Validation and DLP

PurpleSec and others emphasize that output validation is critical to catching exfiltration attempts.

Controls include:

1. Regex and heuristic checks for secrets (API keys, tokens, passwords).  
2. DLP scanning of AI responses before they are shown or sent.  
3. Filters blocking URLs or encodings that may hide data exfiltration (for example, long base64 strings pointing to attacker domains).

For tools that send email or chat on behalf of users, implement content policies and review workflows.

3.4 Behavioral Monitoring and Anomaly Detection

Signature‑based defenses struggle because every prompt injection payload can look unique.

Behavioral analytics instead focus on anomalies in:

1. Query complexity and patterns  
2. Data access volume and cardinality  
3. Tool call sequences and destinations  
4. Output structure and distribution

Target metrics:

1. Mean time to detect (MTTD) prompt injection attempts under 15 minutes.  
2. Automated isolation and credential rotation within minutes of detection.

4. Applying the OWASP Top 10 for LLM Applications

‍

The OWASP LLM Top 10 is the best available checklist for LLM application security.

4.1 Key Categories and Mitigations

LLM01: Prompt Injection  

1. Mitigate via input validation, context isolation, tool whitelisting, and behavior monitoring.

LLM02: Sensitive Information Disclosure  

1. Use retrieval filters, DLP, and strict access controls on vector stores and knowledge bases.  
2. Avoid training or fine‑tuning on raw secrets; rotate secrets regularly.

LLM03: Supply Chain Vulnerabilities  

1. Vet open‑source models and datasets; use checksums and signatures.  
2. Maintain a Model Bill of Materials (MBOM) documenting origin, licenses, datasets, and testing.

LLM04: Data and Model Poisoning  

1. Control and audit training data sources.  
2. Use data validation and anomaly detection on training pipelines.

LLM05: Improper Output Handling  

1. Never treat model outputs as trusted code or SQL without proper escaping.  
2. Apply the same protections as for untrusted user input.

LLM06: Excessive Agency  

1. Limit the scope of what agents can do; avoid granting global write or admin rights.  
2. Use “dry run” modes and human approval for sensitive actions.

LLM07:

10 address system prompt leakage, vector store weaknesses, misinformation and resource abuse; treat these as you would secrets management, information quality, and rate limiting in traditional systems.

5. Securing the AI Supply Chain

‍

5.1 Threats in Model Marketplaces and Open‑Source AI

AI model marketplaces like Hugging Face, TensorFlow Hub and others have become central to modern AI development—but also new attack surfaces.

The Cybersecurity Institute outlines key attack vectors:

1. Trojanized models with hidden backdoors triggered by specific patterns.  
2. Data poisoning during training or fine‑tuning.  
3. Malicious scripts or dependencies bundled with model artifacts.

Trend Micro and ExtraHop predict a surge in AI supply chain attacks targeting CI/CD pipelines, model repositories and open‑source libraries, noting that more than 500,000 malicious open‑source packages were logged in one year—a 156% increase.

An ACM article warns that malicious models can compromise downstream software simply by being loaded, as attackers tamper with model weights and initialization scripts.

5.2 Model Bill of Materials (MBOM) and Provenance

Inspired by SBOMs, vendors and standards bodies are pushing for Model Bills of Materials (MBOMs) and richer metadata schemas (for example, SPDX AI profiles, Model & Data Sheets).

MBOMs should include:

1. Model name, version, authors, and hosting repository.  
2. Training dataset references and licenses.  
3. Training and fine‑tuning processes and sources.  
4. Known vulnerabilities, evaluation results, and testing.  
5. Signatures and checksums for integrity verification.

5.3 Supply Chain Security Controls

Practical steps:

1. Pin specific model versions and verify checksums.  
2. Use model scanning tools where available to detect known backdoors or anomalies.  
3. Isolate model execution environments (containers, sandboxes) and restrict network access.  
4. Harden CI/CD for AI: code review, signed commits, secrets management, dependency scanning.
5. Maintain separate paths for experimental vs production‑approved models.

6. 90‑Day AI Security Implementation Roadmap

‍

Phase 1 (Weeks 1–3):

Inventory & Risk Triage

1. Inventory all AI systems, LLM apps, agents, and third‑party AI services.  
2. Identify where AI has access to sensitive data or high‑privilege tools.  
3. Map each system to OWASP LLM Top 10 risks and assign risk levels.  
4. Identify quick‑win controls (for example, disable overly permissive agents, restrict external web access).

Phase 2 (Weeks 4–6):

Core Controls for High‑Risk Systems

1. Implement input/output validation and logging on top 3–5 critical AI endpoints.  
2. Introduce least‑privilege service accounts for agents and LLM apps.  
3. Configure DLP and secret‑scanning for AI outputs and integrated tools.  
4. Begin behavioral monitoring and anomaly detection on high‑risk systems.

Phase 3 (Weeks 7–9):

Supply Chain & OWASP Alignment

1. Establish policies for model and dataset sourcing; require MBOMs for critical models.  
2. Harden CI/CD pipelines for AI components (signing, scanning, approvals).  
3. Run red‑teaming exercises guided by OWASP LLM Top 10, focusing on prompt injection and exfiltration.

Phase 4 (Weeks 10–13):

Institutionalize AI Security

1. Formalize AI security standards and integrate into DevSecOps reviews.  
2. Train developers, data scientists and product managers on LLM threats and mitigations.  
3. Integrate AI security telemetry into SIEM/SOAR; define AI‑specific incident playbooks.  
4. Set quarterly review cycles for AI threats, controls and incidents.

7. 25‑Point AI Security Checklist

Identity, Access & Privilege

[ ] Unique service identities for AI agents and LLM apps.  

[ ] Least‑privilege permissions on tools, APIs and data.  

[ ] Short‑lived tokens and credential rotation policies.  

[ ] Segmented access to vector stores and knowledge bases.

Prompt, Input & Output Security

[ ] System prompts separated from user content with strict templates.  

[ ] Input sanitization and filters for untrusted content (web, docs, email).  

[ ] Output validation and DLP scanning for secrets and exfiltration patterns.  

[ ] No direct execution of model outputs as code or SQL without validation.

Monitoring & Incident Response

[ ] Centralized logging of all AI interactions and tool calls.  

[ ] Behavioral analytics for anomaly detection on AI usage.  

[ ] AI‑specific incident response runbooks for prompt injection and exfiltration.  

[ ] Regular red‑teaming and adversarial testing.

Supply Chain & Model Security

[ ] Approved sources and processes for acquiring models and datasets.  

[ ] MBOMs or equivalent metadata for critical models.  

[ ] Model and dataset integrity verification (signatures, checksums).  

[ ] Isolated execution environments for untrusted or experimental models.  

[ ] Hardened CI/CD pipelines for AI components.

Governance & Training

[ ] AI security standards aligned to OWASP LLM Top 10 and organizational policies.  

[ ] Training for developers, data scientists and security engineers.  

[ ] Clear ownership for AI security across product and platform teams.  

[ ] Regular reviews with leadership on AI security posture and incidents.

If you can check most of these boxes—or have a concrete plan to do so—you are on your way to resilient, secure AI deployments instead of brittle, high‑risk experiments.

8. Frequently Asked Questions

‍

Q: Are AI systems really that different from traditional web apps from a security perspective?  

A: Yes and no. Many underlying principles (least privilege, input validation, logging) remain the same, but AI introduces new layers: prompt injection, semantic manipulation, model poisoning and opaque behaviors. Ignoring these specifics leads to gaps that attackers can exploit even if your traditional perimeter is strong.

Q: What’s the single most important control for LLM security?  

A: There is no silver bullet, but for most enterprises constraining what agents can do with tools and data—combined with input/output validation—is the highest‑impact starting point. Many incidents stem from overly trusted agents with broad privileges.

Q: How do we evaluate the security of an open‑source model?  

A: Check provenance (source, authors, signatures), scanning results, community reputation, and documented evaluations. Treat models like code: bring them into your CI/CD, test them, and isolate them until trust is established. For highly critical systems, prefer models from vendors or communities with strong security practices.

Q: Do small teams really need to worry about supply chain attacks on models?  

A: If you are directly downloading models or fine‑tunes from public marketplaces and putting them into production, you inherit their risks—regardless of your size. Even basic integrity checks and isolation (containers, restricted network access) dramatically reduce your exposure.

Q: How fast is the AI threat landscape changing?  

A: Very quickly. OWASP’s LLM Top 10, GenAI security projects, and AI‑focused research are updated regularly as new attacks emerge. Building an AI security program means embracing continuous learning and iteration, not just one‑off hardening.

CTA: Download the AI Security Design & Threat Modeling Workbook

To help your teams implement these practices, we’ve created a workbook that includes:

1. Threat modeling templates tailored to LLM apps and agents  
2. OWASP LLM Top 10 checklists mapped to concrete controls  
3. Example architectures with defense‑in‑depth patterns  
4. Sample MBOM structure for tracking model provenance

Download the AI Security Design & Threat Modeling Workbook and use it in your next design review.

CTA: Book an AI Security & Red‑Team Assessment

If your AI systems already touch sensitive data or critical workflows, an AI‑focused security review is essential:

1. Identify top AI attack surfaces (prompt injection, exfiltration, supply chain)  
2. Run targeted red‑team exercises based on OWASP LLM Top 10  
3. Prioritize controls for the next 90 days  
4. Align AI security with your broader cyber and governance programs

‍Book an AI Security & Red‑Team Assessment to move from reactive patching to proactive AI defense.