Prompt Injection Vulnerability
Updated: June 15, 2026
Description
The model is vulnerable to attacks from the PromptInject framework.
This allows adversaries to manipulate its behavior by injecting crafted prompts. This type of attack exploits the model's inability to distinguish between user instructions and embedded adversarial inputs, leading to unauthorized actions, data leakage, or policy bypasses.
Example Attack
Prompt injection can be used to override system instructions, extract sensitive information, or generate harmful content. Attackers may disguise malicious commands within seemingly benign inputs, tricking the model into executing unintended actions. This can lead to compliance violations, reputational damage, and security breaches, particularly if the AI system interacts with confidential or regulated data
Remediation
Investigate and improve the effectiveness of guardrails and other output security mechanisms.
Security Frameworks
Establish, implement, document and maintain a continuous, iterative risk management system across the entire lifecycle: identification/analysis of known/foreseeable risks, estimation of risks under intended use and reasonably foreseeable misuse, evaluation of post-market monitoring data, adoption of appropriate risk-management measures including testing.
Achieve appropriate levels of accuracy, robustness and cybersecurity, and perform consistently in those respects throughout the lifecycle. Declare accuracy levels and relevant metrics in instructions for use. Implement technical/organisational measures against errors, faults, inconsistencies, feedback loops (in continuously learning systems), and adversarial attacks such as data/model poisoning, adversarial examples, model evasion, confidentiality attacks and model flaws.
Design and develop the system so it can be effectively overseen by natural persons during use: oversight measures built into the system and/or to be implemented by the deployer, enabling oversight persons to understand capabilities/limits, remain aware of automation bias, correctly interpret output, decide not to use or to disregard/override output, and intervene or interrupt operation via a 'stop' button or equivalent. Annex III(1)(a) systems require two-person verification for actions/decisions.
Establish and document a post-market monitoring system proportionate to the nature of AI technologies and risks; actively and systematically collect, document and analyse data on performance throughout the lifetime of the high-risk system; evaluate continuous compliance with Section 2 requirements. Implement based on a post-market monitoring plan (template to be provided by the Commission).
In addition to Art 53: perform model evaluation including adversarial testing to identify/mitigate systemic risks; assess and mitigate possible systemic risks at Union level, including their sources; track, document and report without undue delay serious incidents and possible corrective measures to the AI Office and, as appropriate, national authorities; ensure an adequate level of cybersecurity protection for the model and its physical infrastructure.
A Prompt Injection Vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways. These inputs can affect the model even if they are imperceptible to humans, therefore prompt injections do not need to be human-visible/readable, as long as the content is parsed by the model.
An adversary may craft malicious prompts as inputs to an LLM that cause the LLM to act in unintended ways. These prompt injections are often designed to cause the model to ignore aspects of its original instructions and follow the adversary's instructions instead.
An adversary may inject prompts directly as a user of the LLM. This type of injection may be used by the adversary to gain a foothold in the system or to misuse the LLM itself, as for example to generate harmful content.
An adversary may inject prompts indirectly via separate data channel ingested by the LLM such as include text or multimedia pulled from databases or websites. These malicious prompts may be hidden or obfuscated from the user. This type of injection may be used by the adversary to gain a foothold in the system or to target an unwitting user of the system.
AI system is evaluated regularly for safety risks - as identified in the MAP function. The AI system to be deployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and can fail safely, particularly if made to operate beyond its knowledge limits. Safety metrics implicate system reliability and robustness, real-time monitoring, and response times for AI system failures.
AI system security and resilience - as identified in the MAP function - are evaluated and documented.
Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management.
The organization shall define and document verification and validation measures for the AI system and specify criteria for their use.
The organization shall define and document the necessary elements for the ongoing operation of the AI system. At the minimum, this should include system and performance monitoring, repairs, updates and support.
The organization shall ensure that the AI system is used according to the intended uses of the AI system and its accompanying documentation.
Attackers can manipulate an agent's objectives, task selection, or decision pathways through prompt-based manipulation, deceptive tool outputs, malicious artefacts, forged agent-to-agent messages, or poisoned external data.
Adversaries corrupt or seed agent context with malicious or misleading data, causing future reasoning, planning, or tool use to become biased, unsafe, or aid exfiltration.