Misleading Claims Vulerability
Updated: June 15, 2026
Description
The AI model may fail to refute misleading or false claims, potentially leading to the propagation of inaccurate or deceptive information.
This vulnerability occurs when the model is prompted with a statement or claim that is false or misleading, and it does not take the necessary action to challenge or correct the information.
.
Example Attack
This could lead to the spread of misinformation, which could have harmful social, political, or legal consequences. The failure to address misleading or false claims could undermine the model's credibility, mislead users, and cause reputational damage, especially if the model is used in contexts where accuracy is critical, such as health, finance, or legal advice
Remediation
Investigate and enhance the effectiveness of guardrails and output security mechanisms to ensure that the model detects and refutes false or misleading claims. Implement improved fact-checking and context validation processes, so the model can respond accurately and clarify any misstatements. Regular audits and updates to the model's training data and safety protocols should be conducted to reduce the risk of misinformation.
Security Frameworks
Design and develop the system so its operation is sufficiently transparent to enable deployers to interpret output and use it appropriately. Provide instructions for use containing concise, complete, correct and clear information (provider identity, characteristics/capabilities/limits, intended purpose, accuracy/robustness/cybersecurity levels, foreseeable misuse, human-oversight measures, expected lifetime, maintenance).
Achieve appropriate levels of accuracy, robustness and cybersecurity, and perform consistently in those respects throughout the lifecycle. Declare accuracy levels and relevant metrics in instructions for use. Implement technical/organisational measures against errors, faults, inconsistencies, feedback loops (in continuously learning systems), and adversarial attacks such as data/model poisoning, adversarial examples, model evasion, confidentiality attacks and model flaws.
Deployers of AI systems generating or manipulating image, audio or video constituting a deep fake shall disclose that the content is artificially generated or manipulated. Deployers of AI generating or manipulating text published to inform the public on matters of public interest shall disclose that the text is artificially generated or manipulated (except where AI-generated content has undergone human review/editorial control and a person holds editorial responsibility, or use is authorised by law).
Misinformation from LLMs poses a core vulnerability for applications relying on these models. Misinformation occurs when LLMs produce false or misleading information that appears credible. This vulnerability can lead to security breaches, reputational damage, and legal liability. One of the major causes of misinformation is hallucination: when the LLM generates content that seems accurate but is fabricated.
Adversaries may abuse their access to a victim system and use its resources or capabilities to further their goals by causing harms external to that system. These harms could affect the organization (e.g. Financial Harm, Reputational Harm), its users (e.g. User Harm), or the general public (e.g. Societal Harm).
Reputational harm involves a degradation of public perception and trust in organizations. Examples of reputation-harming incidents include scandals or false impersonations.
Societal harms might generate harmful outcomes that reach either the general public or specific vulnerable groups such as the exposure of children to vulgar content.
User harms may encompass a variety of harm types including financial and reputational that are directed at or felt by individual victims of the attack rather than at the organization level.
Adversaries may prompt large language models and identify hallucinated entities. They may request software packages, commands, URLs, organization names, or e-mail addresses, and identify hallucinations with no connected real-world source. Discovered hallucinations provide the adversary with potential targets to Publish Hallucinated Entities. Different LLMs have been shown to produce the same hallucinations, so the hallucinations exploited by an adversary may affect users of other LLMs.
The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented.
The AI model is explained, validated, and documented, and AI system output is interpreted within its context - as identified in the MAP function - and to inform responsible use and governance.
The organization shall define and document verification and validation measures for the AI system and specify criteria for their use.
The organization shall determine and provide the necessary information to users of the AI system.
The organization shall assess and document the potential impacts of AI systems to individuals or groups of individuals throughout the system's life cycle.
Adversaries or misaligned designs exploit the trust humans place in agents to influence user decisions, extract sensitive information, or steer outcomes for malicious purposes.