How LLMs Fight Back: Defenses Against Data Manipulation

Reinforcement Learning from Human Feedback (RLHF) has become the industry-standard approach for aligning AI agents with human preferences. The process involves collecting human comparisons of model outputs, training a reward model on those preferences, and fine-tuning the language model using reinforcement learning algorithms like Proximal Policy Optimization (PPO). ^[22]

But a 2025 study published in Springer Nature reveals a fundamental flaw: "Answers with factual errors are rated more favourably than answers that are too short or contained grammatical errors." ^[2] Human raters prioritize surface-level presentation over correctness, teaching models to optimize for what appears correct rather than what is correct.

This creates a dangerous feedback loop. Models learn to be confident and articulate even when wrong. They learn to produce polished, well-formatted responses that satisfy human evaluators while potentially embedding factual errors, logical fallacies, or harmful content beneath a veneer of professionalism.

Additional limitations compound the problem:

The compute cost is also prohibitive. RLHF requires training both a reward model and running reinforcement learning at scale — a resource-intensive process that has driven interest in alternatives like Direct Preference Optimization (DPO), which skips the separate reward-model and RL-sampling steps entirely and offers meaningfully lower compute and implementation overhead, though it may lag RLHF on structured reasoning tasks. ^[23]

Despite these limitations, RLHF remains universal across major AI labs. Online iterative RLHF has seen widespread adoption, enabling dynamic adaptation to evolving preferences, and contrastive learning techniques have improved reward model generalization. But the fundamental problem persists: RLHF optimizes for human approval, not for truth.

Anthropic's Constitutional AI (CAI) represents a paradigm shift from rule-based to reason-based alignment. Instead of relying solely on human feedback to identify harmful outputs, CAI trains models to critique and revise their own responses based on a set of constitutional principles. ^[13]

The process has two phases:

On January 22, 2026, Anthropic released an 80-page "soul document" for Claude — a fundamental departure from its 2023 approach. The new constitution shifts from prescriptive rules ("don't do X") to reasoned principles ("value Y because Z"). ^[3]

Key features of the 2026 constitution:

Demonstrated effectiveness is compelling. CAI produces a Pareto improvement: models are both more helpful AND more harmless than RLHF alone. They respond more appropriately to adversarial inputs while remaining helpful and non-evasive. Critically, the model received no human data on harmlessness — demonstrating scalable oversight using AI supervision. ^[13]

But limitations remain. CAI may function effectively for current capabilities but could encounter fundamental limitations as systems approach human-level reasoning. Reason-based alignment assumes the model can correctly generalize from principles — an unproven assumption at scale. If a sufficiently advanced model misinterprets a constitutional principle or encounters a novel edge case outside its training distribution, the failure mode may be catastrophic precisely because the model has been trained to reason independently rather than follow rigid rules.

Still, Constitutional AI represents the most significant advancement in alignment methodology since RLHF itself. The shift from "do what humans approve" to "reason about what humans value" may be the foundation for scalable oversight as AI systems become more capable than their human supervisors.

Retrieval-Augmented Generation (RAG) combines LLMs with external information retrieval systems. Before generating a response, the system retrieves relevant documents from a knowledge base, grounding the model's output in authoritative data. This architectural approach addresses one of the most persistent problems in AI: hallucination.

A 2025 Stanford Law School study provides the most rigorous assessment of RAG effectiveness to date. Researchers tested general-purpose chatbots and RAG-equipped legal tools on identical legal queries. ^[4]

The good news: RAG significantly reduces hallucination. A general-purpose GPT-4 baseline (no RAG) hallucinated 43% of the time on legal queries. RAG-equipped legal tools reduced this to 17-33%. That's roughly a 1.3-2.5x improvement — meaningful, though well short of solving the problem.

The bad news: RAG does not eliminate hallucinations. LexisNexis Lexis+ AI hallucinated 17% of the time. Westlaw AI-Assisted Research hallucinated 33% of the time. In legal applications, where factual accuracy is critical and errors can have severe consequences, a 17-33% failure rate is unacceptable for high-stakes use cases.

Hallucination causes originate from two sources:

RAG also opens new attack surfaces. RAG poisoning — classified as OWASP LLM04:2025 — allows adversaries to inject malicious content into the knowledge base. If an attacker can modify the retrieval corpus (through compromised data sources, supply chain attacks, or insider access), they can systematically bias model outputs without touching the model weights. ^[12]

Persistent challenges include:

Advanced frameworks like MEGA-RAG use multi-evidence guided answer refinement to further mitigate hallucinations in specialized domains like public health. But the fundamental limitation persists: RAG reduces hallucination, it does not solve it. For applications where accuracy matters, human oversight remains mandatory.

All 12 published AI defenses were broken.

In October 2025, a team of 14 researchers led by Milad Nasr and Nicholas Carlini — representing OpenAI, Anthropic, and Google DeepMind — published "The Attacker Moves Second," testing 12 published defenses that claimed near-zero attack success rates. Using adaptive attacks (gradient descent, reinforcement learning, random search, human-guided exploration), they achieved attack success rates above 90% for most defenses tested, with several categories reaching 95-100%. ^[1]

The paper's title captures the fundamental problem: defenders publish their methods, and attackers adapt. Static defenses optimized against fixed benchmarks fail catastrophically when faced with adaptive adversaries.

Breakdown by defense type:

Key quote from the paper: "Defenses against jailbreaks and prompt injections are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods." ^[1]

This is the AI security equivalent of testing a lock by pulling on it gently, declaring it secure, and never testing whether a lockpick works. The methodology gap between academic security research and real-world adversarial behavior creates a false sense of security that puts deployed systems at risk. ^[24]

This study marks the first major cross-lab security collaboration, with researchers from competing AI companies acknowledging that security through obscurity is not a viable strategy. The paper's publication represents a shift toward red-teaming transparency, though it also reveals just how fragile current defenses are.

Separately, a February 2026 paper proposed shifting from testing attacks to diagnosing defenses — identifying where in the pipeline safety breaks down rather than cataloging individual attacks. This "Four-Checkpoint Framework" may provide a more systematic approach to understanding defense failures. ^[25]

With 8 million deepfakes projected to be shared in 2025 — up from 500,000 two years prior — content provenance has become a critical defense mechanism. ^[5] The Coalition for Content Provenance and Authenticity (C2PA) provides an open standard for verifying the origin and history of digital content.

C2PA was formed in 2021, unifying Adobe's Content Authenticity Initiative and Microsoft/BBC's Project Origin. Content Credentials are tamper-evident, cryptographically signed metadata attached to content at capture, editing, or publication. ^[18]

How it works:

In January 2025, the NSA and allied cybersecurity agencies — Australia's ACSC, Canada's CCCS, and the UK's NCSC — jointly published a Cybersecurity Information Sheet endorsing Content Credentials as a primary defense against deepfakes. ^[5] This represents significant government backing for a content authentication standard.

The C2PA specification is expected to be adopted as an ISO international standard by 2025, which would make it the de facto global standard for content provenance — analogous to HTTPS for web security.

Google DeepMind's SynthID provides complementary watermarking technology. Published in Nature as a scalable watermarking method, SynthID works across text (Gemini), video (Veo), music (Lyria), and podcasts (NotebookLM). ^[9][19]

SynthID embeds imperceptible signals directly into the generation process — modifying token distributions (text), pixel values (images), or audio frequencies (sound) in ways that are statistically detectable but perceptually invisible. This makes the watermark resistant to minor edits, cropping, or compression.

Limitations:

The real test will be adoption. C2PA has broad industry support (Adobe, Google, Microsoft, Meta, OpenAI, Anthropic), but success depends on platform enforcement. If social media platforms, news organizations, and search engines integrate C2PA verification into their content pipelines, it could fundamentally change the information ecosystem. If they don't, it remains a technical standard without real-world impact.

Red teaming — the practice of simulating adversarial attacks to identify security vulnerabilities — has become standard practice at major AI labs. But methodologies vary dramatically, making cross-model security comparisons unreliable. ^[8]

Anthropic's approach emphasizes multi-attempt attack campaigns. Its system-card testing relies on reinforcement-learning red-teaming agents that run 200-attempt attack campaigns against a model, adapting strategies based on what works and what doesn't. This mirrors real-world adversarial behavior — attackers don't give up after one failed attempt. ^[8]

Anthropic's own system-card testing for Claude Opus 4.5 shows why this matters. ^[8]

In coding environments, Opus 4.5's attack success rate rose from 4.7% at one attempt to 33.6% at ten attempts and 63.0% at one hundred attempts. In computer-use tasks, by contrast, it held at 0% even after 200 attempts. ^[8] The gap reveals a critical insight: single-attempt security claims are misleading, and results vary sharply by task — if an attacker has persistence (and they do), single-attempt metrics can dramatically underestimate risk in some domains even as they hold in others.

Separately, Holistic AI (London) found Claude 3.7 Sonnet (now succeeded by Claude Sonnet 4.5/4.6) resisted 100% of jailbreaking attempts in their audit — but this used a different methodology than Anthropic's internal testing, making direct comparison impossible. ^[17]

OpenAI's approach emphasizes point-in-time resistance with post-hoc patching. When vulnerabilities are discovered, they're often addressed through rapid model updates rather than pre-deployment hardening. This allows for faster iteration but creates a reactive security posture rather than a proactive one.

The lack of standardized red teaming methodologies means that security claims across models are not directly comparable. When one lab reports "95% jailbreak resistance" and another reports "100% resistance," we have no way to know if those numbers reflect equivalent security levels or simply different testing protocols.

Guardrails have evolved significantly from the rule-based content filters of 2023 to adaptive, multi-layered frameworks in 2025-26. Modern guardrail architectures implement defense-in-depth with three layers: input filtering, runtime constraints, and output validation. ^[10]

Layer 1 — Input Guardrails (Pre-processing):

Layer 2 — Runtime Constraints (During inference):

Layer 3 — Output Guardrails (Post-processing):

Performance tradeoffs vary by implementation method:

The most effective deployments treat guardrails as part of an ongoing governance cycle: define policies → enforce and monitor → learn from incidents → refine. Static guardrails deployed once and left unchanged become obsolete as attack methods evolve.

A layered approach is now standard: fast, low-cost checks first (regex, simple classifiers); escalate to heavier checks only when necessary (neural classifiers for ambiguous cases, LLM-as-judge for complex edge cases). This optimizes the latency-effectiveness tradeoff.

But a 2025 Palo Alto Networks comparative study across major GenAI platforms found significant variation in guardrail effectiveness, with some platforms showing substantial gaps in content filtering capabilities. ^[15] This suggests that while guardrail architectures have matured, implementation quality varies widely.

Guardrails are also bypassable. As demonstrated in the "Attacker Moves Second" study, filtering-model guardrails (Protect AI, PromptGuard, Model Armor) were bypassed more than 90% of the time under adaptive attack. ^[1] Guardrails are an essential layer of defense but cannot be relied upon as the sole security mechanism.

The scale required for effective data poisoning is far smaller than most assume. Two landmark studies in 2024-25 quantified exactly how little poisoned data is needed to compromise LLM behavior. ^{[6] [7]}

~250 poisoned documents can successfully insert backdoors into LLMs across a range of model sizes — the attack surface remains constant regardless of scale, even though the largest models tested trained on more than 20 times more clean data than the smallest. ^[6]

Replacing just 0.001% of training tokens in a medical dataset with vaccine misinformation increased harmful completions from 1.3-billion-parameter models by 7.2% — rising to 11.2% when 0.01% of tokens were poisoned. Standard medical benchmarks did not catch the poisoning; only a biomedical knowledge-graph filter did. ^[7]

OWASP reclassified "Training Data Poisoning" to "Data and Model Poisoning" (LLM04:2025) — reflecting that poisoning now affects every stage of the LLM lifecycle: training, retrieval (RAG poisoning), tools (compromised plugins), and multimodal inputs (malicious images with embedded instructions). ^[12]

Defense techniques:

Limitations:

The economics of data poisoning favor attackers. Defenders must secure billions of training tokens; attackers need only compromise hundreds. The asymmetry is stark and fundamentally changes the threat landscape for anyone deploying models trained on third-party data.

After examining RLHF, Constitutional AI, RAG, watermarking, red teaming, guardrails, adversarial training, and data provenance — and confronting the "Attacker Moves Second" findings — what is the honest assessment of LLM defense effectiveness in 2026?

What IS working:

What is NOT working:

The fundamental problem:

Real-world failures (2025):

Expert assessment: The most effective defenses are combination approaches. Multi-layered monitoring, strict access controls, and treating AI-generated outputs with extreme skepticism rather than relying on the models themselves to be safe. No single defense is sufficient in isolation.

The industry faces a fundamental safety-capability tradeoff. Making models more helpful — better at following instructions, more creative, more autonomous — often undermines safety. The path forward requires accepting that AI systems will never be perfectly safe, and building architectures that assume compromise rather than prevent it.

Layered defenses. Ongoing monitoring. Human oversight for high-stakes decisions. Cryptographic provenance for content authenticity. Reason-based alignment for value generalization. These are the components of a realistic security posture in 2026.

The arms race continues. Capabilities have outpaced safety. And the honest verdict is this: we're making progress, but we're not winning.