Data poisoning is the deliberate corruption of the data an AI model learns from — slipping malicious, mislabeled, or misleading examples into a training set, a fine-tuning batch, or a knowledge base an agent queries at runtime, so the model absorbs a hidden behavior or a factual error along with everything legitimate it's supposed to learn. Anthropic's own research found that as few as 250 malicious documents are enough to backdoor a large language model, no matter how big the model is or how much clean data surrounds those 250 files.

That finding overturned what most security teams assumed. The working theory was that poisoning scales with dataset size, so the biggest, best-funded models would need a proportionally massive amount of bad data to be affected, making scale itself a defense. What the research actually found is that success depends on an absolute count of poisoned documents, not a percentage of the training set, which means a frontier model trained on trillions of tokens is exposed to almost exactly the same risk as a small open-source model fine-tuned over a weekend.

What Data Poisoning Actually Looks Like

A poisoning attack doesn't need to touch every example a model sees. It needs the model to learn one specific association strongly enough that it fires reliably later: a trigger phrase that makes the model output gibberish, a brand name that gets a suspiciously favorable comparison, or a code snippet that quietly introduces a vulnerability whenever the model is asked to solve a particular kind of problem. The attacker doesn't need access to a company's training infrastructure — they only need their poisoned content to end up somewhere the model reads from, which for a modern pipeline means anywhere from a scraped web page to a support ticket an agent summarizes.

Why Model Size Doesn't Protect You

In October 2025, Anthropic's Alignment Science team, working with the UK AI Security Institute's Safeguards team and the Alan Turing Institute, tested the question directly across models ranging from 600 million to 13 billion parameters. Every size was successfully backdoored using the same 250 malicious documents — roughly 420,000 tokens — planted among otherwise clean training data. The trigger phrase the researchers used was <SUDO>: once a model had been trained on the poisoned set, encountering that phrase made it output random, gibberish text instead of a normal response. The result held regardless of how much more clean data surrounded those 250 documents, which is what made the study, as widely reported across the security press, the moment "absolute count, not relative proportion" became the assumption teams build defenses around.

Where Poisoned Data Actually Gets In

• Pretraining data scraped from the open web, where anyone can publish a page designed to be scraped.

• Fine-tuning and RLHF feedback, where a coordinated group of raters can nudge a model's behavior over time.

• RAG knowledge bases and vector databases an agent queries at runtime, which don't require touching the model's weights at all — corrupting what gets retrieved is enough.

• Third-party datasets and pretrained checkpoints reused without full provenance, where the poisoning happened before the file ever reached the company using it.

Why This Matters More as Companies Deploy Agents

A poisoned chatbot that occasionally says something odd is a reputational problem. A poisoned model or knowledge base wired into an agent that approves refunds, writes code, or summarizes a contract is an operational one, because the trigger doesn't have to be exotic — it can be a specific customer name, a specific phrase in a support ticket, or a specific line in a document the agent is asked to read. PoisonGPT, a proof-of-concept built years earlier to demonstrate the same class of risk, showed that a poisoned model can pass standard benchmark tests with virtually no loss in accuracy, which is exactly what makes the corrupted behavior hard to catch before it reaches production.

How Companies Are Defending Their Pipelines

None of the fixes are exotic, but they all start with knowing where data came from, which is why poisoning has turned provenance tracking from a nice-to-have into a security control. Our explainer on data lineage covers the same tracing infrastructure teams now lean on to answer a narrower question: if a model's output looks wrong, which of the thousands of documents it was trained or fine-tuned on could explain it? Beyond lineage, the practical defenses are auditing third-party datasets before they enter a pipeline, running anomaly detection on the statistical distribution of new training batches, and red-teaming a model against known trigger patterns before it ships, the same way a security team tests for a conventional exploit before a release.

Recommended Coderhouse Courses

If you're responsible for the pipelines these attacks target, the Data Engineering Course covers the provenance and pipeline design work that makes poisoned data traceable in the first place. For teams building or fine-tuning models directly, the AI Engineering Course goes into the training and evaluation practices that catch a backdoor before deployment, and the AI Agents Course is the right next step for teams worried about what a poisoned knowledge base does once it's wired into an autonomous agent.

Frequently Asked Questions

Does data poisoning require access to a company's internal systems?

No. Because pretraining data is often scraped from the open web, an attacker can publish poisoned content publicly and simply wait for it to be picked up, without ever touching the target company's infrastructure.

Can data poisoning be detected by checking model accuracy?

Not reliably. PoisonGPT and Anthropic's own research both found that a poisoned model can pass standard benchmarks with virtually no accuracy loss, since the attack targets one narrow trigger rather than the model's overall performance.

Is a bigger, more expensive model safer from poisoning?

No. Anthropic's tests across 600 million to 13 billion parameter models found that a near-constant number of documents, around 250, backdoored every size tested, which means scale alone isn't a defense.

Does data poisoning only affect training data?

No. A RAG knowledge base or vector database an agent queries at runtime can be poisoned without ever touching the underlying model, since corrupting what gets retrieved has the same effect as corrupting what was learned.

What's the first practical step a team can take against it?

Start with provenance: know exactly where every training, fine-tuning, and retrieval data source came from, since that's what makes it possible to isolate and remove a poisoned batch once one is suspected.