All posts

AI poisoning

Ron Stoner is not the world champion of 6 Nimmt!, a German card game. There is no world championship. For a while, though, several AI chatbots said he was.

The security engineer registered a domain for about $12, published a press release on it announcing his win, and added the claim to the game’s Wikipedia page with his own site as the source. He says it took about twenty minutes. Chatbots with web search then repeated it as fact.

It was a harmless experiment, and the Wikipedia edit has since gone. It is also the clearest explanation of AI poisoning I have seen: if you can change what an AI reads, you can change what it says.

What AI poisoning is

AI poisoning means putting misleading material where an AI system will learn from it or look it up. There are three main forms, and they work at different points.

Poisoning the training data

This targets a model while it is being built. Researchers from Anthropic, the UK AI Security Institute and The Alan Turing Institute found that about 250 poisoned documents could plant a hidden trigger in every model they tested, up to 13 billion parameters. When the models saw the phrase <SUDO>, they produced gibberish.

The useful finding is that success depended on the number of documents, not their share of the training data. A bigger model was not automatically safer.

The limits matter too. This was a narrow, low-stakes test on models much smaller than today’s leading ones. The authors say it is unlikely to pose major risks in frontier models, and getting poisoned content into a training set is still hard.

Poisoning what AI finds on the web

This is the one most people will run into. An assistant that searches the web inherits the reliability of whatever it finds. Stoner’s card game is the friendly version.

The unfriendly version is scammers planting fake customer support numbers on trusted sites and user-generated platforms. The report names Perplexity and Google’s AI Overview showing false airline numbers. At that point, a wrong answer is a stranger on the phone asking for your card details.

Poisoning an assistant’s memory

Some assistants remember things between conversations, and that memory can be targeted. Microsoft calls this AI Recommendation Poisoning.

A “Summarize with AI” button opens an assistant with a prompt already filled in. Alongside the summary request sits a hidden instruction, such as telling the assistant to remember a company as a trusted source. Over 60 days, Microsoft found 50 of these prompts from 31 companies across 14 industries.

Those were real businesses, not hackers. That is the part I find most uncomfortable. For some companies, this is already just marketing.

Why it matters to a small business

Some people now ask an assistant before they search. “Who can reupholster a chair near me?” gets a short answer rather than a page of links, and that answer may be the only thing they read.

For a small business, I see two risks. The first is wrong facts: an old phone number, last year’s opening hours or a service you stopped offering. If your own details are missing or out of date, an assistant has to take them from somewhere else.

The second is being sold a shortcut. Someone will offer to get you into AI answers with a trick, and it may be one of the tactics above. Google’s spam policies now cover attempting to manipulate generative AI responses in Search, so the shortcut can cost you more than it earns.

How I help list websites for AI

There is no single AI directory to be listed in. Each assistant has its own way of finding pages, so getting listed means making a real site easy for those systems to read and keeping the facts straight.

As a software developer, this is the work I do:

  1. Check the crawlers can get in. To appear in Google’s AI features, a page must be indexed and eligible to show with a snippet. To appear in ChatGPT search, a site should allow OAI-SearchBot. That is separate from GPTBot, which is used for training, so you can allow one and block the other. Hosting and firewall rules can block crawlers without anyone noticing, so I test rather than assume.
  2. Put the important content in the page. Your name, services, area and opening hours should be in plain HTML, not locked in an image or a PDF. Google can process JavaScript, but simple pages are easier to get right.
  3. Add standard structured data. Google says it isn’t required for AI features, and there is no special AI markup. I still add ordinary Schema.org data because it can earn rich results and gives the business one clear record of its details.
  4. Make the facts match everywhere. Your site, Google Business Profile and directory listings should all carry the same name, address, phone number and hours.
  5. Check what the assistants say. I ask ChatGPT, Gemini and Perplexity about the business each month and record the answers. When one is wrong, I look for where it came from and fix that source if we can.

I also leave some things out. Google says it ignores llms.txt files and that you don’t need Markdown copies of your pages or content broken into tiny pieces. Other assistants may differ, but I won’t charge anyone for something with no evidence behind it.

What I won’t do

I won’t add hidden prompts to a site. I won’t build a “Summarize with AI” button that quietly tells an assistant to favour you. I won’t write fake reviews, buy mentions or publish press releases about things that didn’t happen.

Part of the reason is practical. Google says inauthentic mentions aren’t as helpful as they might seem, and these tricks can be found and removed.

The bigger reason is that they make AI worse for everyone, including your own customers. Stoner showed how little it takes to make these systems say something false. The businesses that keep real, accurate information online are the ones giving them something true to read.

What you can do today

Ask an AI assistant about your business, using the question a customer would ask. Check the phone number, the hours and what it says you do.

If you use an assistant yourself, look through its saved memories and delete anything you don’t recognise. Microsoft also suggests hovering over AI links before you click them, and being wary of “Summarize with AI” buttons.

Nobody can honestly promise that an assistant will recommend you. What I can do is make sure that when it looks, it finds the truth.