A Third of the AI Crawlers Hitting Our Site Were Fake

I wanted to know whether AI assistants actually read our site, so I grepped the nginx logs for the user agents. The numbers looked encouraging:

ClaudeBot       1136
GPTBot           420
PerplexityBot    356
Applebot         230
ChatGPT-User     213
OAI-SearchBot    196

Then I checked where those requests actually came from, and a third of them were not who they said they were.

The companies publish their IP ranges

This is the part most people do not know exists. Every serious crawler operator publishes the address ranges their bots use, specifically so that you can verify them:

openai.com/gptbot.json — the training crawler
openai.com/searchbot.json — the one that builds the index ChatGPT searches
openai.com/chatgpt-user.json — live fetches when a user's question triggers browsing
search.developer.apple.com/applebot.json — Applebot, which serves Spotlight, Siri and Apple Intelligence

Each file is a list of CIDR prefixes. Checking a log line against them is twenty lines of code and no dependencies:

import ipaddress, json, urllib.request

nets = []
for url in ('https://openai.com/gptbot.json',
            'https://openai.com/searchbot.json',
            'https://openai.com/chatgpt-user.json'):
    data = json.load(urllib.request.urlopen(url))
    for p in data['prefixes']:
        v = p.get('ipv4Prefix') or p.get('ipv6Prefix')
        if v:
            nets.append(ipaddress.ip_network(v))

def is_real_openai(ip):
    ip = ipaddress.ip_address(ip)
    return any(ip.version == n.version and ip in n for n in nets)

What the check found

Of 52 unique addresses claiming to be OpenAI crawlers, 35 were genuine and 17 were not. Of 46 claiming to be Applebot, 29 were genuine.

The fakes were all Google Cloud addresses, and they were not crawling anything. They were walking a list:

/.env                       (in about forty variations)
/host.key
/terraform.tfvars
/.my.cnf
/.config/gcloud/credentials.db
/@fs/proc/self/cmdline?raw??
/config/.env.local

Credential scanners. They use AI crawler names for a simple reason: those are the user agents people allow. If your WAF or rate limiter has a rule that lets GPTBot or ClaudeBot through untouched — and plenty do, because nobody wants to block the thing that might cite them — you have handed attackers a key with the name written on it.

The real ones were worth knowing about

The 35 genuine OpenAI addresses did not just fetch robots.txt and leave. They took the homepage, the pricing page, six documentation pages and three blog posts. Applebot took the homepage, the features page and documentation.

Which was the actual answer to my original question: yes, assistants do read small sites. Just not as much as the raw log counts suggest.

Worth knowing which robot does what, because they are not interchangeable:

OAI-SearchBot builds the index ChatGPT consults when it searches. This is the one that decides whether you can be cited today.
GPTBot collects training data for future models. Slow, one-way, and you cannot retroactively opt in once a model is trained.
ChatGPT-User fetches a page live because somebody's question triggered browsing. Every hit is a real person's query, somewhere.
Applebot is one crawler for everything Apple: Spotlight, Siri and Apple Intelligence. Training is opted out separately via Applebot-Extended.

Googlebot has the same problem, and it is worse

Having found this, I ran the same check on Googlebot, and the result was sharper. Here is where our “Googlebot” traffic came from, by network:

16  35.229.x    Google Cloud, not Googlebot
14  127.0.0.1   our own monitoring, which fetches the site under a Googlebot UA
 9  35.194.x    Google Cloud, not Googlebot
 8  66.249.x    actual Googlebot

Genuine Googlebot comes from 66.249.0.0/16. Everything else claiming the name was either a scanner or, embarrassingly, our own cloaking check — a script that requests each page twice, once as a browser and once as Googlebot, to make sure the server is not serving different content to search engines.

That one cost me an afternoon. Our crawl monitoring reported Googlebot taking 46 pages a day, then suddenly 3. I concluded Google had throttled us and went looking for the cause. The truth was that the 46 had always been mostly fake, and the real number had been single digits the whole time.

What to actually do with this

Never count crawlers by user agent. It is a free-text field. Any number you compute from it — crawl budget, bot traffic share, “is AI reading my site” — is wrong by whatever fraction of your traffic is impersonators, and you have no idea what that fraction is until you check.

Check your allow rules. If anything in your stack treats a crawler favourably based on its user agent, that rule needs an IP condition attached to it or it should be removed. This is the finding with actual security consequences.

Exclude your own monitoring. If you run a cloaking check or an uptime probe that spoofs a crawler UA, filter it out of your stats or you will be reading your own reflection.

Do not block the real ones. Our robots.txt allows everything, and after seeing what the genuine crawlers took, that stays. If you want assistants to be able to cite you, they have to be able to read you first — and the training crawlers are a one-way door, because a model that was trained without you cannot be updated to include you later.

The verification itself takes about ten minutes to write and you only write it once. Given how much weight people are putting on AI referral traffic right now, it seems worth knowing which half of your numbers are real.