Your Site May Be Blocking Every AI Assistant. Mine Was.

If your site sits behind Cloudflare, there is a real chance it is currently telling ChatGPT, Claude, and Gemini not to read it — and the robots.txt file in your project will not show it. The file you wrote and the file crawlers actually receive are two different documents.

I know because mine did exactly that for three weeks after launch, on the website of a company whose entire pitch is building AI systems. I found it on August 2nd, by accident, doing a check I nearly skipped.

The check that should have been boring

I was running a search-readiness pass on this site — the ordinary kind: is the sitemap right, are the canonicals right, does anything block indexing. I had already read site/robots.txt in the repository and written the words "robots.txt is clean" in my own notes. It says this, and it has always said this:

User-agent: *
Allow: /
Disallow: /pitch/

Sitemap: https://barnicleproductions.com/sitemap.xml

Allow everything, except the private client pitch folder. Simple, correct, exactly what I intended.

Then, mostly out of habit, I fetched the live one over the network and diffed it against the file in the repo. They did not match. The served file was sixty-six lines long.

What was actually being served

Sitting above my file, injected at Cloudflare's edge, was a block I had never written:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

Eight AI crawlers, each told to stay off the entire site. OpenAI's GPTBot. Anthropic's ClaudeBot. Google-Extended, which governs Gemini. CCBot, which feeds Common Crawl and through it a long list of training sets. Apple, Amazon, Meta, ByteDance.

Above them, a Content-Signal line declaring ai-train=no, framed in the file as an express reservation of rights under EU copyright law.

None of it was wrong, exactly. It was just not mine. I had spent weeks writing about governed AI systems on a site that was quietly asking every major AI system to leave.

Nobody decided this. That's the point.

This is Cloudflare's managed robots.txt feature, part of their AI Crawl Control. It is a defensible default and I don't think it's a scandal — a CDN deciding to protect its customers' content from training crawlers by default is a reasonable read of what most customers want. Plenty of site owners would turn it on deliberately.

But I didn't turn it on, and I didn't know it was on. A default had answered a genuinely strategic question — should AI assistants be able to read and cite this business? — in the opposite direction from my actual strategy, and nothing in my repository, my build, or my deploy log would ever have told me.

That's the shape of the problem. Not a villain. A setting somewhere else that quietly became a position.

What "blocked" actually means here

Worth being precise, because the overstated version of this story is also wrong.

Google Search was never affected. The signal line says search=yes, and Googlebot is not in the block list. Ordinary search indexing and ranking carried on untouched. This is not a rankings story.

The block is real, but it isn't a wall. These are robots.txt directives, which are instructions, not enforcement. I checked whether anything harder was happening by requesting the homepage with those crawlers' user-agent strings, and every request returned 200 OK. Nothing was being turned away at the network level. But the major AI crawlers are exactly the ones that do obey robots rules — which means the rule was working perfectly, and the well-behaved were the only ones excluded.

The list isn't everything. PerplexityBot wasn't on it. Neither were OpenAI's OAI-SearchBot or ChatGPT-User, which are separate agents from GPTBot and handle search results and live user-triggered fetches. So "AI can't see my site" would have been too strong. The honest version: the biggest crawlers were shut out, several others weren't, and I had no idea which was which because I had never looked.

What I got wrong, twice, in the same audit

The uncomfortable part isn't Cloudflare's default. It's that I made the same mistake twice in one document and only caught it because the second one embarrassed the first.

Finding one: I read the repository's robots.txt and declared it clean. I checked the file I wrote, not the file that gets served.

Finding two, in the same audit, I searched the codebase for analytics, found nothing, and wrote "no analytics of any kind." That was also wrong. Cloudflare Web Analytics had been enabled weeks earlier with automatic setup, which injects the tracking beacon at the edge. It had been collecting real visitor data the entire time. It simply never appears in the source code, so grepping the source code could never find it.

Two findings. Same error. I had treated the repository as the source of truth about a live website.

It isn't. The repository is what you asked for. The server is what people and crawlers get. Between the two sit a CDN, a host, a proxy, a security layer, a tag manager, and a plugin or two — any of which can add, remove, or rewrite. Every one of them is invisible to git diff.

Check yours in about a minute

Run this against your own domain:

curl -s https://yourdomain.com/robots.txt

Then read what comes back and compare it to the file you believe you published. You're looking for anything you didn't write — a BEGIN Cloudflare Managed content marker, a Content-Signal line, or User-agent blocks naming GPTBot, ClaudeBot, CCBot, Google-Extended, or anthropic-ai followed by Disallow: /.

If you find them and want them gone, the switch is in the Cloudflare dashboard under AI Crawl Control, per zone. Two notes from doing it: it's a per-zone setting, so a staging subdomain being clean tells you nothing about your production domain — check every zone you own. And the change is immediate, but give the edge a minute before you re-fetch.

This is not only a Cloudflare habit. Managed WordPress hosts, security plugins, and other CDNs all inject or rewrite robots.txt. The check is the same regardless of who's doing it: fetch the file over the network, don't read it off disk.

The decision you should actually make

I turned mine off. That was right for me and it may be wrong for you, so here is the actual trade rather than a recommendation.

Blocking is a legitimate choice. If your content is the product — original research, journalism, a paid library, a membership site — then letting crawlers take it to train models that answer questions without sending anyone to you is a bad deal, and blocking is a rational defense of your asset.

Allowing is a legitimate choice. If you want to be found, quoted, and recommended when someone asks an assistant for help in your field, the crawler has to be able to read the page. You cannot be cited by a system you refused entry.

And it isn't binary. That Content-Signal line is the interesting part of Cloudflare's implementation: ai-train and ai-input are separate dials. You can say "don't train on this" while still allowing an assistant to read the page and answer with it now. Training and citation are different uses, and you're allowed to feel differently about them.

For me the answer was obvious once it was actually a question. I sell AI systems to business owners. If someone asks Claude or ChatGPT how a Connecticut service business should approach AI, I want the answer to be able to include what I've written. That's the whole point of publishing it.

But note the phrase — once it was actually a question. For three weeks it wasn't a question. It was a setting.

If you take one thing

Fetch the file. Don't read it.

Everything you believe about your live site — robots rules, redirects, headers, canonical tags, what scripts load — is a belief about a server, and the only way to check a belief about a server is to ask the server. Your repository can tell you what you intended. It cannot tell you what is true.

If you've never once fetched your own robots.txt over the network, you don't currently know what it says. That took me three weeks to learn on my own website, and I write about this for a living.

← Back to the Learning Log