I build AI-visibility audits for other brands. This week I finally pointed one at my own domain, citable.in. The very first thing it flagged was the most ironic problem an advisory like mine could have: my own site was telling the AI crawlers to stay out.
My robots.txt is deliberate. It explicitly welcomes GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, because being citable to answer engines is the entire premise of what I do. I wrote that file by hand. I was proud of it.
The problem: the file my domain actually served was not the file I wrote.
What was actually happening
Cloudflare has a managed setting that writes AI-crawler rules into your robots.txt for you, at the edge, before the response ever leaves their network. Mine was switched on. It had prepended a block above my own rules that said Disallow: / for GPTBot, ClaudeBot, Google-Extended, and CCBot, among others.
So the served file carried two contradictory instructions for the same crawlers: Cloudflare's block saying stay out, and my block, lower down, saying come in. When a robots.txt hands a crawler two conflicting groups, which one wins is up to the crawler. That is not a coin toss I want running on the one file that decides whether ChatGPT and Gemini can read my site.
I never saw it, for a boring and dangerous reason: I only ever read the version in my repository. That file was perfect. The served file, the only one the crawlers actually see, was not.
The fix, and the lesson
The fix took one toggle. I turned the managed AI-crawler setting off in the Cloudflare dashboard, and my own robots.txt started serving verbatim. No redeploy, no code change, because the block never lived in my code. It lived at the edge.
The lesson is smaller than the irony and more useful: audit the robots.txt your domain actually serves, not the one in your repository. Open an incognito tab, fetch yourdomain.com/robots.txt, and read every line. If you are on Cloudflare, check whether its managed robots.txt or AI-crawler controls are on, because they can quietly overrule the file you wrote. A platform default can undo a deliberate decision, and it will not tell you it did.
None of the rest of an AI-visibility strategy matters if the crawlers can't reach the page. Schema, llms.txt, citable copy: all of it assumes the model can get in. Mine couldn't, cleanly, and I would not have known if I hadn't run my own audit.
That is the uncomfortable, useful thing about being your own first client. This is the first note in a short series I'm calling Client Zero: what happens when I run Citable's own audit, and Citable's own playbook, on Citable. The next one is about a gap that had nothing to do with code, and everything to do with nerve.