I Found My Own Website Blocking ChatGPT. Then I Checked 640 Others.
In August I ran my agency's AI search checklist against our own site and got a result I didn't believe. Every request that named an AI crawler came back 403. GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, and PerplexityBot were all refused, including on llms.txt, a file that exists only for those bots to read. Googlebot got a 200, and so did plain curl.
Our robots.txt allowed every one of those crawlers by name. That didn't help, because the block sat in front of it. Cloudflare has a "Block AI bots" setting under Security > Bots that matches on user-agent before a request ever reaches the server, and it never reads robots.txt. I don't know whether it was on by default or got switched on while we hardened the zone after a DDoS attack in July. Nobody on the team remembers choosing it.
Setting it to Allow took two minutes. Two weeks later...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE