AI crawlers and robots.txt: which ones follow your rules, which ones ignore them

On October 9, 2026, a test shot to the top of an SEO subreddit: its author had asked several AI assistants to open secret pages, some of them blocked in robots.txt. Claude read the file and refused the blocked page, ChatGPT opened it on its first visit, and Grok, xAI’s assistant, showed up from 59 IP addresses while posing as an ordinary browser. OpenAI, Google and Perplexity say as much in their own documentation: the bots that act on a user’s request may ignore robots.txt. Here’s what the file still lets you control, and how to protect what it can’t.

What the Reddit test found

Each assistant got its own secret URL that nothing linked to, with a code on the page for it to read. Any visit to that URL could only have come from that assistant.

According to the author, Claude checked robots.txt before making any other request, then refused the blocked page. ChatGPT went straight to the blocked page on its first visit. About 76 seconds later, OpenAI’s search crawler fetched robots.txt, and the next request was refused without a single fetch. Gemini never read robots.txt: it identified itself only as “Google,” from addresses that aren’t on any of Google’s published lists.

Grok sent each request from 9 to 12 different IP addresses, owned by hosting companies and small ISPs in Brazil, Mexico and Poland, and posed as Chrome or Safari every time. That’s 59 addresses for 6 requests, and not a single look at robots.txt. The author is upfront about the limits: a few runs per assistant, on a single day.

What the AI companies themselves say

The bots AI companies run fall into three families: those that collect pages to train models, those that feed an AI search engine, and those that fetch a page because a user asked for it.

OpenAI’s documentation for ChatGPT-User says that “because these actions are initiated by a user, robots.txt rules may not apply.” Google and Perplexity use nearly identical wording for their user-triggered fetchers, including Google-Agent: they “generally ignore” those rules. An agent that visits a page can also act on it: on October 9, 2026, Anthropic disclosed that one of its models had sent a fake tip through a police website’s form, as we cover in our article on Anthropic’s report.

Anthropic’s help page about its bots states that all three of its bots honor robots.txt directives, and that disallowing Claude-User prevents its system from retrieving your content in response to a user’s query. xAI’s documentation for Grok Bot, its agent (not the same as the Grok assistant in the test), mentions no user agent, no IP ranges and no robots.txt (as of October 10, 2026). On the robots.txt question alone, the Reddit results line up with what OpenAI, Google and Anthropic say.

What other tests show

Andre Alpar ran a similar test on 12 assistants and published it on Search Engine World on June 20, 2026. Only three stayed away from the blocked page: ChatGPT, Claude and Perplexity. Gemini, Meta AI, Mistral, Grok, DeepSeek, Qwen, ERNIE and Kimi opened it. Copilot outsourced the fetch to a third-party company, Diffbot.

In that test, a single request to Grok drew visits from 17 IP addresses in countries including France, Italy and Brazil, all using a fake Mac browser user agent. Alpar concludes that robots.txt amounts to a polite request to well-behaved bots and can’t control access.

Back on August 4, 2025, Cloudflare accused Perplexity of something similar: when its declared crawler was blocked, Perplexity used a browser impersonating Chrome on a Mac, from addresses outside its official IP ranges, at 3 to 6 million requests a day. Behavior changes, though: in the June 2026 test, Perplexity followed the rule.

The EU framework: what the signatories promised

The EU AI Act’s rules for general-purpose AI models took effect on August 2, 2025. The final version of the Code of Practice meant to help companies comply came out on July 10, 2025.

Measure 1.3, in the copyright chapter, applies to signatories that collect pages, directly or through a third party, for text and data mining and for training their models. They commit to using crawlers that “read and follow” robots.txt instructions as defined in the RFC 9309 standard, and to publishing information about their crawlers. The text points out that EU Directive 2019/790 lets rightsholders reserve their rights against such mining, for example through machine-readable means.

The Commission’s signatory list (last updated October 7, 2026) includes Amazon, Anthropic, Google, Microsoft, Mistral AI and OpenAI. Meta and Perplexity aren’t on it. xAI signed only the Safety and Security chapter: the Commission says the company will have to demonstrate compliance with its transparency and copyright obligations through alternative adequate means. The code is silent on user-triggered visits.

What this means for your site

Disallowing training bots sends a signal the major signatories have committed to honor. For OpenAI, a GPTBot disallow indicates that your content should not be used to train its models. Anthropic says blocking ClaudeBot signals that your future content should be excluded from its training data.

Shutting out search bots, on the other hand, has a cost. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers, and Perplexity recommends allowing PerplexityBot if you want to appear in its results. For an online store or a freelancer, those answers are a source of visibility, as we explained in our piece on local SEO in the age of AI.

Google says the Google-Extended token has no effect on inclusion or ranking in Google Search. AI Overviews and AI Mode fall under Googlebot, so blocking it would mean blocking Google Search as well. To limit what Google pulls from your pages, its documentation on AI features points to the nosnippet, data-nosnippet, max-snippet and noindex controls.

What to do now

1. Open yoursite.com/robots.txt and list the groups that target AI bots. Make sure a plugin or your host hasn’t added any without telling you. Our guide to technical SEO covers what this file does.
2. Decide family by family, using the table above: training, search and user-requested visits. Google-Extended also covers grounding for Gemini’s answers.
3. Write one group per bot. For both OpenAI and Perplexity, each setting works independently, and changes can take up to about 24 hours to apply. Anthropic asks you to repeat the rules for every subdomain.
4. Check your server logs. A bot’s user agent takes one line to fake: compare the addresses against the lists published by Google, OpenAI, Anthropic and Perplexity, or, for Google, run a reverse DNS lookup.
5. Put anything that must stay private behind a password. The RFC 9309 standard says it plainly: these rules “are not a form of access authorization,” and listing a path in robots.txt makes it public. To keep a page out of Google, use the noindex tag, as our guide to indexing explains.

Here’s a sample for a site that opts out of training (and Gemini grounding) but wants to stay visible in AI search engines:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

What to watch in the coming months

Google’s documentation for Google-Agent says Google is experimenting with the Web Bot Auth protocol, which an IETF working group is standardizing. The protocol lets a bot cryptographically sign its requests. If AI companies adopt it, verifying that a visitor is the bot it claims to be will be simpler than checking lists of addresses.

As for xAI, watch for two signals: a public page describing its bots, and a signature on the copyright chapter of the EU code.

What’s still uncertain

Both tests are small: each covers one site and a handful of runs, and captures a single moment. An assistant that follows the rule today can change its methods tomorrow. Nobody can check from the outside whether a blocked page ended up in training data. Anthropic’s wording covers your “future” content: its page says nothing about material already collected.

For disguised traffic, practitioners on Reddit report that behavioral filtering works better than IP or user-agent lists. When requests come through consumer ISPs, blocking those addresses also blocks real visitors.

Want to know what your robots.txt actually blocks and what your server logs say about AI bots? Request our free SEO audit.