AI crawlers and robots.txt: which ones follow your rules, which ones ignore them
On October 9, 2026, a test shot to the top of an SEO subreddit: its author had asked several AI assistants to open secret pages, some of them blocked in robots.txt. Claude read the file and refused the blocked page, ChatGPT opened it on its first visit, and Grok, xAI’s assistant, showed up from 59 … Read more