On November 17, 2026, Google published a guide that helps websites keep their security filters from blocking its AI agents. Titled “Allow Google-Agent traffic,” it explains how to recognize these agents and let them in. Google-Agent is the name the agents use when they browse a site and take actions on it at a user’s request. Google doesn’t say which products use them or how many visits they account for.
The guide covers three kinds of tools that sit between visitors and a website. A CDN (content delivery network) is a network of servers that keep copies of a site’s pages close to its visitors. A web application firewall (WAF) screens the requests that browsers and bots send to a site before they reach it. A bot-detection service spots and blocks automated visits.
A new page in Google’s crawler documentation
The guide is part of the technical documentation Google keeps for its crawlers, the programs that visit web pages automatically. The changelog, Google’s running record of changes to those pages, dates the guide November 17, 2026. It also states the guide’s purpose: “to help publishers that want to enable Google-Agent for their website.”
Barry Schwartz flagged the guide the same morning on Search Engine Roundtable, a site that covers search engines. Dr. Web, a German site, covered it that day. Abondance, a French site, followed on November 19.
The guide has three parts: how Google-Agent identifies itself, how to allow it, and what to check if it’s still blocked. It came six days after a November 11 blog post in which Google explained how to move a site to a CDN without overloading its server or blocking Googlebot. Googlebot is the crawler that scans the web for Google Search.
A bot that acts for a user
Google-Agent isn’t new. On March 20, 2026, Google added it to its list of “user-triggered fetchers.” These are bots that visit a page because someone asked for it inside a Google product. At the time, the changelog said the agent would roll out “over the next few weeks.”
Search Engine Roundtable reproduced the page’s wording on March 23. According to that report, the page then gave one example: Project Mariner, a Google prototype agent that carries out tasks in a web browser. The current entry for Google-Agent no longer names any product.

Google-Agent differs from Googlebot in what its visits are for. Googlebot reads pages so Google can list them in Search results. Google-Agent uses pages to act on someone’s behalf, like other AI agents that browse websites. The guide’s troubleshooting list makes that clear: it mentions on-site search, filtering, detail pages, add-to-cart and booking.
Robots.txt, the file that tells bots which pages they may visit, isn’t enough to control these agents. Google says user-triggered fetchers “generally ignore robots.txt rules” because a person requested the fetch. Our article on AI crawlers and robots.txt covers how several AI companies apply that exception.
The November 17 guide doesn’t mention robots.txt at all. For Google-Agent, sites control access through their security tools instead.
Three signals that identify the agent
The first signal is the User-Agent header, a label in each request that names the software sending it. Google-Agent’s header has a mobile and a desktop version, and both contain the string “Google-Agent.” Google’s list of user-triggered fetchers warns that “the user agent string can be spoofed.” Any bot can claim that name.
The second signal is the IP address, the number that identifies a machine on the internet. Google-Agent requests come from the address ranges listed in a public file, user-triggered-agents.json.
The copy of the file we downloaded on November 20 was generated on November 19 and listed 20 address ranges. Twelve use IPv4, the older address format, and eight use IPv6, the newer one. The guide asks sites to keep their filtering rules in sync with this file.
The third signal, still experimental, is a digital signature: a code attached to a request that proves who sent it. Google is testing a protocol called Web Bot Auth for this. The protocol is being drafted at the IETF, the body that sets the internet’s technical standards.
Google-Agent signs some requests using RFC 9421, the IETF standard for signing messages sent over the web. A signed request carries a Signature-Agent header that points to the agent’s identity, https://agent.bot.goog. To check the signature, the site uses the public keys Google publishes for that identity. On November 20, the key directory listed five keys.
Google notes that not every request is signed. It asks sites to keep checking IP addresses as well.
Google’s proposed rule and six reasons it can fail
The rule Google suggests comes down to three checks. The site confirms that the User-Agent header contains “Google-Agent” and that the IP address is in Google’s file. It also verifies the signature when the request carries one. A request that passes these checks should skip bot challenges, the tests sites use to screen out bots.
According to Google, some CDNs, firewalls and bot-detection services can already verify Google-Agent. Google names none of them and tells sites to ask their provider for the exact steps.
The guide’s troubleshooting section lists six reasons the agent can stay blocked even with this rule in place. Two of them concern the rule itself. First, the rule has to take priority over broad blocks. These include blocks on entire cloud and data center networks, the bot rules the provider manages and policies that block AI agents as a category.
Second, the rule has to exempt the agent from two kinds of bot challenges. A CAPTCHA asks the visitor to prove they’re human. A JavaScript challenge makes the browser run a small program. The exemption also covers CAPTCHA widgets on the page that appear on a search or a form submission.
The other four reasons concern the agent’s path through the site and the plumbing behind it. Third, the rule has to cover actions beyond the home page, such as on-site search, filtering, detail pages, the cart and booking. It also has to cover the technical calls that power those actions, such as APIs, the interfaces programs use to exchange data.
Fourth, per-IP rate limits need to account for Google-Agent. These limits cap how many requests a single IP address can send. Several of the agent’s users can share the same IP addresses, so their requests count toward the same cap.
Fifth, sites have to keep their rules current as Google updates the address file. Sixth, load balancers and proxies have to pass on the agent’s signals: the original IP address, the User-Agent header and the signature headers. A load balancer spreads traffic across several servers, and a proxy is an intermediate server that relays requests.
What the guide leaves out
The guide gives no traffic numbers. It doesn’t say how many requests Google-Agent sends, what share of them are signed, or which products use the agent.
Abondance points out the same gaps and adds that the guide offers no way to block the agent. Dr. Web notes that Google doesn’t say how often the agent visits German sites. The changelog presents the page as help for publishers who want to let Google-Agent in.
The guide says that allowing the agent “helps prevent legitimate requests from being blocked while preserving your existing security controls.” But it also asks that per-IP rate limits account for Google’s address ranges or exempt them. It doesn’t describe how Google itself caps the volume of requests from its agents. Nor does it say whether these visits show up in analytics tools, or whether blocking them affects search rankings.
What it means
For search crawlers such as Googlebot, sites keep control through robots.txt. For agents that take actions, Google points to security tools instead: CDNs, firewalls and bot-detection services.
Google published its Web Bot Auth page on May 4 and updated it in October. That page already explained how to verify Google’s AI agents and add them to an allowlist, a list of bots a site lets through. Compared with that page, the November 17 guide adds the User-Agent check, the IP address list and six reasons the agent can stay blocked.
Going by the steps Google lists, from the cart to form submissions, the sites most affected are those where an agent has a task to finish. That points to online stores, booking sites and sites with request forms. On these sites, a block on data center networks alone can stop the agent, even when an allow rule exists. That’s the first case Google’s troubleshooting list mentions.
What Google has shown is limited: a file of 20 address ranges, five public keys and a list of rules. A site that opens its filters accepts traffic whose volume and value Google hasn’t disclosed.
Several things are worth watching: which Google products run through Google-Agent, the share of signed requests and Web Bot Auth’s progress at the IETF. Another is whether CDN and bot-detection providers add the agent to their lists of verified bots, and how. One question remains open: whether Google will ever publish numbers on these visits.
Sources
- Allow Google-Agent traffic, Google, crawling documentation, November 17, 2026
- Changelog (crawling documentation), Google, crawling documentation, November 17, 2026
- List of Google user-triggered fetchers, Google, crawling documentation, August 19, 2026
- Authenticate requests with Web Bot Auth (experimental), Google, crawling documentation, October 13, 2026
- New Google Search Help Document On How To Allow Google-Agent Traffic, Search Engine Roundtable, November 17, 2026
- Google Adds Google-Agent User Agent, Search Engine Roundtable, March 23, 2026
- Agents IA : Google demande aux sites d’ouvrir leurs CDN et pare-feu à Google-Agent, Abondance, November 19, 2026
- Google beschreibt die Firewall-Freigabe für den Google-Agent, Dr. Web, November 17, 2026
Header image: MACCUS diagram based on Google’s “Allow Google-Agent traffic” guide, November 17, 2026



