Moving a website to a content delivery network, or CDN, can overload the site’s own server at first, Google warned on November 11, 2026. In a post about moving a site to a CDN on its Search Central blog, Google explains why its crawlers hit that server harder after the switch. It also warns against blocking every bot in response: if that block stays in place, pages start dropping out of Google Search.
A CDN is a network of servers that keep copies of a site’s pages close to its visitors. The server that hosts the site itself is called the origin server. Crawlers, such as Google’s Googlebot, are programs that automatically visit websites page by page.
The authors, Cherry Prommawin and Gary Illyes, are analysts on Google’s Search team. They write that site owners sometimes tell them crawler traffic to their server “spikes almost immediately” after the switch.
Google gives two reasons. A CDN responds faster, which encourages Google to crawl more. But its cache, the store of copies it serves from, starts out empty, so the first requests still go back to the origin server.
A faster server gets more visits from Googlebot
Google gives every site a crawl capacity limit, also known as “hostload.” It caps the time a server spends on Google’s connections, based on how many are open and how long they last. Every site starts at the same conservative default.
Google raises that limit when a site responds quickly and consistently. It lowers it when responses slow down or the server returns 5xx errors or a 429 code. The 5xx codes signal a server error, and 429 means too many requests.
Once traffic goes through the CDN, connection and response times drop sharply. “To our crawling systems, it looks like your server just got a massive hardware upgrade,” the authors write. Crawlers then speed up gradually, but only if Google wants to crawl more of the site. Google’s documentation calls this crawl demand.
Each Google crawler has its own demand, but they all share the same capacity limit. When one crawler wants more, less capacity is left for the others. Google’s example is the crawler for Google Shopping, its product search. It has higher demand for the products in a site’s merchant feeds, the catalogs a site sends to Google.
An empty cache sends crawlers back to the origin server
A new CDN starts with an empty, or “cold,” cache. On the first request for a page, the CDN has no copy, which is called a cache miss. The CDN fetches the page from the origin server, keeps a copy and then responds. After that, it serves the page itself.
Human visitors usually cluster around a few pages: the homepage, top articles, current promotions. Those pages land in the cache quickly. Crawlers, by contrast, methodically request every page on the site, so each page nobody has asked for yet comes from the origin server. “Basically, the real bottleneck during a CDN launch is still the origin server,” the authors sum up.
Two things soften the effect, according to Google. Its crawlers don’t ramp up all at once, and many CDNs pace connections on their own to smooth out spikes. Until the cache fills up, though, the origin server remains exposed.
The bot protection that pushes pages out of Google
Right after the switch, a flood of automated requests in the server logs can alarm an administrator who doesn’t know where they come from. The post describes the temptation: switching on blanket bot protection in the CDN’s dashboard. Without exemptions, that protection also catches the crawlers the site wants to let through.
The CDN can then show a CAPTCHA, a test meant to tell humans from bots. A web application firewall, a filter that screens out suspicious requests, can do the same. When that happens, the post says, “Google’s crawlers fail the test.”
The crawlers then get stuck on an error page, and to Google the site looks deliberately restricted or broken. If the rule stays in place, the post warns, Google’s crawlers back off and pages begin to drop out of Search, shopping listings included.
The post also warns against “403 walls,” meaning blocks that return the 403 code, which denies access. Google’s documentation on HTTP status codes says an already indexed page that returns a 4xx error, such as 403, is removed from the index, and so from search results. Except for 429, these codes “have no effect on crawl rate.”
Google has made at least one exception to this. On November 5, Gary Illyes said Google had temporarily kept VLC’s home page in its index despite a 418 error, another 4xx code. Google did so because it considered the VLC media player’s page very important.
Google’s advice: exempt its crawlers and pre-warm the cache
The post makes three recommendations. The first is for sites that need to filter bots: let verified bots through in the CDN and firewall settings. According to Google, most major CDNs and web application firewalls offer a built-in toggle or a ready-made rule for this.
Sites can recognize Google’s crawlers in two ways. One is to check the visitor’s IP address, which identifies a machine online, against Google’s published IP address ranges. The other uses DNS, the internet’s directory that links addresses to domain names. The address must point to a googlebot.com or google.com name, and that name must point back to the same address.
Google says it’s better to target unwanted bots specifically, with the robots.txt file or dedicated rules. That file tells bots what they may crawl, but not all of them obey it, as our article on AI crawlers and robots.txt explains.
The second recommendation is to “pre-warm” the cache before sending all traffic through the CDN. That means having the CDN fetch key pages in advance, at a controlled pace. Google lists the most visited pages, the most crawled ones and pages for an upcoming traffic peak, such as a promotion or a seasonal launch.
The emergency brake slows crawling across the whole site
The third recommendation is an emergency brake for when the origin server stays overloaded while the cache fills. The server, or the CDN’s rate-limiting rules, can return a 429 code, or a 503 code meaning service unavailable. Google says to use it only for “a short window (a few hours, or 1-2 days at most).”
Google’s documentation on reducing the crawl rate spells out the cost of that brake. The slowdown hits every page on the hostname, such as www.example.com, including pages that work fine. Googlebot then discovers fewer new pages. Even without a slowdown, Googlebot takes about 20 hours to discover a new page, according to internal estimates Gary Illyes presented on October 2.
Googlebot also refreshes known pages less often, so changes to product prices or availability can take longer to show up in Search. Campaigns on Google Ads, Google’s advertising platform, may be canceled or paused.
After a day or two, the brake itself becomes a risk. If Googlebot sees 429 or 503 codes on the same page for several days in a row, that page may be dropped from the index.
The post’s limits
The post doesn’t put numbers on the problem. Google doesn’t say how much crawling increases after a switch or how long the cold-cache phase lasts. It doesn’t name the CDNs that let sites exempt verified crawlers.
The post describes how Google’s crawlers already work and announces no new rule. Search Engine Roundtable, a news site that covers search engines, picked it up on November 11. It noted that “most, if not all,” of the post comes from Google’s existing help documents.
A companion video, published the same day, closes the first season of “The Non-Canonical Cabinet,” Google Search Central’s video series.
What it means
The post documents a chain of events that site owners report to Google, according to its authors. The costliest risk comes from how site owners react: blanket bot protection that blocks Googlebot pushes pages out of Search.
The most exposed sites have many pages relative to their human traffic, such as online stores with large catalogs, publishers with long archives and directories. Google’s documentation lists calendars and faceted navigation, the filters and sort options on a catalog, among the most common causes of a crawling spike.
Small sites are less exposed, with fewer pages to fetch from the origin server. Google’s crawl budget guide, which covers the capacity limit, is aimed mainly at sites with more than a million pages that change every week. It also includes sites with more than 10,000 pages that change every day. Google calls both thresholds rough estimates.
Telling the real Googlebot from fakes remains a weak spot. PPC Land, a site that covers online advertising, reported a comment Gary Illyes made in a podcast published on October 28. Setting up a script that crawls while posing as Googlebot would take him “probably 30 seconds,” he said.
According to PPC Land, Google already signs some requests from Google-Agent, its AI agent, as an experiment, with a digital signature that lets sites verify where they come from. Googlebot’s requests aren’t signed, so for now only its IP addresses and DNS can identify it.
The timing gives the post extra weight. Black Friday, the big day for retail promotions, falls on November 27, 2026. Google lists promotional pages among the pages to pre-warm.
Sources
- Moving to a CDN without overwhelming your origin (or blocking Googlebot), Google Search Central, November 11, 2026
- Moved to a CDN: Why Is Everything Crawling Everywhere All at Once?, Google Search Central (YouTube), November 11, 2026
- Optimize your crawl budget, Google, accessed November 12, 2026
- Reduce the Google crawl rate, Google, accessed November 12, 2026
- Verify requests from Google crawlers and fetchers, Google, accessed November 12, 2026
- How HTTP status codes affect Google’s crawlers, Google, accessed November 12, 2026
- Google: How Moving to a CDN Can Accidentally Overwhelm Your Website, Search Engine Roundtable, November 11, 2026
- Faking Googlebot takes 30 seconds, Google’s Gary Illyes says, PPC Land, October 31, 2026
Header image: MACCUS diagram, based on Google’s Search Central blog (November 11, 2026)
