Signals from Cloudflare’s network helped OpenAI’s search crawler discover and index “hundreds of millions of new web pages,” according to a post published on Cloudflare’s blog on October 15, 2026. These are the early results of a research pilot the two companies have run since July. ChatGPT can now draw on those pages when it answers search queries.
The post was written by two Cloudflare employees, Matthew Conroy and Jack Galilee, and two OpenAI employees, Henry Aspegren and James Balfour. The pilot was limited to a cohort of participating sites, about 1% of Cloudflare’s total network. The authors call the results preliminary and say another phase is planned.
A pilot announced on July 8
Cloudflare announced the program on July 8, 2026, in a press release. The goal was to test whether insights from “participating websites” on its network could help AI search engines discover and index content on the open web more effectively. The release gave three examples of signals: content freshness, traffic quality, and actual page changes.
Cloudflare noted that more than 20% of the web sits behind its network, and described OpenAI’s contribution: frontier models, a large-scale search and answer system, and real user queries for testing. “Up-to-date information is important for delivering accurate answers to people using ChatGPT,” said Nick Ryder, VP of Research at OpenAI. Matthew Prince, Cloudflare’s co-founder and CEO, spoke of finding “a better way to make AI search more efficient.”
Three uses for the network signals
The October 15 post starts with discovery: pointing OpenAI’s crawler to pages it had never seen. The authors cite the manual for a specific washing machine model, the current requirements and fees for a passport application, and a restaurant’s latest menu. Such pages “can be valuable even if few other sites link to them,” they write.
Next comes filtering, before the crawler even sends a request. According to the post, automated traffic can make a page that doesn’t exist look interesting, because bots keep asking for files that aren’t there. Some sites also redirect almost any address to their homepage, and other addresses lead to copies of the same content. Spotting those patterns saves the crawler wasted visits.
The third use is change signals, which tell the crawler when a page is worth another visit. The authors warn, though, that finding new pages can increase crawling at first, while those pages are evaluated. They are also testing signals that can spot a compromised page on a legitimate site, such as a fake login form planted by an attacker, so that only that page gets flagged rather than the whole domain.
Ranking stays with OpenAI. Per the post, the company then independently evaluates and ranks the pages based on how relevant and helpful they are.
Shared data and site settings
The post says the research uses metadata about the web for search and discovery, “rather than sharing page content for model training.” It adds that a signal about a page doesn’t grant permission to access it, and that domain owners’ choices matter throughout.
In its crawler documentation, OpenAI distinguishes OAI-SearchBot, which surfaces sites in ChatGPT’s search features, from GPTBot, which crawls content that may be used to train its models. A site that blocks OAI-SearchBot won’t appear in ChatGPT Search answers, though it can still show up as a navigational link.
According to OpenAI, each setting is independent of the others. The company also says that, for search results, it can take about 24 hours after a robots.txt update for its systems to adjust. Our article on AI crawlers and robots.txt covers what each of these bots does.
Cloudflare points site owners to AI Crawl Control, available on all its plans. The tool shows which AI services visit a site, lets owners allow or block each crawler, and tracks robots.txt compliance. Its pay per crawl feature is still in private beta.
What the post leaves out
The post doesn’t say how the 1% cohort was chosen, whether the sites in it agreed to take part, or which countries and languages they represent. Back in July, Luis Rijo noted in PPC Land that the release didn’t name the participating sites, say how many there were, or describe any safeguards for the shared data. He also wrote that nothing in it indicated publishers were paid.
The “hundreds of millions” figure is a rough order of magnitude. The post doesn’t give an exact total, the size of OpenAI’s index before the pilot, or the share of those pages actually cited in answers. No figures have been published on the visits sent to those sites.
The post says Cloudflare is continuing to work with OpenAI to understand how better site discovery can translate into real business outcomes. On October 16, William Chan wrote in Tech Times that the traffic question remains open.
Measuring those visits isn’t easy. From the evening of October 8 until October 15 or 16, ChatGPT opened a large share of its links without a tracking parameter or a referrer. Google Analytics 4 then counted those visits as Direct, as we reported in our article on the ChatGPT traffic bug.
Tech Times also says a site that opts out through AI Crawl Control contributes no signals to the pilot. Cloudflare’s post doesn’t say that. Nor does it detail how ChatGPT ranks pages, beyond relevance and helpfulness. On Reddit’s r/SEO forum, one poster read the post as a sign that ChatGPT’s index will lean on links less than Google’s does, a reading neither Cloudflare nor OpenAI has confirmed.
Cloudflare changes course, says The Next Web
In July, Ana Maria Constantin had already pointed out the contrast in The Next Web. Cloudflare had spent a year arming websites against AI crawlers, she wrote, and the year before had let sites block those crawlers unless AI firms paid publishers. With this pilot, the company is helping an AI engine crawl the web better.
The October post mentions only OpenAI. Google, Perplexity and Anthropic don’t appear in it. Google still crawls the web with its own bots, following the links it finds and the sitemaps sites submit to it. We covered Google’s indexing times on October 11.
What it means
The pilot adds a discovery route that doesn’t depend on inbound links: signals collected by the infrastructure provider a site’s traffic passes through. For a page to be cited by an answer engine, it first has to be in that engine’s index. That alone isn’t enough, since OpenAI keeps control of ranking.
As the post describes it, the pilot covered only its cohort of participating sites, about 1% of Cloudflare’s network, and neither Cloudflare nor OpenAI has published the list. For every other site, Cloudflare customer or not, allowing OAI-SearchBot in robots.txt remains the setting OpenAI documents for showing up in ChatGPT Search.
The results come from the two partners, with no independent measurement. The post’s only quantified result is an order of magnitude. There is no number yet for the traffic benefit to sites, and the authors themselves defer that to the next phase.
In this pilot, a provider that sits in front of more than a fifth of the web shares what it sees with a single AI engine. The post doesn’t say whether other engines will get access.
According to the post, the next phase will improve page selection, test faster and more reliable change signals, and measure what happens once content is discovered. The open questions are whether the pilot expands beyond 1% of the network and whether Cloudflare customers get a setting to opt in or out. For publishers, the figure to watch is how much traffic ChatGPT sends to pages discovered this way.
Sources
- Cloudflare and OpenAI: making AI discovery better for everyone, Cloudflare, October 15, 2026
- Cloudflare Announces Research Pilot with OpenAI, Cloudflare, July 8, 2026
- Overview of OpenAI Crawlers, OpenAI, accessed October 19, 2026
- AI Crawl Control, Cloudflare, updated August 14, 2026
- Cloudflare gives OpenAI network signals covering 20% of the web, PPC Land, July 12, 2026
- ChatGPT Search Gained Hundreds of Millions of Pages Link Crawlers Never Found, Tech Times, October 16, 2026
- Cloudflare spent a year blocking AI crawlers. Now it’s helping OpenAI index the web, The Next Web, July 8, 2026
Header image: MACCUS chart, data from Cloudflare (October 15, 2026 blog post and July 8, 2026 press release)
