Cloudflare and OpenAI say a research pilot has brought hundreds of millions of previously unknown web pages into OpenAI’s search index. The project uses signals from Cloudflare’s network to identify pages that may be useful to ChatGPT users, including specific manuals, current requirements and local information that attract few inbound links.
The pilot began in July and covers a participating group representing about 1% of Cloudflare’s network. Cloudflare supplies discovery signals, while OpenAI independently decides how pages are evaluated and ranked. ChatGPT Search can then cite or link to an original source.
The same signals can reduce wasted crawling. Repeated requests for nonexistent files, redirects that lead back to a homepage and duplicate pages can be filtered before a fetch. Change signals can indicate when a crawler should revisit a page instead of repeatedly downloading material that has not changed. The companies are also testing page-level warnings for potentially harmful content, such as a fake login page planted on an otherwise legitimate site.
Cloudflare stresses that a discovery signal is not permission to access content. Site owners retain crawler controls through AI Crawl Control, and the pilot shares web metadata for search discovery rather than page content for model training.