Cloudflare and OpenAI say a July 2026 research pilot has already made ChatGPT better at finding useful web pages, with early results showing the partnership helped surface hundreds of millions of previously undiscovered pages for OpenAI’s search index. The companies say the goal is not just broader indexing, but better discovery: helping AI systems find timely, specific information, point users to original sources, and avoid wasting site owners’ resources.
Why Cloudflare and OpenAI say discovery matters
The companies argue that optimizing a site for AI is only useful if AI systems can actually find the relevant pages in the first place. That matters most for pages that are highly specific and often valuable even if they are not widely linked elsewhere, such as installation guides, current fees, product manuals, or local business details.
In the past, discoverability mostly meant ranking well in search results and then relying on a person to click through and judge the content. Cloudflare and OpenAI say that as more people use AI for answers and tasks, discovery now has to work for both systems and humans: AI needs to identify useful sources, understand them, and route people toward the right information, whether through a direct answer, a citation, or a visit to the website itself.
What the Cloudflare and OpenAI pilot found
According to Cloudflare, the pilot used a participating cohort representing about 1% of Cloudflare’s total network. With those signals, OpenAI was able to discover and index hundreds of millions of new web pages, expanding the material available to ChatGPT users.
Cloudflare says its network signals helped OpenAI’s search crawler find more useful pages across the web, while OpenAI independently evaluated and ranked those pages based on relevance and usefulness. The companies describe that process as a feedback loop that improves both discovery and the quality of information surfaced to users.
Cloudflare also says ChatGPT Search can link back to relevant sources, giving people a path to the original content and the publishers behind it. The company frames that as a way to help users explore the source material rather than stopping at an AI-generated summary.
How AI discovery is changing web crawling
Cloudflare says the project has also revealed how AI systems can waste effort on pages that do not help users. Automated traffic can make nonexistent pages appear more interesting than they are, and some sites redirect almost any request to a homepage. In other cases, multiple URLs can point to duplicate content.
Recognizing those patterns allows crawlers to avoid unnecessary fetches and cut down on wasted work. Cloudflare says that benefits website owners by reducing load, while also helping OpenAI spend more of its search resources on pages that are more likely to matter.
Signals Cloudflare says can improve crawling efficiency
- Filtering out pages that do not exist or that simply redirect elsewhere
- Using change signals to determine when a page is worth revisiting
- Spotting duplicate content so crawlers do not process the same material repeatedly
- Flagging potentially harmful pages without penalizing an entire domain
Finding harmful pages without taking down an entire site
One of the more practical claims in the post is that the system can identify suspicious individual pages rather than labeling an entire domain as risky. Cloudflare gives the example of a legitimate site that might have a single compromised page, such as a fake login form planted by an attacker.
Flagging only that page, Cloudflare says, could keep users safer without punishing the site owner for a broader site-wide security problem. The company says early results suggest these signals can help identify pages that should stay out of an index.
What this means for website owners
For publishers and businesses, the pitch is straightforward: if AI systems can discover more of the right content, then interested users may find their sites more often. Cloudflare says that as more people use AI to look for information, website owners have a new opportunity to reach audiences who are already looking for answers, products, or services.
At the same time, Cloudflare emphasizes that crawlers should not simply fetch every page more frequently. Doing so would add cost and load without necessarily improving results. Instead, the company says change signals can tell crawlers when content has changed enough to justify another visit.
That balance matters. Discovering a new page can temporarily increase crawling while the page is evaluated and indexed, but the broader goal is to reduce unnecessary work over time while keeping information current.
Cloudflare’s stated benefits for site owners
- Less unnecessary load from repeated crawls of unchanged pages
- Better filtering of pages that do not exist or redirect elsewhere
- More targeted revisits based on change signals
- Potentially better exposure to people searching through ChatGPT
What happens next in the Cloudflare and OpenAI effort
Cloudflare says the next phase will focus on improving page selection, testing faster and more reliable change signals, and measuring what happens after content is discovered. The company says it is continuing to work with OpenAI to understand how better site discovery can translate into business outcomes.
Importantly, Cloudflare says a signal about a page does not grant permission to access it. The company also says the research uses metadata about the web for search and discovery rather than sharing page content for model training.
For site owners who want more control over how AI systems access their content, Cloudflare points to its AI Crawl Control tool. That positions the company’s latest work as both a search-discovery project and part of a broader push to give publishers more visibility into AI crawling.
Source: Original report
Was this helpful?
Explore more: AI Automation Services More AI & Automation Tech News
Last Modified: October 15, 2026 at 10:34 pm
2 views
