Cloudflare has introduced a “Disallow AI Training” setting that lets website owners remain visible in search while refusing training use by the same company’s crawler. Apple, Google and Microsoft either support the preference or have committed to honor it within a specified timeframe.
The control addresses mixed-use crawlers, which serve both conventional search and model training. Blocking one of these bots previously risked blocking both purposes. Cloudflare says fewer than 1% of sites on its network block search bots, while 17% use some mechanism to resist AI training, showing why a single allow-or-block choice is too blunt.
Cloudflare argues that a robots.txt instruction alone cannot verify a crawler’s identity or purpose, nor stop a bot that ignores the instruction. Its network-level approach publishes the owner’s preference and can identify and manage crawler traffic. The company is also introducing an “Accountable” designation for operators that follow declared rules.
AI-generated summaries remain a separate issue. Cloudflare says mixed-use crawler operators must offer an opt-out, and it aims by early next year to let customers control how much of their material appears in summaries from one Cloudflare setting.