Cloudflare is giving AI companies a deadline to distinguish crawlers used for search from those used for training and agent products. Publishers using the policy can block non-compliant crawlers by default.
The move gives publishers more leverage in negotiations over how their content is used by AI systems. It also makes crawler identity and purpose a central technical issue for companies building AI search, assistants, and training pipelines.