Cloudflare added new AI traffic controls that let website owners manage crawler access by purpose. Instead of a single AI-bot switch, customers can distinguish Search, Agent, and Training crawlers.
The company says the options are available to all customers, including Free tier users. Cloudflare also described new defaults scheduled for September 15, 2026, when Training and Agent crawlers will be blocked by default on pages that display ads for new domains onboarding to Cloudflare, while Search remains allowed.
WHY THE TAXONOMY MATTERS
Publishers and site owners do not view every crawler the same way. Search crawling can send referral traffic. Agent crawling may perform tasks on behalf of users. Training crawling may use content to improve models without sending direct visitors back.
Separating those purposes gives owners a more realistic policy choice. A site may want to be discoverable in search while blocking training use for paywalled, ad-supported, or original reporting pages.
WHO SHOULD ACT
Publishers, affiliate sites, SaaS documentation teams, ecommerce guides, and data providers should review settings before the September default date. Sites with advertising or subscriptions should be especially careful because crawler access affects both discovery and monetization.
Teams should also review robots policies, analytics, and contractual content licensing. Dashboard controls are useful, but they should align with the broader content strategy.
OPEN QUESTIONS
The biggest unresolved issue is crawler transparency. Controls work best when bot operators label intent clearly and separate crawlers by purpose. Cloudflare is pushing that direction, but site owners still need monitoring to see how traffic actually behaves.
Another question is whether blocking agent crawlers reduces useful user-driven discovery. Some sites may want AI agents to access public pages when acting for a person, but not when collecting content for training. That boundary will remain hard to enforce without ecosystem cooperation.
BOTTOM LINE
Cloudflare is giving site owners a more precise AI traffic policy. The value is not simply blocking bots; it is separating discoverability, automation, and training so content owners can make different choices for different kinds of pages.
