ByThe Cloudflare AI crawler block took effect on September 15, 2026, automatically blocking “mixed-use” bots — crawlers that blend search indexing with AI training or agent data collection — from any ad-supported website on Cloudflare’s network, unless the site owner opts back in. If you run a WordPress site behind Cloudflare, this changed your traffic overnight, whether you noticed it or not.
This guide breaks down exactly what the Cloudflare AI crawler block does, who it affects, why Cloudflare built it, and the specific settings you need to check today so you don’t accidentally cut off the AI answer engines that increasingly send readers to your articles, or hand over training rights you meant to keep.
What Is the Cloudflare AI Crawler Block?
The Cloudflare AI crawler block is a default network setting that stops “mixed-use” crawlers from accessing any page that carries advertising on a Cloudflare-protected site. Cloudflare announced the policy on its blog under the banner “Content Independence Day” in July 2026, then flipped the default to “on” for a defined group of customers starting September 15, 2026.
According to Cloudflare’s own analysis of its network traffic, published on its blog in July 2026, more than a third of the automated requests hitting its edge network could not be cleanly labeled as “search,” “training,” or “agent” activity. A single bot might index your page for a search engine one moment and feed the same page into a model’s training set the next, with no way for your server logs to tell the difference. The new default forces that bot to declare its purpose or lose access to ad-monetized pages.
The mechanism runs through a tool Cloudflare calls AI Crawl Control, the renamed and expanded successor to what the company previously shipped as AI Audit. It sits in front of every request and checks the crawler’s declared intent against the page’s monetization status before deciding whether to let the request through.
Why Did Cloudflare Roll This Out Now?
Publisher complaints drove the timeline. Through 2025 and into 2026, news sites, blogs, and documentation publishers reported AI companies scraping their content for model training at a scale that dwarfed the traffic they got in return, while referral clicks from traditional search kept declining as AI answer boxes absorbed queries that used to land on a publisher’s own page.
Cloudflare’s response was to sit in the middle as an enforcement layer rather than leave the decision to a voluntary text file. The company frames the mixed-use category specifically as a loophole: a crawler that claims to be indexing for search, while actually also harvesting the page for training, was previously invisible to a site owner who only had an allow/block switch for “AI bots” as one undifferentiated group. Forcing crawlers to self-declare closes that gap, at least for bots that comply.
Which Websites Does the September 15 Deadline Affect?
The rollout targets three groups: every new Cloudflare customer who signs up from now on, every new site added by an existing customer, and — importantly — all existing customers on Cloudflare’s free plan. If your WordPress site sits behind Cloudflare’s free tier, the block is already live on your account whether you clicked anything or not.
Paid Cloudflare plans keep their prior crawler settings untouched for now, so nothing changes automatically if you’re on a Pro, Business, or Enterprise plan. Every site owner, regardless of plan, can still open the dashboard and adjust the policy manually — the default simply decides what happens if you never touch the setting yourself.
One detail trips people up: the block only applies to pages Cloudflare’s system detects as carrying ads. A documentation page with no ad units, for example, is not covered by the default and continues to allow the same crawlers it always did unless you change that separately.
What Counts as a “Mixed-Use” AI Crawler?
A mixed-use crawler is any bot that combines two or more purposes — typically search indexing plus AI model training, or search plus live retrieval for an AI agent answering a user’s question — inside a single user agent string. Because the bot never separates those purposes, a site owner previously had no way to allow one and block the other.
Cloudflare’s fix asks every crawler operator to self-declare intent through a machine-readable signal on each request: training, search indexing, or agentic retrieval. Crawlers that refuse to declare, or that try to claim all three at once without separating them, fall into the mixed-use bucket and get the default block on ad-supported pages.
In practice, this pushes large AI companies toward running separate, clearly labeled crawlers for each purpose instead of one general-purpose bot. A company that wants to keep indexing pages for its search product while also training a model needs two distinct crawlers now, each declaring one job, or it risks losing access to a large share of the ad-supported web in one move.
How Does the Cloudflare AI Crawler Block Change SEO and AI Visibility?
Standard search engine bots — Googlebot, Bingbot, and similar single-purpose crawlers — are not touched by this policy, so classic organic search rankings are not at risk from the default change itself. The real shift is on the AI side: if the crawler behind a chatbot’s live answers is flagged mixed-use and you never adjust the setting, that assistant may stop citing your pages the next time it tries to fetch them.
That matters because a growing share of referral traffic now comes from AI answer engines rather than a traditional results page. Site owners who already track this shift with a tool like Search Console’s AI Visibility report should treat September 15 as a checkpoint: pull your AI referral numbers from before and after the date, and if they dropped, the crawler settings are the first place to look.
The broader discipline here is the same one behind generative engine optimization for WordPress: getting cited by AI systems now depends as much on crawler permissions as it does on content quality. A perfectly optimized article that a mixed-use bot can’t reach never makes it into an AI answer at all, no matter how well it targets a query.
What Is Cloudflare’s Pay Per Use Content Model?
Alongside the block, Cloudflare is expanding its existing Pay Per Crawl marketplace into what it calls Pay Per Use. Instead of a binary allow-or-deny choice, a site owner can set a price for specific crawler categories: an AI company that wants to keep training on your content, or an agent platform that wants to keep quoting your pages, can pay per request instead of being blocked outright.
This is opt-in and does not activate on its own. Cloudflare’s stated reasoning, reported by TechCrunch in its July 2026 coverage of the policy, is that most publishers want to stay discoverable in search while no longer giving away training data for free — Pay Per Use is the middle option between a full block and unrestricted access, and it lets a small WordPress blog negotiate on the same terms as a large news publisher, at least in principle.
Pricing is set per category rather than per page, so a site owner picks a rate for “training” access and a separate rate for “agentic retrieval” access, then leaves the marketplace to handle billing whenever a crawler that agrees to pay comes through.
How Does This Compare to Other Hosts and CDNs?
Cloudflare is the largest player to make mixed-use blocking the default rather than an opt-in toggle, but it isn’t acting alone. Other CDNs and hosting-level firewalls have shipped similar AI-bot management features over the past year, typically as manual allow lists rather than an automatic default. The difference matters for a WordPress site owner: on most other platforms, nothing changes until you go looking for the setting, while on Cloudflare’s free plan the change already happened for you on September 15.
If your WordPress site runs on shared hosting — Hostinger, Bluehost, SiteGround, and similar providers all commonly route traffic through Cloudflare by default — you’re almost certainly covered by this policy even if you never configured Cloudflare yourself. It’s worth logging into whichever dashboard gives you access to the Cloudflare zone for your domain, even if that’s through your host’s control panel rather than a direct Cloudflare account, just to confirm what’s actually been applied.
How Do Cloudflare’s AI Crawler Categories Compare?
Cloudflare now sorts bot traffic into distinct categories rather than one generic “AI bot” bucket, and each category gets its own default treatment. The table below shows how each is handled under the September 15 default on a free-plan, ad-supported site, so you can see at a glance which categories need your attention.
| Crawler Category | Example Use | Default Status (Free Plan, Ad Pages) |
|---|---|---|
| Search indexing (single-purpose) | Googlebot, Bingbot | Allowed |
| AI training (single-purpose) | Model pretraining crawls | Blocked by default |
| Agentic retrieval (single-purpose) | Live AI assistant fetching a page to answer a question | Allowed, adjustable |
| Mixed-use (undeclared or combined) | Bots blending search, training, and agent use | Blocked by default |
| Pay Per Use (opted in) | Any category the site owner prices individually | Allowed at the price you set |
How Do You Check and Adjust Your Cloudflare AI Crawler Settings?
Log in to the Cloudflare dashboard and open the domain for your WordPress site. From there, go to Security, then Bots, then AI Crawl Control — this is the renamed version of the tool Cloudflare previously shipped as AI Audit. The panel lists every crawler category Cloudflare recognizes along with its current status on your site.
- Open Security > Bots > AI Crawl Control for your domain.
- Check the status shown for each category: search, training, agentic retrieval, and mixed-use.
- Decide category by category — don’t just flip a single global switch and move on.
- Allow agentic retrieval explicitly if you want AI assistants to keep citing your content.
- Leave training blocked if you don’t want your articles used to train models.
- Save, then confirm the change against your traffic logs a few days later.
Review each category individually rather than accepting the blanket default. If a specific AI company’s crawler is mixed-use and important to your traffic, either wait for it to update its declaration or add it to your allow list manually while you watch for abuse.
Save your changes and check back after a few days using your server logs or Cloudflare’s analytics tab to confirm the crawlers you allowed are actually returning. Settings take effect immediately, but a bot that was previously blocked may take time to retry your domain, so don’t judge the change from the first hour alone.
What Should WordPress Site Owners Do This Week?
Start by confirming which Cloudflare plan your site is actually on — many WordPress owners set up Cloudflare years ago through a hosting panel and have never checked whether they’re on the free tier that this default now applies to. Hostinger, SiteGround, and similar hosts that bundle Cloudflare typically default new accounts to the free plan.
Next, decide your priority: if AI referral traffic already matters to you, spend ten minutes in AI Crawl Control allowing agentic retrieval crawlers before you do anything else. If you’d rather keep every AI bot out entirely and rely only on human and search traffic, the September 15 default already does that job for you on free-plan, ad-supported pages, and you don’t need to change anything.
Finally, put a reminder on your calendar to revisit the setting in a month. Cloudflare has iterated on this policy multiple times since its original July 2026 announcement, and the categories or defaults may shift again as more AI companies adopt the declaration standard.
Should You Opt Back Into AI Crawler Access?
There’s no universal answer, but the decision comes down to where your traffic and revenue actually come from. A site that earns primarily from display ads and has little AI-referral traffic loses little by leaving the default block in place. A site — like a tutorial or documentation site — that increasingly gets found through AI assistants has a real incentive to allow agentic retrieval crawlers while still blocking pure training bots.
Sites already experimenting with AI integrations, such as those following a Claude connector setup in WordPress or exploring the new WordPress Abilities API, are exactly the kind of publisher Cloudflare’s Pay Per Use model is aimed at: comfortable working with AI systems, but on terms they set rather than defaults imposed on them.
Weigh it against your ad revenue too. If display ads are a meaningful income source, opting a training crawler back in for free gives up leverage for nothing in return — Pay Per Use exists precisely so you can charge for that access instead of handing it over.
Frequently Asked Questions
Does the Cloudflare AI crawler block affect Googlebot or regular search rankings?
No. The policy targets crawlers that mix AI training or agent use with search indexing in a single undeclared bot. Single-purpose search engine crawlers like Googlebot and Bingbot are not classified as mixed-use and continue to index your site exactly as before, so your organic rankings are not directly affected by this default.
Do I need to do anything if I’m on Cloudflare’s free plan?
Only if you want a different outcome than the default. The block is already active on free-plan accounts as of September 15, 2026. If that’s the outcome you want, no action is needed; if you want to allow specific AI crawlers, visit Security > Bots > AI Crawl Control and change the setting yourself, category by category.
Is this the same thing as blocking bots with robots.txt?
No. Robots.txt is a voluntary request that well-behaved bots choose to honor. Cloudflare’s block operates at the network edge, actively refusing requests from classified crawlers regardless of whether the bot respects robots.txt, which is why Cloudflare frames it as an enforceable control rather than a polite suggestion.
Will this stop ChatGPT, Perplexity, or Gemini from citing my WordPress articles?
Only if the specific crawler those tools use to fetch live pages is flagged mixed-use and you leave the default block in place. Many AI assistants use separate, declared agentic-retrieval crawlers for live answers, which stay allowed by default — check the AI Crawl Control panel for the exact status of each one on your account before assuming you’ve lost citations.
Does opting into Pay Per Use cost site owners anything?
No, Pay Per Use is a revenue feature for publishers, not a fee. You set a price for AI companies that want continued access to a crawler category, and Cloudflare handles the billing between the crawler operator and your account; you don’t pay anything to participate in the marketplace.
What happens to sites that don’t use Cloudflare at all?
This policy only applies to traffic passing through Cloudflare’s network, so sites hosted elsewhere without Cloudflare in front of them are unaffected by the September 15 default and must rely on robots.txt, server-level rules, or their host’s own bot-management tools instead to control AI crawler access.
Subscribe for Newsletter
