Cloudflare's September 2026 Crawler Wall: What Pay Per Use Means For Your Content

On 15 September 2026 Cloudflare’s default settings started blocking “mixed-use” AI crawlers from pages that carry ads, and the company began turning its old Pay Per Crawl marketplace into a Pay Per Use one. The framing shifted from charging AI companies to fetch your content to paying publishers when that content actually shows up in an answer.

If you run a site behind Cloudflare, this is not a setting you can ignore, and it is easy to get wrong in both directions. Turn the wall on too hard and you lock out the search bots you actually want. Leave it fully open and you keep handing training and agent data away for nothing. Here is what changed and how to think about it.

What Actually Flipped On

A “mixed-use” crawler, in Cloudflare’s definition, is one that combines traditional search indexing with data collection for AI training or agent development. Starting 15 September, the default blocks those crawlers on any page hosting ads. Per TechCrunch’s reporting on the policy, the new defaults apply to new Cloudflare customers, new sites created by existing customers, and all existing free customers. If you are on a free plan and have not touched your bot settings, the new behavior is already live on your site.

The second piece is the marketplace. Pay Per Crawl charged AI companies each time they fetched a page. Pay Per Use is meant to pay publishers when their content generates value inside an AI result, including access fees for premium content, rather than only on the fetch. Whether that produces real revenue for a small site is unproven, and worth treating as a claim to watch rather than a check in the mail.

The Trap Is Blocking Your Own Search Traffic

The failure mode here is not subtle. Site owners flip on blanket crawl control to stop aggressive AI scrapers, and quietly lock out legitimate search engine indexing at the same time, because the same broad setting catches both. If your pages stop appearing in ordinary search a week after you tightened the rules, the crawler wall is the first thing to check.

The distinction that matters is between a crawler that indexes you so people can find you, and one that ingests you so a model can answer without sending anyone your way. You generally want the first and get to decide about the second. Whether you should block AI crawlers at all walks through that decision for a site that lives on search traffic, and the answer is rarely “block everything.”

Decide Per Bot, Not With One Switch

Treat this as a policy you write, not a toggle you flip. List the crawlers reaching your site, sort them into “helps people find me,” “trains a model on me,” and “acts as an agent for a buyer,” and set access per category rather than all at once.

Before:

We turned on the strictest bot blocking to stop AI stealing our articles.

After:

We are blocking training crawlers and mixed-use bots on our ad pages, allowing verified search indexers, and evaluating Pay Per Use for our premium guides. Search traffic is unaffected because we allowlisted the indexing bots explicitly.

The second version is a decision someone can audit in three months. The first is a switch nobody remembers the reason for.

A Wrivio Context for writing a crawler policy note could say:

Rewrite this into a short internal policy note for our website. Keep every bot name, product name, and date exactly as written. State which crawlers are allowed and which are blocked and why, in plain language a non-technical colleague can follow. Do not add claims about revenue or SEO impact that are not already in the draft.

Press Ctrl+Shift+Space, paste your rough policy, and check the diff. Watch that it did not silently merge “search bots” and “AI training bots” into one category, because that merge is exactly the mistake the default settings encourage.

This Is Part Of A Bigger Shift

The crawler wall is one move in a larger rebalancing of who pays whom in AI search. Answer engines increasingly cite a narrower, different set of sources than classic search ranked, and access is being negotiated at the infrastructure layer rather than assumed. Agentic search optimization covers the other half: making your content usable by the agents you do want to reach you, once you have decided which those are. The two questions, who you block and who you court, are now the same conversation.

Common Questions

What is a mixed-use AI crawler?

Cloudflare defines it as a crawler that combines traditional search indexing with data collection for AI training or agent development. Because it does both jobs at once, blocking it to stop AI training also risks blocking the indexing that search results depend on, which is why Cloudflare now treats it as its own category.

Did the September 2026 defaults change my site automatically?

If you are an existing free customer or created a new site, yes: the new blocking defaults apply without you doing anything. Existing paid sites you already configured are less likely to change silently, but checking your bot settings after 15 September 2026 is the only way to be sure.

Will Pay Per Use actually pay me?

That is unproven for a small publisher. The model is designed to pay when your content is used in an AI result, launching with a limited set of partners, so treat any revenue as a possibility to monitor rather than income to plan around in 2026.

How do I avoid blocking search engines by accident?

Do not use a single blanket block. Allowlist the verified search indexing bots explicitly, then apply stricter rules to training and mixed-use crawlers. After changing settings, confirm your pages still appear in ordinary search before assuming the configuration is correct.

Does this affect what my content looks like to AI answers?

Indirectly. If you block the crawlers an answer engine relies on, you stop appearing in its answers, for better or worse. The content itself still needs to be structured to be quotable, which is a separate task from deciding who is allowed to read it.

Download Wrivio for Windows to draft your crawler policy and site notices privately, without the working copy passing through a chatbot you do not control.