Should You Charge AI Crawlers? Allow, Charge, or Block.

TL;DR: For most businesses the right answer is allow: the AI crawlers that fetch your pages to answer live questions are the ones that cite you, and citations are the new front door. Charge only when your content is itself the product, and block only for real risk. Cloudflare’s Pay Per Crawl makes charging possible over HTTP 402, but it is a per-crawler decision, not a site-wide switch, and the cost of getting it wrong is invisible: you simply stop appearing in answers.

Part 2 of 5 in the Field Notes series When Agents Pay. Previously: how AI agents pay for things over HTTP 402. Next: x402 vs MPP.

What are your three options for each AI crawler?

Cloudflare’s AI Crawl Control reduces the decision to three actions per crawler, and the vocabulary is worth adopting even if you are not on Cloudflare:

• Allow: free access, no charge.
• Charge: the crawler pays your set price for each successful content access, over HTTP 402.
• Block: no access at all.

The important word is each. GPTBot (OpenAI’s training crawler), OAI-SearchBot (the one that powers ChatGPT search results), ChatGPT-User (fetching a page because a user asked), ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, and Bytespider are different agents doing different jobs for different companies. A single policy for “AI” treats a crawler that will cite you tomorrow the same as one scraping for a training corpus that may never mention you. Decide per crawler, per job.

How does Pay Per Crawl actually work?

Pay Per Crawl is Cloudflare’s application of the 402 payment flow to crawlers. The mechanics, from Cloudflare’s own documentation:

1. You set one price that applies to every crawler you configure with Charge. Crawlers set to Allow pay nothing; crawlers set to Block never reach the price.
2. A charged crawler requesting a page receives HTTP 402 Payment Required with a crawler-price header, for example crawler-price: USD 0.01.
3. The crawler retries with either crawler-exact-price (matching your price) or crawler-max-price (the most it will pay for any content). Those headers must be included in the crawler’s Web Bot Auth signature, so an unsigned request cannot claim to have paid.
4. On success the response carries crawler-charged: USD 0.01, the amount billed to the crawler operator’s Cloudflare account. Error responses are not billed. /robots.txt, /sitemap.xml, and /security.txt stay free so crawlers can still discover your rules.

Cloudflare acts as the Merchant of Record and pays out to you. Crawler operators must be verified through Web Bot Auth (Part 5 of this series covers it), and WAF or Bot Management blocks take precedence over charging: a blocked crawler gets nothing, not a bill. The feature is in closed beta as of mid-2026, available on request and to Enterprise accounts through their account team.

Two facts to keep straight when you read coverage of this. Cloudflare has not published a rate card; the price is yours. And the revenue figures circulating in third-party articles are projections, not published results from Cloudflare.

What does blocking actually cost you?

Nothing you can see in analytics, which is the problem.

When someone asks ChatGPT, Perplexity, or Google AI Overviews a buying question, the answer is assembled from pages the platform can fetch and trust. If your pages return 403 to Claude-SearchBot or PerplexityBot, you are not in the answer, and the person never learns you existed. There is no bounce, no impression, no lost click to count. The cost shows up as a competitor’s name in a place yours should have been.

That is why we treat crawler policy as an AI Visibility decision first and a monetization decision second. The three pillars pull in different directions here: Modern Web wants fast pages, AI Visibility wants to be read and cited, Agent Readiness wants the interaction to be governed. Charging is a governance move. Make it after you know what your citations are worth.

The exception is real: if your content is the product, if people already pay for the archive or the dataset or the research, then a crawler reading it for free is a licensing problem, and Charge is the honest answer.

Which crawlers should you allow, and how do you see who’s crawling?

Start by looking. AI Crawl Control is available on every Cloudflare plan and shows which AI services are fetching your content, how often, and whether they are honoring your robots.txt. Its robots.txt monitoring lets you “track which crawlers follow your directives and create enforcement rules,” which is the first time most site owners get evidence rather than assumption about crawler behavior.

Then sort by job, not by company:

Note that Cloudflare’s own docs carry a warning here: blocking or charging search engine crawlers can “negatively impact your site’s SEO performance.” Keep Googlebot and Bingbot out of the AI policy entirely.

What do we do on boldcrow.ai?

We allow. Our robots.txt explicitly welcomes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, meta-externalagent, Applebot-Extended, Amazonbot, Bytespider, and DuckAssistBot, carries a Content-Signal line that permits search, AI answers, and training, and says why: the site is meant to be read by machines as well as people. We publish llms.txt and run a public MCP server for the same reason. Being cited is worth more to a consultancy than a cent per page.

And here is the honest part. When we ran our own audit crawler against boldcrow.ai in August, it flagged conflicting rules. The served robots.txt began with a Cloudflare-managed block that disallowed GPTBot, ClaudeBot, Google-Extended, and others, ahead of our own block allowing them. Under RFC 9309 the tie resolves to Allow, but a crawler that takes the first matching group reads it as blocked. The rule was not in our code; it came from a dashboard setting in AI Crawl Control’s managed robots.txt. We switched it off in September, and the file crawlers now receive is the one in our repository.

The lesson generalizes: audit the robots.txt that is actually served, not the one in your repository. Edge products, CDN defaults, and managed rules can rewrite your crawler policy without a commit. It is one of the checks in the Agent Readiness Audit because we needed it ourselves.

A decision table by content type.

Every row assumes you can tell the crawlers apart. That is what verification is for, and it is why the payment layer in Part 1 and the identity layer in Part 5 sit under this decision.

The question isn’t whether AI crawlers should pay. It’s which of them you want reading you for free, because those are the ones that will put your name in the answer.

Not sure what your crawler policy actually says?

Bold Crow AI checks the served robots.txt, crawler behavior, and answer-engine citations as part of the Agent Readiness Audit, and builds agent-ready websites that make the allow decision pay off.

Contact Bold Crow AI to find out who is reading your site, and who isn’t.