Web Bot Auth: How to Verify an AI Agent Is Who It Claims.

TL;DR: You cannot know an AI agent is who it claims from its User-Agent header; anyone can type “GPTBot.” Web Bot Auth fixes that with cryptography: the bot operator publishes Ed25519 public keys at /.well-known/http-message-signatures-directory, signs every request using HTTP Message Signatures (RFC 9421), and points to its key directory with a Signature-Agent header. The receiver fetches the keys and verifies the signature. Identity is the layer under everything else in this series: you cannot price, allow, or meter an agent you cannot verify.

Part 5 of 5 in the Field Notes series When Agents Pay. Previously: how AI agents pay over HTTP 402, should you charge AI crawlers, x402 vs MPP, and how to charge for MCP tools.

Why can’t you trust a User-Agent header?

Because it is a string the client writes about itself. A scraper can send User-Agent: GPTBot and inherit whatever access you granted OpenAI. A competitor’s agent can claim to be Googlebot. Every allow, charge, or block rule you write against a user-agent name is a rule against a costume.

The traditional workaround was IP allow-lists: publish the address ranges your crawler uses and let sites check against them. That works until crawlers move to shared cloud infrastructure, until ranges rotate, and until the number of legitimate agents grows past what any site can maintain by hand. It also tells you nothing about the tens of thousands of new agents that will never be big enough to publish ranges.

The web needed a way for a client to prove identity in the request itself. That is what signatures do.

What is Web Bot Auth?

Web Bot Auth is Cloudflare’s name for a method that “leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot,” used as a verification method for verified bots and agents. It rests on three standards documents:

• RFC 9421, HTTP Message Signatures: the IETF standard for signing selected parts of an HTTP message so a receiver can confirm who sent it and that it was not changed in transit.
• The HTTP Message Signatures Directory draft: how a bot operator publishes its public keys so anyone can find them.
• The Web Bot Auth architecture draft: how those keys are attached to crawler identity on each request.

The signature algorithm is Ed25519, exclusively. Cloudflare publishes open-source implementations in Rust and TypeScript, and a test endpoint at crawltest.com for operators to check their signing before registration.

What’s in a signed request?

Three headers ride along with an otherwise ordinary HTTP request:

Signature-Input names what was signed and under which parameters. It includes the signed components (at minimum the @authority, the host being requested), the algorithm (ed25519), a keyid (the JWK thumbprint of the signing key), created and expires timestamps that bound the signature’s validity to a short window, an optional nonce to prevent replay, and a tag set to web-bot-auth.

Signature carries the Ed25519 signature bytes over those components.

Signature-Agent points to the bot’s key directory as a structured string, for example "https://signature-agent.test", telling the receiver where to fetch the public key that matches keyid.

The receiver’s job is mechanical: read Signature-Agent, fetch /.well-known/http-message-signatures-directory from that origin, find the key whose thumbprint matches keyid, verify Signature over the components in Signature-Input, and check that created and expires bracket the current time. If all of that holds, the request came from whoever controls that key directory. If any of it fails, it did not, whatever the User-Agent says.

How does a bot operator register?

For the operator of a crawler or agent, Cloudflare’s documentation lays out the sequence:

1. Generate an Ed25519 key pair and convert the public key to JWK format.
2. Host a key directory at /.well-known/http-message-signatures-directory on a domain you control, serving a JSON Web Key Set with signed responses.
3. Sign outgoing requests with the three headers above.
4. Register the bot in the Cloudflare dashboard, selecting “Request Signature” as the verification method. Cloudflare accepts “all valid Ed25519 keys found in your key directory.”

For a site owner, none of that is your work. Your work is deciding what verified identity earns: which verified agents you allow, which you charge, and what happens to traffic that cannot prove who it is.

How does identity connect to payments and crawl control?

Every earlier post in this series quietly depended on this one.

Pay Per Crawl requires crawlers to be identified through Web Bot Auth, and its documentation requires the crawler-exact-price and crawler-max-price headers to be included in the signed components. The payment claim and the identity claim are bound together in one signature: nobody can present your price under someone else’s name.

An HTTP 402 credential proves that a payment happened. It does not, by itself, prove who made the request. For anonymous micro-purchases that may be fine. For anything with a rate limit, a per-buyer price, an audit trail, or a dispute process, you need both: the signature says who, the credential says paid.

And an allow, charge, or block rule in AI Crawl Control is only as good as the identification behind it. Verified identity is what makes the “Allow” you extend to Claude-SearchBot mean Claude-SearchBot.

In the six-level Agent Readiness framework we use in the audit, this is the Trusted level: Discoverable and Understandable get an agent to your door; Trusted is where you know which agent is standing there. Everything above it (Accessible, Transactable, Actionable) assumes it.

What should a site owner do this quarter?

Four moves, in order, none of them expensive:

1. Turn on visibility. See which agents are actually reaching you and whether they are verified. On Cloudflare that is AI Crawl Control on any plan; elsewhere it is your logs plus the User-Agent list you already distrust.
2. Prefer verified agents in policy. Where a product lets you distinguish verified from unverified traffic, write your Allow rules for verified identities and put unverified traffic on the conservative rule.
3. Keep the payment layer designed, not necessarily live. If you plan to charge for MCP tools or crawls, design fulfillment so it can key on a verified identity, even if the first version only checks the payment.
4. Audit the served configuration. Rules live in dashboards, CDNs, and managed files as well as in your repo. Check what a real request sees.

The question isn’t whether an agent can prove who it is. The standards for that exist. The question is whether your rules are written against identities or against costumes.

Want to know which agents are actually reaching you?

Bold Crow AI’s Agent Readiness Audit scores your site across all six levels, Trusted included, with evidence from real crawler and agent requests, and our agent-ready websites are built so verified agents can find, read, and use them.

Contact Bold Crow AI to see who is at your door.