How to charge AI crawlers to read your website
AI companies send crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot and others) to read websites for search answers and training. Today most sites can only allow them or block them. There is now a third option: let people read free and ask the crawler to pay.
How paying works: x402
x402 is an open standard built on the old HTTP status 402 Payment Required. When a crawler asks for a page, the site answers 402 with a price (for example $0.01 in USDC) and where to pay. A crawler that can pay signs a payment, retries, and gets the page with a receipt. One that cannot pay simply does not get it. People in a browser never see any of this.
What it does not do
- The big crawlers do not carry wallets today, so for them a 402 works like a polite block, not income. Paying clients are AI agents built to pay with x402.
- A scraper that pretends to be a normal browser is not stopped by price alone; that needs bot checks (Tollbooth's AI Shield does those).
Try it now
Our demo copy of this site answers people normally and AI crawlers with 402 (test money only, on Base Sepolia):
curl -i https://toll-demo.dusklin.ai/ curl -i -A "GPTBot/1.0" https://toll-demo.dusklin.ai/
The first returns the page. The second returns 402 with the price in a PAYMENT-REQUIRED header. For blocking instead of charging, try shield-demo.dusklin.ai: a browser reads it, GPTBot is refused.
Tollbooth puts this in front of your own site: founding members $49 for the first 12 months from launch.