Paywall for Bots: Testing HTTP 402 Payment Enforcement Against AI Crawlers

As AI scraping becomes ubiquitous, one developer implemented a machine-readable payment protocol using HTTP 402 to charge crawlers for content access. The experiment reveals critical insights about AI bot behavior: while 5,811 crawler requests arrived over 30 days, only 5 paid, exposing that most AI agents either lack payment compliance or actively evade it—a signal that industry-wide monetization through standard HTTP mechanisms remains largely ineffective without broader adoption.