Why 96% of the Web Is Invisible to AI Agents (And What You Can Do About It)

A study of 834 websites found that 96% are partially or fully invisible to AI agents, which cannot render JavaScript, bypass CAPTCHAs, or parse unstructured HTML. Key barriers include client-side rendering (SPAs), aggressive bot blocking, lack of structured data and llms.txt files, and absence of machine-readable APIs. Top-scoring sites share common traits: server-side rendered HTML, llms.txt files, JSON-LD schema markup, and documented APIs.