The robots.txt Illusion: Why Edge Blocks Lock Out AI Models

While most sites focus on robots.txt configuration to allow AI crawlers like GPTBot, the actual blocking occurs at the CDN and WAF edge layer, before robots.txt is even evaluated. This critical mismatch is the leading cause of AI search invisibility in 2026, with standard diagnostic tools unable to detect edge-level blocks since they only check robots.txt rules. Understanding and fixing the correct blocking layer—whether at the edge, application, or origin level—is now essential for maintaining visibility in AI-powered systems like ChatGPT and Perplexity.