The LLM Judge Paradox: Why Rules Beat AI at Detecting Agent Failures

The industry standard of using LLMs as judges for detecting agent failures faces a surprising challenge from empirical results. Rule-based heuristic detectors achieve 60.1% accuracy on the TRAIL benchmark versus just 11.9% for GPT-5.4, with zero false positives and no API costs. A tiered approach combining fast heuristic detection with selective Claude Sonnet calls offers a practical, cost-effective alternative for production systems.