Heuristic vs Semantic Eval: When <1ms Matters More Than LLM-as-Judge

The article challenges the default assumption that agent evaluation requires LLM-as-Judge, arguing that many common checks—PII detection, prompt injection detection, output completeness—can be handled by heuristic rules (regex, string matching) in microseconds with zero cost and full determinism. LLM-as-Judge adds latency, cost, and non-determinism, and should be reserved for evaluations that genuinely require semantic understanding such as subjective quality or factual accuracy assessment.