Why Standard Eval Frameworks Fail at Benchmarking Code-Intelligent Agents

Traditional evaluation frameworks like Inspect AI and Promptfoo were designed for single-turn model outputs, making them inadequate for benchmarking MCP servers where the variable is the tool, not the model. The author's custom methodology reveals that two of four tested MCP servers actually underperform a simple grep-based baseline, highlighting the critical importance of measuring hallucination, cost efficiency, and filesystem grounding in AI code tools. This work establishes methodological standards for transparent, reproducible benchmarking in an area where bias and measurement challenges are often hidden.