When Test Metrics Deceive: How Impressive Numbers Masked a Critical Bug

Test-selection tools promise faster CI by skipping unaffected tests, but impressive efficiency metrics can mask silent failures. A real bug in coverage.py's sysmon feature revealed how a tool claiming 73% test skip rates was actually creating false negatives—the worst possible outcome in testing. This matters because it demonstrates that metrics which seem too good to be true in optimization tools often are, and verification must precede celebration.