Binary Logic Over Scores: LLM Evaluation Gets Clearer in CI Pipelines

Replacing vague 5-point rubrics with binary criteria nearly doubled inter-rater agreement (Cohen's kappa 0.47→0.78), directly improving CI reliability. While the move demands new threshold logic and criterion-specific prompts, it eliminates averaging noise, makes regressions instantly actionable, and reduces weekly calibration time by 45%.