Writing Evals for an LLM Security Tool: How I Know It Didn't Get Worse

Every time a new model ships, I face the same question for spectr-ai: does my contract auditor get better or worse on the new model? "Vibes" is not an answer when the tool tells people whether their c...