The regression your eval set is too small to catch

TL;DR. To catch a drop from a 0.90 pass rate to 0.85 at 80% power (one-sided, alpha 0.05), you need about 253 examples. A 50-example set has roughly 35% power, so it misses that regression about two t...