EsoLang-Bench: Evaluating Genuine Reasoning in LLMs via Esoteric Languages

EsoLang-Bench is a new benchmark evaluating LLM code generation across five esoteric programming languages where training data is 5,000–100,000x scarcer than Python. Frontier models that score 85–95% on standard benchmarks collapse to 0–11% accuracy on equivalent esoteric tasks, with Whitespace remaining completely unsolved at 0%. The results demonstrate that high scores on mainstream language benchmarks largely reflect memorization of training data rather than genuine reasoning or general programming ability.