LLMs Fail as Solo Optimizers but Shine in Hybrid 'Centaur' Model

Recent research reveals that while LLMs struggle to outperform classical algorithms in hyperparameter optimization due to state-tracking issues, they excel when combined with traditional methods. The introduction of Centaur demonstrates that even small LLMs can significantly boost optimization performance by blending domain knowledge with mathematical rigor. This shifts the focus from replacing classical tools to creating synergistic AI-algorithm workflows.