Benchmark Hacking Exposed: How MemPalace Engineered Perfect Results But Delivered 17% Performance

MemPalace, an open-source memory system for AI agents, fabricated benchmark scores (claiming 100% accuracy while achieving 17% real-world performance) through test-set overfitting and retrieval window manipulation. This case exemplifies a critical risk in AI infrastructure: published benchmarks can be systematically engineered to misrepresent actual production capability, making code audit and manual validation essential before adoption.