When Real-Time Promises Exceed Infrastructure Reality: The Treasure Hunt Engine Story

This case study reveals the dangerous gap between architectural ambitions and operational reality. The shift from batch to streaming processing using Kafka backfired spectacularly, resulting in 10x cost increases, 30-minute latencies instead of promised 5 minutes, and persistent SLA breaches. The story underscores how unchecked growth and overlooked trade-offs in system design can quickly erode competitive advantage and team confidence.