Silent Cascade: When 800,000 API Retries Go Undetected

When an API key was rotated and returned 401 auth errors, the system kept retrying silently without detection for two hours, accumulating 800,000 failed requests and filling the disk. This incident reveals a critical flaw in error handling strategy: not all errors should be retried equally. The author implemented a simple circuit breaker pattern to recognize permanent failures like 401s and stop the cascade before it becomes catastrophic, demonstrating why differentiating between recoverable and permanent errors is essential for system resilience.