Scraping millions of pages a day: what actually breaks

I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper...