Challenge 11: Capstone: web crawler storage
Fetchers bring back 400 pages a second (about a billion a month), and 30% of them repeat a page already stored. A month's crawl is 500 TB of unique pages, plus 2 TB of metadata. Store them, compute, storage and per-request charges together, for at most 17,500 USD a month, with p99 under 200 ms, at least 99.9% availability, and surviving the loss of any instance.
Traffic demand400 RPS0% reads, 100% writes
p99 latency target≤ 200 msSlowest 1% of requests
Availability≥ 99.9%N-1 failure survival (0 drops)
Monthly cloud budget≤ 17,500 USDA month, everything included