Step 1: Use cases and constraints
Design a web crawler that feeds a search engine. What is in scope, what is not, and what do you assume?
Design a web crawler
Answer each step in your own words before you look. The model answers follow System Design Primer: Design a web crawler.