Fetching, DNS caching and fault tolerance

DNS is a bottleneck

Every download starts with a host-name lookup. Najork and Heydon found that the standard Unix resolver (gethostbyname) was synchronized, so "only one DNS request per address space on an uncached name could be outstanding at once". That caused "a severe bottleneck".

Their fix was a multi-threaded resolver of their own that forwards requests to a local name server and lets many lookups run in parallel. Before the change, DNS lookups took 70% of each thread's elapsed time; afterward, 14%.

The primer gives the same advice: "the Crawler Service can keep its own DNS lookup that is refreshed periodically". Cache each answer for the record's own TTL, and partition the cache with the hosts (lesson 2).

Fetching efficiently

The primer suggests:

• Connection pooling: keep many connections open at a time, which "can improve performance and reduce memory usage".
• Enough bandwidth: at the primer's 1,600 pages per second, about 6.4 Gbit/s (lesson 1).
• Sharding and federation for the Reverse Index Service and Document Service, given the data size and request load.
• A memory cache for popular searches, which also absorbs uneven traffic and spikes.

Freshness

Pages change, so the crawl never ends. The primer keeps a timestamp per crawled page and refreshes every page after a default period, such as a week, and popular or frequently updated sites more often. With some data mining, you can estimate how often a page changes and re-crawl it at that rate. Mercator's continuous crawling puts a downloaded URL back in the frontier with a priority based on its download history.

Spider traps and cycles

The primer's loop breaks cycles by lowering the priority of a link whose page signature matches one already crawled. Some sites generate unlimited distinct URLs, such as calendars with a "next month" link forever or repeating path segments. Common defenses, each a policy you tune:

• a maximum link depth from the seed URLs;
• rejecting URLs whose path repeats the same segments;
• a cap on pages per host per crawl cycle;
• checking Content-Type and Content-Length before downloading a body.

Storing pages: WARC files

Writing billions of separate small objects is slow and expensive. Crawlers append pages to large archive files instead. The WARC format, whose specification the IIPC publishes, records each response with its metadata. It recommends "1 GB (10⁹ bytes) as a practical target size for WARC files".

When a crawler machine fails

Mercator checkpoints the crawl's state to stable storage on a schedule, run from a background thread. Distributed crawlers also need to detect dead workers and hand their hosts to another process. A host's URLs live with its frontier, so moving the frontier partition moves its work too.

A URL fetched twice because of a crash produces a duplicate page. The content signature check (lesson 3) catches it before it is indexed again.