%0 Report
%9 Technical Report
%A Cho, Junghoo
%A Garcia-Molina, Hector
%D 2002
%F ilprints:733
%I Stanford InfoLab
%K Web crawler, paralellism, distributed crawler
%T Parallel Crawlers
%U http://ilpubs.stanford.edu:8090/733/
%X In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.