%0 Report
%9 Technical Report
%A Cho, J.
%A Garcia-Molina, H.
%D 1999
%F ilprints:376
%I Stanford InfoLab
%K web evolution, incremental crawler, web change model
%T The Evolution of the Web and Implications for an Incremental Crawler
%U http://ilpubs.stanford.edu:8090/376/
%X In this paper we study how to build an effective incremental crawler. The crawler selectively and incrementally updates its index and/or local collection of web pages, instead of periodically refreshing the collection in batch mode. The incremental crawler can improve the ``freshness'' of the collection significantly and bring in new pages in a more timely manner. We first present results from an experiment conducted on more than half million web pages over 4 months, to estimate how web pages evolve over time. Based on these experimental results, we compare various design choices for an incremental crawler and discuss their trade-offs. We propose an architecture for the incremental crawler, which combines the best design choices.