Solution: Site-based crawl

Site

Manager

Crawler

Crawler

Site set request

crawl

crawl

WWW

Site set request

• Partition Web into sets of disjoint site sets

• Size of site sets controls visit rate

• ‘Slow but steady’ crawling

Previous slide

Next slide

Back to first slide

View graphic version