= Web crawler
A <web crawler> automatically discovers and fetches web resources, often to supply a <web search engine>. Starting from known addresses, it can extract links and schedule newly discovered addresses for later fetching. Discovery, fetching and indexing are distinct: a fetched page need not be accepted into the searchable collection.<In-depth guide to how Google Search works by Google>[Crawling, discovery through links and limits on fetching/indexing]{ref}
In a toy web, page `A` links to `B` and `C`, while `B` links back to `A`. Recording previously encountered addresses prevents that cycle from causing endless rediscovery. A pending-work list determines what to fetch next; revisits are still useful when pages change. Real <web crawlers> also control request rates and handle unavailable resources. This makes crawling a continuing resource-allocation task rather than a single traversal that permanently captures the entire web.
Back to article page
