A web crawler automatically discovers and fetches web resources, often to supply a web search engine. Starting from known addresses, it can extract links and schedule newly discovered addresses for later fetching. Discovery, fetching and indexing are distinct: a fetched page need not be accepted into the searchable collection.[ref]
In-depth guide to how Google Search works by Google. Crawling, discovery through links and limits on fetching/indexing
In a toy web, page
A links to B and C, while B links back to A. Recording previously encountered addresses prevents that cycle from causing endless rediscovery. A pending-work list determines what to fetch next; revisits are still useful when pages change. Real web crawlers also control request rates and handle unavailable resources. This makes crawling a continuing resource-allocation task rather than a single traversal that permanently captures the entire web. New to topics? Read the docs here!