what are search engine crawlers?
Crawling is the process used by search engine web crawlers (bots or spiders) to visit and download a page and extract its links in order to discover additional pages. ... If a search engine detects changes to a page after crawling a page, it will update it's index in response to these detected changes.