crawler. It supports Siri and other products. Bingbot is the name of Microsoft's Bing webcrawler. It replaced Msnbot. Baiduspider is Baidu's web crawler. Googlebot is described in some detail, but the reference is only about an early version of its architecture, which was written in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server. WebCrawler was used to build the first publicly available full-text index of a subset of the Web. It was based on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query. WebFountain is a distributed, modular crawler similar to Mercator but written in C++. Xenon is a web crawler used by government tax authorities to detect fraud. Commercial web crawlers The following web crawlers are available, for a price:: Diffbot - programmatic general web crawler, available as an API SortSite - crawler for analyzing websites, available for Windows and Mac OS Swiftbot - Swiftype's web crawler, available as software as a service Open-source crawlers GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites. GRUB was an open source distributed search crawler that Wikia Search used to crawl the web. Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java. ht://Dig includes a Web crawler in its indexing engine. HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL. mnoGoSearch is a crawler, indexer and a search engine written in C and licensed under the GPL (*NIX machines only) Apache Nutch is a highly extensible and scalable web crawler written in Java and released under an Apache License. It is based on Apache Hadoop and can be