used with Apache Solr or Elasticsearch. Open Search Server is a search engine and web crawler software release under the GPL. Scrapy, an open source webcrawler framework, written in python (licensed under BSD). Seeks, a free distributed search engine (licensed under AGPL). StormCrawler, a collection of resources for building low-latency, scalable web crawlers on Apache Storm (Apache License). tkWWW Robot, a crawler based on the tkWWW web browser (licensed under GPL). Xapian, a search crawler engine, written in c++. YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL). See also Automatic indexing Gnutella crawler Web archiving Webgraph Website mirroring software Search Engine Scraping Web scraping References Further reading Cho, Junghoo, "Web Crawling Project", UCLA Computer Science Department. A History of Search Engines, from Wiley WIVET is a benchmarking project by OWASP, which aims to measure if a web crawler can identify all the hyperlinks in a target website. Shestakov, Denis, "Current Challenges in Web Crawling" and "Intelligent Web Crawling", slides for tutorials given at ICWE'13 and WI-IAT'13. Search engine software Internet search algorithms