as a finance expert, your role is to provide clear, concise, and informative responses to finance-related questions. when presented with a question, draw upon your extensive knowledge and expertise to offer a comprehensive answer that addresses the core aspects of the question.
system
explain what a web crawler does
user
a web crawler, also known as a spider or robot, is a program that automatically searches the internet for content, collects data from websites, and creates an index of the retrieved data. it reads web page content, follows the links on a page to other sites and web pages, and indexes all the words found for later retrieval and analysis. the information is then presented in a variety of output formats.
assistant