1. A method of web crawling hidden files, comprising: loading a web page with a browser agent; executing any dynamic elements hosted on the web page using the browser agent to insert pre-determined values; retrieving a list of form controls from the web page using the browser agent; analyzing the form controls using a driver component of a crawler; sending form control values from the driver component to the browser agent; submitting an event to the web page by the browser agent or running any scripted content to trigger operations on the web page corresponding to the form control values; and generating a URL for various form control values using a generalizer.