1. A method for crawling content, comprising: identifying a first information submission mechanism in content that is being processed by a crawler, wherein processing of the first information submission mechanism requires the crawler to construct a plurality of requests for dynamic content based at least upon the first information submission mechanism; determining that the first information submission mechanism matches similarity criteria with a second information submission mechanism that has already been processed; in response to determining that the first information submission mechanism matches similarity criteria with the second information submission mechanism, terminating processing of the first information submission mechanism; wherein the method is performed by one or more processors at one or more computing devices.