1. A method for crawling content, comprising: identifying a first information submission mechanism in content that is being processed by a crawler, wherein processing of the first information submission mechanism requires the crawler to construct a plurality of requests for dynamic content based at least upon the first information submission mechanism; determining that the first information submission mechanism is a duplicate version of a second information submission mechanism that has already been processed; wherein determining that the first information submission mechanism is a duplicate version of the second information submission mechanism comprises: determining that the first information submission mechanism comprises first metadata that is similar to second metadata that the second information submission mechanism comprises; in response to determining that the first information submission mechanism is a duplicate version of the second information submission mechanism, terminating processing of the first information submission mechanism; wherein the method is performed by one or more processors at one or more computing devices.