1. A computer-implemented method of downloading Hidden Web pages comprising: a) selecting a query term; b) issuing a query to a site-specific search interface containing Hidden Web pages; c) acquiring a results index; d) downloading the Hidden Web pages from the results index; e) identifying a plurality of potential query terms from the downloaded Hidden Web pages; f) estimating the efficiency of each potential query term based on a ratio of the number of new pages returned for a particular query to the cost of issuing the particular query wherein the cost of issuing the particular query is equal to c q +c r P(q i )+c d P new (q i ) where P(q i ) represents the fraction of pages returned for a particular query (q i ) and P new (q i ) represents the fraction of new pages returned for a particular query (q i ), and where c q represents the cost of submitting the particular query, c r represents the cost of retrieving a results index page, and c d represents the cost for downloading a matching document; g) selecting a next query term from the plurality of potential query terms, wherein the next selected query term has the greatest efficiency; and h) issuing a next query to the site-specific search interface using the next query term.
1. A computer-implemented method of downloading Hidden Web pages comprising: a) selecting a query term; b) issuing a query to a site-specific search interface containing Hidden Web pages; c) acquiring a results index; d) downloading the Hidden Web pages from the results index; e) identifying a plurality of potential query terms from the downloaded Hidden Web pages; f) estimating the efficiency of each potential query term based on a ratio of the number of new pages returned for a particular query to the cost of issuing the particular query wherein the cost of issuing the particular query is equal to c q +c r P(q i )+c d P new (q i ) where P(q i ) represents the fraction of pages returned for a particular query (q i ) and P new (q i ) represents the fraction of new pages returned for a particular query (q i ), and where c q represents the cost of submitting the particular query, c r represents the cost of retrieving a results index page, and c d represents the cost for downloading a matching document; g) selecting a next query term from the plurality of potential query terms, wherein the next selected query term has the greatest efficiency; and h) issuing a next query to the site-specific search interface using the next query term. 12. The method of claim 1 , wherein in step (d), the Hidden Web pages are downloaded from a plurality of partial results indexes.