13. The system of claim 8 , wherein the plurality of documents are partitioned and each partition is processed independently.
8. A system for generating and using a semantic space in a computer system, comprising one or more computer processors configured to: receive a plurality of documents and a plurality of random document vectors, wherein each random document vector in the plurality of random document vectors is generated based on random indexing and is associated with a corresponding document in the plurality of documents; select a term for which to generate a term vector; for a first document in the plurality of documents, determine if the term appears in the document; and if the term appears in the document: determine a frequency of the term in the document; add an associated random document vector of the document to the term vector, wherein the associated random document vector is scaled by the term frequency; determine if the term appears in any remaining documents in the plurality of documents; if the term does not appear in any remaining documents in the plurality of documents, generate a normalized version of the term vector with the added associated random document vector; output the normalized term vector; receiving a query; and generating, based at least in part on the semantic space including the normalized version of the term vector, a user interface displaying similar and near-duplicate documents.