8. A computer program product for indexing documents, the computer program product comprising a computer readable memory device having computer readable program code stored therein, the computer readable program code comprising: computer readable program code configured to index a document written in one or more natural languages, wherein indexing includes: applying language-specific rules of one of the one or more natural languages to content of the document; and generating a number of tokens from the content of the document that comply with the language-specific rules; computer readable program code configured to determine a success metric based on the generated number of tokens that complied with the language-specific rules; computer readable program code configured to compare the success metric to a threshold value, wherein the threshold value indicates a predetermined amount of content in the document designating a multi-lingual document; computer readable program code configured to identify the document as a multi-lingual document based upon the comparing; and computer readable program code configured to queue the document for multi-lingual indexing in response to identifying the document as a multi-lingual document.
8. A computer program product for indexing documents, the computer program product comprising a computer readable memory device having computer readable program code stored therein, the computer readable program code comprising: computer readable program code configured to index a document written in one or more natural languages, wherein indexing includes: applying language-specific rules of one of the one or more natural languages to content of the document; and generating a number of tokens from the content of the document that comply with the language-specific rules; computer readable program code configured to determine a success metric based on the generated number of tokens that complied with the language-specific rules; computer readable program code configured to compare the success metric to a threshold value, wherein the threshold value indicates a predetermined amount of content in the document designating a multi-lingual document; computer readable program code configured to identify the document as a multi-lingual document based upon the comparing; and computer readable program code configured to queue the document for multi-lingual indexing in response to identifying the document as a multi-lingual document. 9. The computer program product of claim 8 further comprising computer readable program code configured to fragment the document into a plurality of smaller documents and separately index the smaller documents individually.