10. A method for generating feature vectors of documents in different languages, the method comprising: receiving at a processing unit a document and identifying the language of the document as one of a base language or a second language; selecting a corresponding tokenizer based on the identified language to parse the received document and produce a feature vector of the received document, the tokenizer selected from a plurality of tokenizers stored in a memory unit, each tokenizer of the plurality of tokenizers associated with a language and a respective keyword set of a plurality of keyword sets stored in a keyword repository of the memory unit; parsing the received document using the selected tokenizer to identify keywords occurring in the received document, the keywords stored in a keyword set associated with the language of the selected tokenizer and an identifier (ID) of a corresponding keyword in a base language keyword set, wherein each respective ID is unique for each keyword of the base language keyword set; and generating a feature vector from a plurality of ID:score pairs, each pair associating a score with an ID of a keyword in the associated keyword set occurring in the document, the score based on the frequency of occurrence of the ID corresponding to the keyword in the document and a relevance weighting associated with the keyword, wherein the relevance weighting is based on a frequency of occurrence of the keyword in a set of previously collected documents, the feature vector providing scores associated with keywords defined in a base language for use by a profiler for generating or updating a user profile defining user preferences.
10. A method for generating feature vectors of documents in different languages, the method comprising: receiving at a processing unit a document and identifying the language of the document as one of a base language or a second language; selecting a corresponding tokenizer based on the identified language to parse the received document and produce a feature vector of the received document, the tokenizer selected from a plurality of tokenizers stored in a memory unit, each tokenizer of the plurality of tokenizers associated with a language and a respective keyword set of a plurality of keyword sets stored in a keyword repository of the memory unit; parsing the received document using the selected tokenizer to identify keywords occurring in the received document, the keywords stored in a keyword set associated with the language of the selected tokenizer and an identifier (ID) of a corresponding keyword in a base language keyword set, wherein each respective ID is unique for each keyword of the base language keyword set; and generating a feature vector from a plurality of ID:score pairs, each pair associating a score with an ID of a keyword in the associated keyword set occurring in the document, the score based on the frequency of occurrence of the ID corresponding to the keyword in the document and a relevance weighting associated with the keyword, wherein the relevance weighting is based on a frequency of occurrence of the keyword in a set of previously collected documents, the feature vector providing scores associated with keywords defined in a base language for use by a profiler for generating or updating a user profile defining user preferences. 13. The method of claim 10 , wherein parsing the received document further comprises comparing words of the received document to keywords from the keyword set associated with the tokenizer and mapping a word to an ID of the corresponding keyword when the word matches the keyword.