4. The system of claim 1 , wherein each tokenizer comprises an ID mapper for comparing words of the received document to keywords from the keyword set associated with the tokenizer and mapping a word to an ID of the corresponding keyword when the word matches the keyword.
1. A content modeling system comprising: a memory unit for storing instructions and data; and a processing unit for executing the instructions to generate feature vectors of documents in different languages, the feature vectors providing scores associated with keywords defined in a base language for use by a profiler for generating or updating a user profile defining user preferences, the processing unit further for executing the instructions to provide: a keyword repository storing a plurality of keyword sets comprising: a base language keyword set comprising a plurality of base language keywords each associated with a respective identifier (ID), wherein each respective ID is unique for each of the plurality of base language keywords; and a second language keyword set comprising a plurality of second language keywords each corresponding in meaning to a respective one of the base language keywords and associated with the ID of the corresponding base language keyword; a plurality of tokenizers, each tokenizer associated with a language and a respective keyword set of the plurality of keyword sets, each tokenizer for: receiving a text based document in the associated language of the tokenizer; parsing the received document to identify keywords from the associated keyword set occurring in the received document; and generating, as a feature vector, a plurality of ID:score pairs, each pair associating a score with an ID of a keyword in the associated keyword set occurring in the document, the score based on the frequency of occurrence of the ID corresponding to the keyword in the document and a relevance weighting associated with the keyword, wherein the relevance weighting is based on a frequency of occurrence of the keyword in a set of previously collected documents; a language identifier for receiving the document and identifying the language of the document as one of the base language or the second language; and a tokenizer selector for receiving the identified language and selecting a corresponding tokenizer to parse the received document and produce a feature vector of the received document from the generated ID:score pairs.