Searching documents at scale: how to maintain cleaned documents? I have a document-store database (MarkLogic) with hundreds of thousands of news articles in raw format. I am building a content recommender on a representative subset of that data on my local machine. I'm likely going to use Tfidf or Jaccard similarity to recommend documents, but I'm concerned about how I might actually implement whatever recommender algorithm I come up with back into my larger production database, as I will be cleaning raw text data (i.e. removing stopwords, punctuation, stemming, etc.) to build the model (so that, for example, I have one feature representing the word "hello" -- not "Hello", "hello", "hello.", and so on).

How do large search engines like Google and Yahoo! implement these types of algorithms (i.e. algorithms that were built using cleaned data, but must work for raw, uncleaned data)? I can't imagine they maintain two schematically-identical databases - one clean and the other unclean - for these types of problems. I also think that, while Google has insane compute power, they are certainly not cleaning every document in their database each time a query comes in.
Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:


Read in raw documents.
Stem words, remove stop words.
Perform TF-IDF
Train model on cleaned up data.


Now when it comes time to score:


Load your saved model.
Repeat steps #2 and #3 from above.
Send the cleaned data into your model for a prediction/recommendation.


Here is an example in Python of something like that:

http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html