Judgments of similarity between samples of writing
It sounds like you're talking about Latent Semantic Analysis. Here's their rundown of what it is.


  However, LSA as currently practiced has some additional limitations. It makes no use of word order, thus of syntactic relations or logic, or of morphology. Remarkably, it manages to extract correct reflections of passage and word meanings quite well without these aids, but it must still be suspected of incompleteness or likely error on some occasions.


It's used in a lot of automated grading programs, or systems designed to give a score to paper based on some criteria. However, because order doesn't matter, it makes syntactic and grammatical coherence much harder to measure, so quantifying 'style' might be a little hard, unless by 'style' you mean something like word choice, phrase choice, etc. 

Here's a gscholar search. The first three should give a good overview.