1. A processing system for representing documents relative to importance of words in the document, including: a processor comprising a weighting model of word importance of a word w in a document d in a collection relative to an importance of the word w in other documents in the collection by computing a deviation of distribution of the word w from a probability distribution of the word w in other documents in the collection, wherein the deviation distribution is weighted in accordance with a concavity function comprising a selectively variable concavity control parameter η for setting a curvature of a logarithm of the concavity function and wherein the concavity control parameter η is effected relative to a probable frequency of the word w in the document d according to −ln η P(T w ≧t wd |λ w ) where T w is a continuous random variable modeling normalized term frequencies for a probability distribution parameterized by a set of parameters λ w and t wd represents frequency of the word w in the document d.
1. A processing system for representing documents relative to importance of words in the document, including: a processor comprising a weighting model of word importance of a word w in a document d in a collection relative to an importance of the word w in other documents in the collection by computing a deviation of distribution of the word w from a probability distribution of the word w in other documents in the collection, wherein the deviation distribution is weighted in accordance with a concavity function comprising a selectively variable concavity control parameter η for setting a curvature of a logarithm of the concavity function and wherein the concavity control parameter η is effected relative to a probable frequency of the word w in the document d according to −ln η P(T w ≧t wd |λ w ) where T w is a continuous random variable modeling normalized term frequencies for a probability distribution parameterized by a set of parameters λ w and t wd represents frequency of the word w in the document d. 3. The system of claim 1 , wherein the concavity control parameter η is varied in accordance with predetermined standards to better effect text classification or latent semantic indexing.