Python: tf-idf-cosine: to find document similarity
Question: I was following a tutorial which was available at [Part
1](http://blog.christianperone.com/?p=1589) & [Part
2](http://blog.christianperone.com/?p=1747) unfortunately author didn't have
time for the final section which involves using cosine to actually find the
similarity between two documents. I followed the examples in the article with
the help of following link from
[stackoverflow](http://stackoverflow.com/questions/11911469/tfidf-for-search-
queries) I have included the code that is mentioned in the above link just to
make answers life easy.

    
    
    from sklearn.feature_extraction.text import CountVectorizer
    from sklearn.feature_extraction.text import TfidfTransformer
    from nltk.corpus import stopwords
    import numpy as np
    import numpy.linalg as LA
    
    train_set = ["The sky is blue.", "The sun is bright."] #Documents
    test_set = ["The sun in the sky is bright."] #Query
    stopWords = stopwords.words('english')
    
    vectorizer = CountVectorizer(stop_words = stopWords)
    #print vectorizer
    transformer = TfidfTransformer()
    #print transformer
    
    trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
    testVectorizerArray = vectorizer.transform(test_set).toarray()
    print 'Fit Vectorizer to train set', trainVectorizerArray
    print 'Transform Vectorizer to test set', testVectorizerArray
    
    transformer.fit(trainVectorizerArray)
    print
    print transformer.transform(trainVectorizerArray).toarray()
    
    transformer.fit(testVectorizerArray)
    print 
    tfidf = transformer.transform(testVectorizerArray)
    print tfidf.todense()
    

as a result of above code I have following matrix

    
    
    Fit Vectorizer to train set [[1 0 1 0]
     [0 1 0 1]]
    Transform Vectorizer to test set [[0 1 1 1]]
    
    [[ 0.70710678  0.          0.70710678  0.        ]
     [ 0.          0.70710678  0.          0.70710678]]
    
    [[ 0.          0.57735027  0.57735027  0.57735027]]
    

I am not sure how to use this output to calculate cosine similarity, I know
how to implement cosine similarity respect to two vectors with similar length
but here I am not sure how to identify the two vectors.


Answer: First off, if you want to extract count features and apply TF-IDF
normalization and row-wise euclidean normalization you can do it in one
operation with `TfidfVectorizer`:

    
    
    >>> from sklearn.feature_extraction.text import TfidfVectorizer
    >>> from sklearn.datasets import fetch_20newsgroups
    >>> twenty = fetch_20newsgroups()
    
    >>> tfidf = TfidfVectorizer().fit_transform(twenty.data)
    >>> tfidf
    <11314x130088 sparse matrix of type '<type 'numpy.float64'>'
        with 1787553 stored elements in Compressed Sparse Row format>
    

Now to find the cosine distances of one document (e.g. the first in the
dataset) and all of the others you just need to compute the dot products of
the first vector with all of the others as the tfidf vectors are already row-
normalized. The scipy sparse matrix API is a bit weird (not as flexible as
dense N-dimensional numpy arrays). To get the first vector you need to slice
the matrix row-wise to get a submatrix with a single row:

    
    
    >>> tfidf[0:1]
    <1x130088 sparse matrix of type '<type 'numpy.float64'>'
        with 89 stored elements in Compressed Sparse Row format>
    

scikit-learn already provides pairwise metrics (a.k.a. kernels in machine
learning parlance) that work for both dense and sparse representations of
vector collections. In this case we need a dot product that is also known as
the linear kernel:

    
    
    >>> from sklearn.metrics.pairwise import linear_kernel
    >>> cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
    >>> cosine_similarities
    array([ 1.        ,  0.04405952,  0.11016969, ...,  0.04433602,
        0.04457106,  0.03293218])
    

Hence to find the top 5 related documents, we can use `argsort` and some
negative array slicing (most related documents have highest cosine similarity
values, hence at the end of the sorted indices array):

    
    
    >>> related_docs_indices = cosine_similarities.argsort()[:-5:-1]
    >>> related_docs_indices
    array([    0,   958, 10576,  3277])
    >>> cosine_similarities[related_docs_indices]
    array([ 1.        ,  0.54967926,  0.32902194,  0.2825788 ])
    

The first result is a sanity check: we find the query document as the most
similar document with a cosine similarity score of 1 which has the following
text:

    
    
    >>> print twenty.data[0]
    From: lerxst@wam.umd.edu (where's my thing)
    Subject: WHAT car is this!?
    Nntp-Posting-Host: rac3.wam.umd.edu
    Organization: University of Maryland, College Park
    Lines: 15
    
     I was wondering if anyone out there could enlighten me on this car I saw
    the other day. It was a 2-door sports car, looked to be from the late 60s/
    early 70s. It was called a Bricklin. The doors were really small. In addition,
    the front bumper was separate from the rest of the body. This is
    all I know. If anyone can tellme a model name, engine specs, years
    of production, where this car is made, history, or whatever info you
    have on this funky looking car, please e-mail.
    
    Thanks,
    - IL
       ---- brought to you by your neighborhood Lerxst ----
    

The second most similar document is a reply that quotes the original message
hence has many common words:

    
    
    >>> print twenty.data[958]
    From: rseymour@reed.edu (Robert Seymour)
    Subject: Re: WHAT car is this!?
    Article-I.D.: reed.1993Apr21.032905.29286
    Reply-To: rseymour@reed.edu
    Organization: Reed College, Portland, OR
    Lines: 26
    
    In article <1993Apr20.174246.14375@wam.umd.edu> lerxst@wam.umd.edu (where's my
    thing) writes:
    >
    >  I was wondering if anyone out there could enlighten me on this car I saw
    > the other day. It was a 2-door sports car, looked to be from the late 60s/
    > early 70s. It was called a Bricklin. The doors were really small. In
    addition,
    > the front bumper was separate from the rest of the body. This is
    > all I know. If anyone can tellme a model name, engine specs, years
    > of production, where this car is made, history, or whatever info you
    > have on this funky looking car, please e-mail.
    
    Bricklins were manufactured in the 70s with engines from Ford. They are rather
    odd looking with the encased front bumper. There aren't a lot of them around,
    but Hemmings (Motor News) ususally has ten or so listed. Basically, they are a
    performance Ford with new styling slapped on top.
    
    >    ---- brought to you by your neighborhood Lerxst ----
    
    Rush fan?
    
    --
    Robert Seymour              rseymour@reed.edu
    Physics and Philosophy, Reed College    (NeXTmail accepted)
    Artificial Life Project         Reed College
    Reed Solar Energy Project (SolTrain)    Portland, OR
    

