Python-how to calculate the highest tf-idf value of the first 100 words in different tweeets
Question: I have tens of thounds of tweets saved in one .txt file, I want to calculate
calculate the highest tf-idf value of the first 100 words in these tweeets, in
other words, I want to compare the word's tf-idf value between different
tweets, presently,the only thing that I could complete is comparing word's tf-
idf value in the same tweets, I cannot find a way to compare word's tf-idf
value between different tweets.

Please help me,I have been upset for a long time because of this problem.
/(ㄒoㄒ)/~~

Blow is my code:(only able to calculate the term's tfidf value in same tweets)

    
    
    with open('D:/Data/ows/ows_sample.txt','rb') as f:
        tweet=f.readlines()
    lines = csv.reader((line.replace('\x00','') for line in tweet), delimiter=',', quotechar='"')
    wordterm=[]
    for i in lines:
        i[1]= re.sub(r'http[s]?://(?:[a-z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-f][0-9a-f]))+|(?:@[\w_]+)', "", i[1])
        tweets=re.split(r"\W+",i[1])
        tweets=[w.lower() for w in tweets if w!=""]
        stopwords = open("D:/Data/ows/stopwords.txt", "r").read().split()
        terms = [t for t in tweets if not t in stopwords]
        wordterm.append(terms)
    
    word=[' '.join(t) for t in wordterm]
    tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
    tfidf_matrix = tfidf_vectorizer.fit_transform(word)
    terms_name = tfidf_vectorizer.get_feature_names()
    toarry=tfidf_matrix.todense()
    
    #below code will output the tf-idf value of each tweets' terms.
    for ii in range(0,len(toarry)):
        print u"第"+ ii +u"个tweets"
        for jj in range(0,len(terms_name)):
            print terms_name[jj],'-',tfidf_matrix[ii,jj]
    


Answer: Now that I understand your question, I will try to answer your question a
little better.

To get the top 100 'tf-idf' scores in a way that is comparable across all
tweets would either mean that you are letting go of the notion that there are
distinct tweets, or you want to be able to compare the same words to each
other by tf-idf score.

So for the first scenario, imagine that all your words are in 1 'document'.
This would essentially eliminate the 'idf' aspect of tf-idf, and what you'll
get is basically a word count vectorizer, which can be compared with one
another and you can get the top 100 words this way.

    
    
    words = ['the cat sat on the mat cat cat']
    tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
    tfidf_matrix = tfidf_vectorizer.fit_transform(words)
    terms_name = tfidf_vectorizer.get_feature_names()
    toarry=tfidf_matrix.todense()
    
    toarry:
        matrix([ .75,  0.25,  0.25,  0.25,  0.5])
    

The other scenario is that you take each tweet separately, and then you
compare the scores by their tf-idf scores. This would result in the same words
having different scores, because that's what tf-idf does - **it calculates the
importance of the word in the document relative to the corpus**.

    
    
    words = ['the cat sat on the mat cat', 'the fat rat sat on a mat', 'the bat and a rat sat on a mat']
    tfidf_vectorizer = TfidfVectorizer(min_df = 1,use_idf=True)
    tfidf_matrix = tfidf_vectorizer.fit_transform(words)
    terms_name = tfidf_vectorizer.get_feature_names()
    toarry=tfidf_matrix.todense()
    for i in tfidf_matrix.toarray():
        print zip(terms_name, i)
    
    [(u'and', 0.0), (u'bat', 0.0), (u'cat', 0.78800079617844954), (u'fat', 0.0), (u'mat', 0.23270298212286766), (u'on', 0.23270298212286766), (u'rat', 0.0), (u'sat', 0.23270298212286766), (u'the', 0.46540596424573533)]
    [(u'and', 0.0), (u'bat', 0.0), (u'cat', 0.0), (u'fat', 0.57989687146162439), (u'mat', 0.34249643393071422), (u'on', 0.34249643393071422), (u'rat', 0.44102651785124652), (u'sat', 0.34249643393071422), (u'the', 0.34249643393071422)]
    [(u'and', 0.50165133177159349), (u'bat', 0.50165133177159349), (u'cat', 0.0), (u'fat', 0.0), (u'mat', 0.29628335772067432), (u'on', 0.29628335772067432), (u'rat', 0.38151876810273028), (u'sat', 0.29628335772067432), (u'the', 0.29628335772067432)]
    

As you can see in the results, the same words will have different scores in
each document, because tf-idf is a score of that term within each document. So
these are the two methods available to you, so depending on what you want, you
can choose what's better for your purposes.

