Python: Finding unknown repeated word(s) in a list of strings
Question: I have a list of strings, which are subjects from different email
conversations. I would like to see if there are words or word combinations
which are being used frequently.

An example list would be:

    
    
    subjects = [
                  'Proposal to cooperate - Company Name',
                  'Company Name Introduction',
                  'Into Other Firm / Company Name',
                  'Request for Proposal'
               ]
    

The function would have to detect that "Company Name" as combination is used
more than once, and that "Proposal" is being used more than once. These words
won't be known in advance though, so I guess it would have to start trying all
possible combinations.

The actual list is of course a lot longer than this example, so manually
trying all combinations doesn't seem like the best way to go. What would be
the best way to go about this?

**UPDATE**

I've used Tim Pietzcker's answer to start developing a function for this, but
I get stuck on applying the Counter correctly. It keeps returning the length
of the list as count for all phrases.

The phrases function, including punctuation filter and a check if this phrase
has already been checked, and a max length per phrase of 3 words:

    
    
    def phrases(string, phrase_list):
      words = string.split()
      result = []
      punctuation = '\'\"-_,.:;!? '
      for number in range(len(words)):
          for start in range(len(words)-number):
            if number+1 <= 3:
              phrase = " ".join(words[start:start+number+1])
              if phrase in phrase_list:
                pass
              else:
                phrase_list.append(phrase)
                phrase = phrase.strip(punctuation).lower()
                if phrase:
                   result.append(phrase)
      return result, phrase_list
    

And then the loop through the list of subjects:

    
    
    phrase_list = []
    ranking = {}
    for s in subjects:
        result, phrase_list = phrases(s, phrase_list)
        all_phrases = collections.Counter(phrase.lower() for s in subjects for phrase in result)
    

"all_phrases" returns a list with tuples where each count value is 167, which
is the length of the subject list I'm using. Not sure what I'm missing here...


Answer: You also want to find phrases that are composed of more than single words. No
problem. This should even scale quite well.

    
    
    import collections
    
    subjects = [
                  'Proposal to cooperate - Company Name',
                  'Company Name Introduction',
                  'Into Other Firm / Company Name',
                  'Request for Proposal',
                  'Some more Firm / Company Names'
               ]
    
    def phrases(string):
        words = string.split()
        result = []
        for number in range(len(words)):
            for start in range(len(words)-number):
                 result.append(" ".join(words[start:start+number+1]))
        return result
    

The function `phrases()` splits the input string on whitespace and returns all
possible substrings of any length:

    
    
    In [2]: phrases("A Day in the Life")
    Out[2]:
    ['A',
     'Day',
     'in',
     'the',
     'Life',
     'A Day',
     'Day in',
     'in the',
     'the Life',
     'A Day in',
     'Day in the',
     'in the Life',
     'A Day in the',
     'Day in the Life',
     'A Day in the Life']
    

Now you can count how many times each of these phrases are found in all your
subjects:

    
    
    all_phrases = collections.Counter(phrase for subject in subjects for phrase in phrases(subject))
    

Result:

    
    
    In [3]: print([(phrase, count) for phrase, count in all_phrases.items() if count > 1])
    Out [3]:
    [('Company', 4), ('Proposal', 2), ('Firm', 2), ('Name', 3), ('Company Name', 3), 
     ('Firm /', 2), ('/', 2), ('/ Company', 2), ('Firm / Company', 2)]
    

Note that you might want to use other criteria than simply splitting on
whitespace, maybe ignore punctuation and case etc.

