How to count number of occurences of permutation (overlapping) in large text in python3?
Question: I am having a list of words and I'd like to find out how many time each
permutation occurs in this list of word. And I'd like to count overlapping
permutation also. So count() doesn't seem to be appropriate. for example: the
permutation aba appears twice in this string:

ababa

However count() would say one.

So I designed this little script, but I am not too sure that is efficient. The
array of word is an external file, I just removed this part to make it
simplier.

    
    
    import itertools
    import itertools
    
    
    
    #Occurence counting function
    def occ(string, sub):
        count = start = 0
        while True:
            start = string.find(sub, start) + 1
            if start > 0:
                count+=1
            else:
                return count
    
    
    #permutation generator
    abc="ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    permut = [''.join(p) for p in itertools.product(abc,repeat=2)]
    
    
    #Transform osd7 in array
    
    arrayofWords=['word1',"word2","word3","word4"]
    
    
    dict_output['total']=0
    
    #create the array
    for perm in permut:
        dict_output[perm]=0
    
    #iterate over the arrayofWords and permutation
    for word in arrayofWords:
        for perm in permut:
            dict_output[perm]=dict_output[perm]+occ(word,perm)
            dict_output['total']=dict_output['total']+occ(word,perm)
    

It is working, but it takes looonnnggg time. If I change,
product(abc,repeat=2) by product(abc,repeat=3) or product(abc,repeat=4)... It
will take a full week!

**The question: Is there a more efficient way?**


Answer: Very simple: count only what you need to count.

    
    
    from collections import defaultdict
    
    quadrigrams = defaultdict(lambda: 0)    
    for word in arrayofWords:
        for i in range(len(word) - 3):
            quadrigrams[word[i:i+4]] += 1
    

