how to extract high frequency phrase This is the corpus

abcdef
abcd
xabcd
b
bcef
ef
aef


what I want to extract is abcd, ef, then I try to count all n-grams:

abcdef 1
abcd 3
abc 3
...


My question is how to distinguish abcd with abcdef and abc, since abcdef just appear once, and abc is the sub-string of 'abcd'. Of course, I can filter with this two conditions, but just want to know any existing algorithm target on solving this problem.
This uses python and NLTK

import nltk

text = """abcdef
abcd
xabcd
b
bcef
ef
aef
"""

def tokens(lines):
    for line in lines:
        for n in range(1, len(line)):
            ts = nltk.ngrams(line, n)
            for t in ts: yield str.join('', t) 
        yield line

ts = tokens(text.splitlines())
freq = nltk.FreqDist(ts)
print(freq.most_common())
# [('b', 5), ('c', 4), ('ef', 4), ('bc', 4), ('e', 4)...


There are other solutions, of course, but I think this meets your criteria. You may need to optimize if your lines are huge as this will create every possible n-gram.