How to separate words that are together in a large data set in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'

i tried this but this is slow

def split(sentence, word_to_index):
# words to index dictionary of valid words
    sentence_words = sentence.split()
    lst12=[] 
    for w in sentence_words:
            lst=[] 
            ws=w[:]
            # Set the (i,j)th entry of X_indices to the index of the correct word.
            while True:
                for wd in lst:
                    ws = ws.replace(wd,"")
                if ws=="":
                    lst=" ".join(lst)
                    lst12.append(lst)
                    break
                wrdind = ws[:]
                for kl in range(len(ws)):
                    try:
                        word_to_index[wrdind]
                        lst.append(wrdind)
                        break     

                    except KeyError:
                            wrdind = wrdind[:len(wrdind)-1]
                            continue
   return " ".join(lst12)  
```
This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can split the entire string.

Here is a version (inspired by this answer):

from functools import lru_cache

@lru_cache(None)
def wordbreak(string):
    if len(string) &lt; 1:
        return " "
    else:
        for i in range(len(string)):
            substring = string[:i+1]
            if substring in valid_words:
                remaining_split = wordbreak(string[i+1:])
                if len(remaining_split) &gt; 0:
                    return substring + " " + remaining_split
                else:
                    return ""
            if (i == len(string) - 1) and (substring not in valid_words):
                return ""

valid_words = {'man', 'boycott', 'bears'}                                            
assert wordbreak('boycottbears') == 'boycott bears'
assert not wordbreak('abc')


However, creating a complete collection of valid words is impossible for Twitter data since there are always new words.