Using a function as argument to re.sub in Python?
Question: I'm writing a program to split the words contained in an hashtag.

For example I want to split the hashtags:

    
    
    #Whatthehello #goback
    

into:

    
    
    What the hello go back
    

I'm having troubles when using
[`re.sub`](https://docs.python.org/3.5/library/re.html#re.sub) with a
functional argument.

The code I've written is:

    
    
    import re,pdb
    
    def func_replace(each_func):
        i=0
        wordsineach_func=[] 
        while len(each_func) >0:
            i=i+1
            word_found=longest_word(each_func)
            if len(word_found)>0:
                wordsineach_func.append(word_found)
                each_func=each_func.replace(word_found,"")
        return ' '.join(wordsineach_func)
    
    def longest_word(phrase):
        phrase_length=len(phrase)
        words_found=[];index=0
        outerstring=""
        while index < phrase_length:
            outerstring=outerstring+phrase[index]
            index=index+1
            if outerstring in words or outerstring.lower() in words:
                words_found.append(outerstring)
        if len(words_found) ==0:
            words_found.append(phrase)
        return max(words_found, key=len)        
    
    words=[]
    # The file corncob_lowercase.txt contains a list of dictionary words
    with open('corncob_lowercase.txt') as f:
        read_words=f.readlines()
    
    for read_word in read_words:
        words.append(read_word.replace("\n","").replace("\r",""))
    

For example when using these functions like this:

    
    
    s="#Whatthehello #goback"
    
    #checking if the function is able to segment words
    hashtags=re.findall(r"#(\w+)", s)
    print func_replace(hashtags[0])
    
    # using the function for re.sub
    print re.sub(r"#(\w+)", lambda m: func_replace(m.group()), s)
    

The output I obtain is:

    
    
    What the hello
    #Whatthehello #goback
    

Which is not the output I had expected:

    
    
    What the hello
    What the hello go back
    

Why is this happening? In particular I've used the suggestion from [this
answer](http://stackoverflow.com/a/18737964/3646408) but I don't understand
what goes wrong in this code.


Answer: Notice that `m.group()` returns the entire string that matched, whether or not
it was part of a capturing group:

    
    
    In [19]: m = re.search(r"#(\w+)", s)
    
    In [20]: m.group()
    Out[20]: '#Whatthehello'
    

`m.group(0)` also returns the entire match:

    
    
    In [23]: m.group(0)
    Out[23]: '#Whatthehello'
    

In contrast, `m.groups()` returns all capturing groups:

    
    
    In [21]: m.groups()
    Out[21]: ('Whatthehello',)
    

and `m.group(1)` returns the first capturing group:

    
    
    In [22]: m.group(1)
    Out[22]: 'Whatthehello'
    

So the problem in your code originates with the use of `m.group` in

    
    
    re.sub(r"#(\w+)", lambda m: func_replace(m.group()), s)
    

since

    
    
    In [7]: re.search(r"#(\w+)", s).group()
    Out[7]: '#Whatthehello'
    

whereas if you had used `.group(1)`, you would have gotten

    
    
    In [24]: re.search(r"#(\w+)", s).group(1)
    Out[24]: 'Whatthehello'
    

and the preceding `#` makes all the difference:

    
    
    In [25]: func_replace('#Whatthehello')
    Out[25]: '#Whatthehello'
    
    In [26]: func_replace('Whatthehello')
    Out[26]: 'What the hello'
    

Thus, changing `m.group()` to `m.group(1)`, and substituting
`/usr/share/dict/words` for `corncob_lowercase.txt`,

    
    
    import re
    
    def func_replace(each_func):
        i = 0
        wordsineach_func = []
        while len(each_func) > 0:
            i = i + 1
            word_found = longest_word(each_func)
            if len(word_found) > 0:
                wordsineach_func.append(word_found)
                each_func = each_func.replace(word_found, "")
        return ' '.join(wordsineach_func)
    
    
    def longest_word(phrase):
        phrase_length = len(phrase)
        words_found = []
        index = 0
        outerstring = ""
        while index < phrase_length:
            outerstring = outerstring + phrase[index]
            index = index + 1
            if outerstring in words or outerstring.lower() in words:
                words_found.append(outerstring)
        if len(words_found) == 0:
            words_found.append(phrase)
        return max(words_found, key=len)
    
    words = []
    # corncob_lowercase.txt contains a list of dictionary words
    with open('/usr/share/dict/words', 'rb') as f:
        for read_word in f:
            words.append(read_word.strip())
    s = "#Whatthehello #goback"
    hashtags = re.findall(r"#(\w+)", s)
    print func_replace(hashtags[0])
    print re.sub(r"#(\w+)", lambda m: func_replace(m.group(1)), s)
    

prints

    
    
    What the hello
    What the hello gob a c k
    

since, alas, `'gob'` is longer than `'go'`.

* * *

One way you could have debugged this is to replace the `lambda` function with
a regular function and then add print statements:

    
    
    def foo(m):
        result = func_replace(m.group())
        print(m.group(), result)
        return result
    
    In [35]: re.sub(r"#(\w+)", foo, s)
    ('#Whatthehello', '#Whatthehello')   <-- This shows you what `m.group()` and `func_replace(m.group())` returns
    ('#goback', '#goback')
    Out[35]: '#Whatthehello #goback'
    

That would focus your attention on

    
    
    In [25]: func_replace('#Whatthehello')
    Out[25]: '#Whatthehello'
    

which you could then compare with

    
    
    In [26]: func_replace(hashtags[0])
    Out[26]: 'What the hello'
    
    In [27]: func_replace('Whatthehello')
    Out[27]: 'What the hello'
    

That would lead you to ask the question, if `m.group()` returns
`'#Whatthehello'`, what method do I need to return `'Whatthehello'`. A dive
into [the docs](https://docs.python.org/3/library/re.html#re.match.group) then
solves the problem.

