How to replace keywords in dict values in list of dicts (with case insensitivity)?
Question: I have a list of keywords:

    
    
    keywords = ["test", "Ok", "great stuff", "PaaS", "mydata"]
    

And a list of dicts:

    
    
    statements = [
    {"id":"1","text":"Test, this is OK, great stuff, PaaS."},
    {"id":"2","text":"I would like to test this, Great stuff."}
    ]
    

**Desired Behavior**

When the `keyword` is present in `statement['text']` (regardless of case), I
want to replace the keyword with a "marked up" version of the keyword, ie the
matched keyword `Test` would become:

    
    
    <span class="my_class" data-mydata="<a href=&quot;#&quot;>test</a>">Test</span>
    

**What I've Tried**

Below is what I've tried, the observations/considerations being:

**01)** It is not replacing the keywords.

**02)** If it was, once the markup is applied, I do not want matches occuring
within the markup - ie the `mydata` within the markup should not be matched.

**03)** I may have started out in the wrong direction with this, and need to
redesign the logic from the start.

**Python 2.7 Code**

    
    
    import re
    
    keywords = ["test", "ok", "great stuff", "paas"]
    
    statements = [
    {"id":"1","text":"Test, this is OK, great stuff, PaaS."},
    {"id":"2","text":"I would like to test this, Great stuff."}
    ]
    
    keyword_markup = {}
    
    print "\nKEYWORDS (all lowercase):\n"
    
    for i in keywords:
        print "\"" + i + "\" "
    
    print "\nORIGINAL STATEMENTS:\n"
    
    for statement in statements:
        print statement['text'] + "\n"
    
    statement_counter = 1
    # for each statement
    for statement in statements:
        print "\nIN STATEMENT " + str(statement_counter) + ": \n"
        # get the original statement
        original_statement = statement['text']
        # for each keyword in the keyword list
        for keyword in keywords:
            # if the keyword is not in the keyword_markup dict
            # add it (with a lowercase key)
            if keyword.lower() not in keyword_markup:
                keyword_markup[keyword.lower()] = "<span class=\"my_class\" data-mydata=\"<a href=&quot;#&quot;>" + keyword + "</a>\">" + keyword + "</span>"
                print "The key added to the keyword_markup dict is: " + keyword.lower()
            # if the keyword is in a lowercase version of the statement
            if keyword in original_statement.lower():
                # sanity check - print the matched keyword
                print "The keyword matched in the statement is: " + keyword
                # change the text value of the statement "in place"
                # by replacing the keyword, with its marked up equivalent.
                # using the original_statement as the source string
                statement['text'] = re.sub(keyword,keyword_markup[keyword.lower()],original_statement)
        statement_counter += 1
    
    print "\nMARKED UP KEYWORDS AVAILABLE:\n"
    
    for i in keyword_markup:
        print keyword_markup[i]
    
    print "\nNEW STATEMENTS:\n"
    
    for statement in statements:
        print statement['text'] + "\n"
    

**Results**

    
    
    KEYWORDS (all lowercase):
    
    "test" 
    "ok" 
    "great stuff" 
    "paas" 
    
    ORIGINAL STATEMENTS:
    
    Test, this is OK, great stuff, PaaS.
    
    I would like to test this, Great stuff.
    
    
    IN STATEMENT 1: 
    
    The key added to the keyword_markup dict is: test
    The keyword matched in the statement is: test
    The key added to the keyword_markup dict is: ok
    The keyword matched in the statement is: ok
    The key added to the keyword_markup dict is: great stuff
    The keyword matched in the statement is: great stuff
    The key added to the keyword_markup dict is: paas
    The keyword matched in the statement is: paas
    
    IN STATEMENT 2: 
    
    The keyword matched in the statement is: test
    The keyword matched in the statement is: great stuff
    
    MARKED UP KEYWORDS AVAILABLE:
    
    <span class="my_class" data-mydata="<a href=&quot;#&quot;>test</a>">test</span>
    <span class="my_class" data-mydata="<a href=&quot;#&quot;>paas</a>">paas</span>
    <span class="my_class" data-mydata="<a href=&quot;#&quot;>ok</a>">ok</span>
    <span class="my_class" data-mydata="<a href=&quot;#&quot;>great stuff</a>">great stuff</span>
    
    NEW STATEMENTS:
    
    Test, this is OK, great stuff, PaaS.
    
    I would like to test this, Great stuff.
    


Answer: I was able to do this without regular expressions, though re.sub or re.findall
with re.IGNORECASE would be a good place to start (as you discovered) if
that's the direction you want to go.

I also started thinking about writing a one-pass tokenizer, but decided a
multi-pass system was simpler to understand and maintain than some ugly state
machine.

The code below is optimized for readability, not performance.

    
    
    def main():
        keywords = ["test", "ok", "great stuff", "paas"]
    
        statements = [
            {"id":"1","text":"Test, this is OK, great stuff, PaaS."},
            {"id":"2","text":"I would like to test this, Great stuff."}
        ]
    
        for statement in statements:
            m = markup_statement(statement['text'], keywords)
            print('id={}, text={}'.format(statement['id'], m))
    

Produces the following output:

    
    
    id=1, text=<a href="#">Test</a>, this is <a href="#">OK</a>, <a href="#">great stuff</a>, <a href="#">PaaS</a>.
    id=2, text=I would like to <a href="#">test</a> this, <a href="#">Great stuff</a>.
    

Here are the supporting functions:

    
    
    def markup_statement(statement, keywords):
        """Returns a string where keywords in statement are marked up
    
        >>> markup_statement('ThIs is a tEst stAtement', ['is', 'test'])
        'Th<a href="#">Is</a> <a href="#">is</a> a <a href="#">tEst</a> stAtement'
        """
        markedup_statement = []
        keywords_lower = {k.lower() for k in keywords}
        for token in tokenize(statement, keywords):
            if token.lower() in keywords_lower:
                markedup_statement.append(markup(token))
            else:
                markedup_statement.append(token)
        return ''.join(markedup_statement)
    
    def markup(keyword):
        """returns the marked up version of a keyword/token (retains the original case)
        This function provides the same markup regardless of keyword, but it could be
        modified to provide keyword-specific markup
    
        >>> markup("tEst")
        '<a href="#">tEst</a>'
        """
        return '<a href="#">{}</a>'.format(keyword)
    

This tokenizer makes multiple passes over the statement, one pass for each
keyword. The order of keywords may affect the tokens returned by `tokenize`.
For example, if the markup replacement function is `markup = {'at': lambda x:
'@', 'statement': lambda x: '<code>{}</code>'.format(x)}.get` then `'This is a
statement statement'` could either be `'This is a st@ement'` or `'This is a
<code>statement</code>'`.

    
    
    def tokenize(statement, keywords):
        """Adapted from https://docs.python.org/3/library/re.html#writing-a-tokenizer
        Splits statement on keywords
        Assumes that there is no overlap between keywords in statement
    
        >>> tokenize('ThIs is a tEst stAtement', ['is', 'test'])
        ['Th', 'Is', ' ', 'is', ' a ', 'tEst', ' stAtement']
        >>> ''.join(tokenize(statement, keywords)) == statement
        True
        """
        statement_fragments = [statement]
        for keyword in keywords:
            statement_fragments = list(split(statement_fragments, keyword))
        return statement_fragments
    

This is not a particularly fast splitter, but is simple enough to explain the
idea. I could have used `re.split(pattern, string, flags=re.IGNORECASE)` here,
but I avoid regular expressions when vanilla python logic works, since regex
code is seldom readable and not particularly fast.

    
    
    def split(statement_fragments, keyword):
        """Split each statement fragment by keywords
        statement_fragments: list of strings
        keyword: string
        returns list of strings, which may be the same length or longer than statement_fragments
    
        This repeatedly trims and lowercases strings. If it's a bottleneck,
        rewrite it with a start and end index slices
    
        >>> split(['ThIs is a tEst stAtement'], 'is')
        ['Th', 'Is', ' ', 'is', ' a tEst stAtement']
        """
        keyword_lower = keyword.lower()
        length = len(keyword)
        for fragment in statement_fragments:
            i = fragment.lower().find(keyword_lower)
            while i != -1:
                yield fragment[:i]
                yield fragment[i:i+length]
                fragment = fragment[i+length:]
                i = fragment.lower().find(keyword_lower)
            # yield whatever is left over
            yield fragment
    

Without comments, that's around 30 lines of code with no imports.

