regular expression execution time jumps sharply after 100th pattern
<p>I started with the first code snippet below to search a list of lines and convert all keywords (identified in a separate list) in each line to lower case. For my test list of lines about 800 lines long, the keyword substitution for the entire list of lines takes less than a second as long as my keyword list is 100 items or fewer. When I extend the list to 101 items or more, the processing time jumps to over 9 seconds.</p>

<p>Using the second snippet (where all the patterns for the keyword list are compiled) drops the total processing time back down below 1 second.</p>

<p>Does anyone know why the processing time for the non-compiled substitution search is so sensitive to the number of items searched per input line? I'm surprised it jumps so sharply after 100 keywords.</p>

<p>snippet #1</p>

<pre><code>for line in lines_in:
    for keyword in keywords:
        rawstring = r'\b' + keyword + r'\b'
        line = re.sub(rawstring, keyword, line, 0, re.IGNORECASE)
</code></pre>

<p>snippet #2</p>

<pre><code>for i in range(len(keywords)):
    re_pattern = re.compile(r'\b' + keywords[i] + r'\b', re.IGNORECASE)
    pattern.append(re_pattern)

for line in lines_in:
    for i in range(len(keywords)):
        line = pattern[i].sub(keywords[i], line, 0)
</code></pre>
