title: Python regex replacing string that should not match
body: <p><strong>Update</strong>: This issue was caused by a bug in the <code>regex</code> module which was resolved by the developer in <strong><a href="https://bitbucket.org/mrabarnett/mrab-regex/commits/be893e9f7e60946eb8cdec66e5e105f369d9437b" rel="nofollow noreferrer">commit be893e9</a></strong> </p>
  
  <p>If you encounter a similiar problem, update your <code>regex</code> module. <br>
  You need version <code>2017.04.23</code> or above.</p>
  
  <p><strong><a href="https://stackoverflow.com/questions/43563819/python-regex-vs-re-pattern-mismatch">See here</a></strong> for further information.</p>
  
  <hr>
  
  <p>Background: I'm using a collection of regular expression (<code>english.lex</code>) in a 3rd party Text2Speech engine to normalize the input text before speaking it.</p>
  
  <p>For debugging purposes, I wrote the script below to see what impact my regex collection actually has on the input text.</p>
  
  <p><strong>My problem is that it's replacing a regex that <a href="https://regex101.com/r/n6OCBc/2" rel="nofollow noreferrer">simply does not match</a></strong></p>
  
  <hr>
  
  <h2>I have 3 files:</h2>
  
  <p><strong>regex_preview.py</strong></p>
  
  <pre><code>#!/usr/bin/env python
  import codecs
  import regex as re
  
  input="Text2Speach Regex Test.txt"
  dictionary="english.lex"
  
  with codecs.open(dictionary, "r", "utf16") as f:
      reg_exen = f.readlines()
      with codecs.open(input, "r+", "utf16") as g:
          content = g.read().replace(r'\\\\\"','"')
  
          # apply all regular expressions to content
          for line in reg_exen:
              line=line.strip()
  
              # skip comments
              if line == "" or line[0] == "#":
                  pass
              else:
                  # remove " from lines and split them into pattern and substitue
                  pattern=re.sub('" "(.*[^\\\\])?"$','', line)[1:].replace('\\"','"')
                  substitute=re.sub('\\\\"', '"', re.sub('^".*[^\\\\]" "', '', line)[:-1]).replace('\\"','"')
  
                  print("\n'%s' ==&gt; '%s'" % (pattern, substitute))
  
                  print(content.strip())
                  content = re.sub(pattern, substitute, content)
                  print(content.strip())
  </code></pre>
  
  <p><strong>english.lex</strong> - utf16 encoded</p>
  
  <pre><code># punctuation normalization
  "(《|》|⟪|⟫|&lt;|&gt;|«|»|”|“|″|‴)+" "\""
  "(…|—)" "..."
  
  # stammered words: more general version accepting all words like ab... abcde (stammered words with vocal in stammered part)
  "(?i)(?&lt;=\b)(?:(\w{1,3})(?:-|\.{2,10})[\t\f ]?)+(\1\w{2,})" "\1-\2"
  # this should not match, but somehow it does o.O
  </code></pre>
  
  <p><strong>Text2Speach Regex Test.txt</strong> - utf16 encoded</p>
  
  <pre><code>“Erm….yes. Thank you for that.”
  </code></pre>
  
  <hr>
  
  <p>Running the script produces this output with the last regex somehow matching the content:</p>
  
  <pre><code>'(《|》|⟪|⟫|&lt;|&gt;|«|»|”|“|″|‴)+' ==&gt; '"'
  “Erm….yes. Thank you for that.”
  "Erm….yes. Thank you for that."
  
  '(…|—)' ==&gt; '...'
  "Erm….yes. Thank you for that."
  "Erm....yes. Thank you for that."
  
  '(?i)(?&lt;=\b)(?:(\w{1,3})(?:-|\.{2,10})[\t ]?)+(\1\w{2,})' ==&gt; '\1-\2'
  "Erm....yes. Thank you for that."
  "-yes. Thank you for that."
  </code></pre>
  
  <hr>
  
  <h2>What I tried so far:</h2>
  
  <p>I created this snipped to reproduce the issue:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import re
  import codecs
  
  content = u'"Erm....yes. Thank you for that."\n'
  pattern = r"(?i)(?&lt;=\b)(?:(\w{1,3})(?:-|\.{2,10})[\t ]?)+(\1\w{2,})"
  substitute = r"\1-\2"
  content = re.sub(pattern, substitute, content)
  
  print(content)
  </code></pre>
  
  <p>But this actually behaves like it should. So I'm at a loss at what's happening here.</p>
  
  <p>Hopefully, someone can point me in the right direction for further investigation...</p>