title: Python regular expression, substituting a multiline match not working
body: <p>I am trying to clean a text from math formulas. I am using a regex to substitute them but it doesn't seem to work for formulas that span to more than one line. </p>
  
  <p>For example, I have this (dirty) string:</p>
  
  <pre><code>tst = """
  The appropriate tool to deal in a statistical mechanics framework with a
  system of quantum numbers latexmath:[$\vec X=(N,S,Q,C,B)$] is the
  canonical partition function (*???*; *???*)
  latexmath:[\[Z_{N,S,Q,C,B} = \frac {1}{(2\pi)^5}\int\limits^{\pi}_{-\pi}
  d^5\vec\phi\;e^{i\vec\phi\vec X} \exp{(\sum_j z_j)},
  \label{eq:partition1}\]] where
  latexmath:[\[z_j= g_j\frac{V}{(2\pi)^3}\int d^3p\;\ln(1\pm
  \exp{(-\sqrt{p^2+m_j^2}/T -i\vec x_j\vec \phi)})^{\pm 1},
  \label{eq:partition1b}\]] 
  """
  </code></pre>
  
  <p>This is the regular expression that I have to match the formulas:</p>
  
  <pre><code>regex = r"(fm|cm)?latexmath:\[.+?\]\]?"
  </code></pre>
  
  <p>I have also tested it in <a href="https://regex101.com" rel="nofollow noreferrer">regex101</a> and it seems to work.</p>
  
  <p>I use <code>re.DOTALL</code> to match multilines. It can replace the formula that does not span between two lines but not the rest. However, when I use <code>finditer</code> I see that it also finds the multiline expression.</p>
  
  <p>Here's the result using <code>re.finditer</code>. The re.DOTALL flag works here:</p>
  
  <pre><code>In [48]: for match in re.finditer(regex, tst):
      ...:     print(repr(match.group(0)))
      ...:     
  latexmath:[$\vec X=(N,S,Q,C,B)$]
  
  In [49]: for match in re.finditer(regex, tst, re.DOTALL):
      ...:     print(repr(match.group(0)))
      ...:     
      ...:     
  'latexmath:[$\\vec X=(N,S,Q,C,B)$]'
  'latexmath:[\\[Z_{N,S,Q,C,B} = \\frac {1}{(2\\pi)^5}\\int\\limits^{\\pi}_{-\\pi}\nd^5\\vec\\phi\\;e^{i\\vec\\phi\\vec X} \\exp{(\\sum_j z_j)},\n\\label{eq:partition1}\\]]'
  'latexmath:[\\[z_j= g_j\\frac{V}{(2\\pi)^3}\\int d^3p\\;\\ln(1\\pm\n\\exp{(-\\sqrt{p^2+m_j^2}/T -i\\vec x_j\\vec \\phi)})^{\\pm 1},\n\\label{eq:partition1b}\\]]'
  </code></pre>
  
  <p>However, the same regex does not work with <code>re.sub</code> (it only replaces the first occurrence):</p>
  
  <pre><code>In [50]: re.sub(regex, 'XXXX' , tst, re.DOTALL)
  Out[50]: '\nThe appropriate tool to deal in a statistical mechanics framework with a\nsystem of quantum numbers XXXX is the\ncanonical partition function (*???*; *???*)\nlatexmath:[\\[Z_{N,S,Q,C,B} = \\frac {1}{(2\\pi)^5}\\int\\limits^{\\pi}_{-\\pi}\nd^5\\vec\\phi\\;e^{i\\vec\\phi\\vec X} \\exp{(\\sum_j z_j)},\n\\label{eq:partition1}\\]] where\nlatexmath:[\\[z_j= g_j\\frac{V}{(2\\pi)^3}\\int d^3p\\;\\ln(1\\pm\n\\exp{(-\\sqrt{p^2+m_j^2}/T -i\\vec x_j\\vec \\phi)})^{\\pm 1},\n\\label{eq:partition1b}\\]] '
  
  In [51]: re.sub(regex, 'XXXX' , tst)
  Out[51]: '\nThe appropriate tool to deal in a statistical mechanics framework with a\nsystem of quantum numbers XXXX is the\ncanonical partition function (*???*; *???*)\nlatexmath:[\\[Z_{N,S,Q,C,B} = \\frac {1}{(2\\pi)^5}\\int\\limits^{\\pi}_{-\\pi}\nd^5\\vec\\phi\\;e^{i\\vec\\phi\\vec X} \\exp{(\\sum_j z_j)},\n\\label{eq:partition1}\\]] where\nlatexmath:[\\[z_j= g_j\\frac{V}{(2\\pi)^3}\\int d^3p\\;\\ln(1\\pm\n\\exp{(-\\sqrt{p^2+m_j^2}/T -i\\vec x_j\\vec \\phi)})^{\\pm 1},\n\\label{eq:partition1b}\\]] '
  </code></pre>
  
  <p>I'm not sure why <code>re.sub</code> is not working with multiline expression here.</p>