title: Correctly accounting for multiple backslashes when tokenizing custom mini-format
body: <p>I am writing a small tokenizer in Python for a custom mini-format which looks like this (it can be nested too):</p>
  
  <pre><code>&lt;tag:some_text&gt;
  </code></pre>
  
  <p><code>tag</code> is a combination of a finite set of values and <code>some_text</code> is just text. The delimiters <code>&lt;</code>, <code>:</code> and <code>&gt;</code> can be escaped by a single <code>\</code> if they appear in, and as, text.</p>
  
  <p>I used the regex <code>r"((\\)?[&lt;:&gt;])"</code> along with <code>re.finditer</code> to find delimiters and then remove the backslash if necessary by checking with <code>token.startswith('\\')</code>. The problem is that if more backslashes come before the delimiter, the regex is wrong, e.g. <code>"&lt;tag:Some \\\\&lt; text&gt;" -&gt; ['&lt;', 'tag', ':', 'Some \\\\', '&lt;', ' text', '&gt;']</code>.</p>
  
  <p>I cannot find a sensible solution using regexes, and I am considering just writing the tokenization in pure Python, i.e. no regex magic etc (but that may be slow?) or am I overcomplicating this? Any suggestions?</p>