title: Most efficient way to split strings in Python
body: <p>My current Python project will require a lot of string splitting to process incoming packages. Since I will be running it on a pretty slow system, I was wondering what the most efficient way to go about this would be. The strings would be formatted something like this:</p>
  <pre><code>Item 1 | Item 2 | Item 3 &lt;&gt; Item 4 &lt;&gt; Item 5
  </code></pre>
  <p>Explanation: This particular example would come from a list where the first two items are a title and a date, while item 3 to item 5 would be invited people (the number of those can be anything from zero to n, where n is the number of registered users on the server).</p>
  <p>From what I see, I have the following options:</p>
  <ol>
  <li>repeatedly use <code>split()</code></li>
  <li>Use a regular expression and regular expression functions</li>
  <li>Some other Python functions I have not thought about yet (there are probably some)</li>
  </ol>
  <p>Solution 1 would include splitting at <code>|</code> and then splitting the last element of the resulting list at <code>&lt;&gt;</code> for this example, while solution 2 would probably result in a regular expression like:</p>
  <p><code>((.+)|)+((.+)(&lt;&gt;)?)+</code></p>
  <p>Okay, this regular expression is horrible, I can see that myself. It is also untested. But you get the idea.</p>
  <p>Now, I am looking for the way that a) takes the least amount of time and b) ideally uses the least amount of memory. If only one of the two is possible, I would prefer less time. The ideal solution would also work for strings that have more items separated with <code>|</code> and strings that completely lack the <code>&lt;&gt;</code>. At least the regular expression-based solution would do that.</p>
  <p>My understanding would be that <code>split()</code> would use more memory (since you basically get two resulting lists, one that splits at <code>|</code> and the second one that splits at <code>&lt;&gt;</code>), but I don't know enough about Python's implementation of regular expressions to judge how the regular expression would perform. <code>split()</code> is also less dynamic than a regular expression if it comes to different numbers of items and the absence of the second separator. Still, I can't shake the impression that Python can do this better without regular expressions, and that's why I am asking.</p>
  <p>Some notes:</p>
  <ul>
  <li>Yes, I could just benchmark both solutions, but I'm trying to learn something about Python in general and how it works here, and if I just benchmark these two, I still don't know what Python functions I have missed.</li>
  <li>Yes, optimizing at this level is only really required for high-performance stuff, but as I said, I am trying to learn things about Python.</li>
  <li><strong>Addition:</strong> in the original question, I completely forgot to mention that I need to be able to distinguish the parts that were separated by <code>|</code> from the parts with the separator <code>&lt;&gt;</code>, so a simple flat list as generated by <code>re.split(\||&lt;&gt;,input)</code> (as <a href="https://stackoverflow.com/questions/9602856/most-efficient-way-to-split-strings-in-python/9603035#9603035">proposed by obmarg</a>) will not work too well. Solutions fitting this criterium are much appreciated.</li>
  </ul>
  <p>To sum the question up: Which solution would be the most efficient one, for what reasons?</p>
  <p>Due to multiple requests, I have run some <em>timeit</em> on the <code>split()</code>-solution and the first proposed regular expression by obmarg, as well as the <a href="https://stackoverflow.com/questions/9602856/most-efficient-way-to-split-strings-in-python/9603443#9603443">solutions by mgibsonbr</a> <a href="https://stackoverflow.com/questions/9602856/most-efficient-way-to-split-strings-in-python/9604407#9604407">and duncan</a>:</p>
  <pre><code>import timeit
  import re
  
  def splitit(input):
      res0 = input.split(&quot;|&quot;)
      res = []
      for element in res0:
          t = element.split(&quot;&lt;&gt;&quot;)
          if t != [element]:
              res0.remove(element)
              res.append(t)
      return (res0, res)
  
  def regexit(input):
      return re.split( &quot;\||&lt;&gt;&quot;, input )
  
  
  def mgibsonbr(input): # Solution by mgibsonbr
      items = re.split(r'\||&lt;&gt;', input) # Split input in items
      offset = 0
      result = [] # The result: strings for regular items, lists for &lt;&gt; separated ones
      acc = None
      for i in items:
          delimiter = '|' if offset+len(i) &lt; len(input) and input[offset+len(i)] == '|' else '&lt;&gt;'
          offset += len(i) + len(delimiter)
          if delimiter == '&lt;&gt;': # Will always put the item in a list
              if acc is None:
                  acc = [i] # Create one if doesn't exist
                  result.append(acc)
              else:
                  acc.append(i)
          else:
              if acc is not None: # If there was a list, put the last item in it
                  acc.append(i)
              else:
                  result.append(i) # Add the regular items
              acc = None # Clear the list, since what will come next is a regular item or a new list
      return result
  
  def split2(input): # Solution by duncan
      res0 = input.split(&quot;|&quot;)
      res1, res2 = [], []
      for r in res0:
          if &quot;&lt;&gt;&quot; in r:
              res2.append(r.split(&quot;&lt;&gt;&quot;))
          else:
              res1.append(r)
      return res1, res2
  
  print &quot;mgibs:&quot;, timeit.Timer(&quot;mgibsonbr('a|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import mgibsonbr&quot;).timeit()
  print &quot;split:&quot;, timeit.Timer(&quot;splitit('a|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import splitit&quot;).timeit()
  print &quot;split2:&quot;, timeit.Timer(&quot;split2('a|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import split2&quot;).timeit()
  print &quot;regex:&quot;, timeit.Timer(&quot;regexit('a|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import regexit&quot;).timeit()
  print &quot;mgibs:&quot;, timeit.Timer(&quot;mgibsonbr('a|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import mgibsonbr&quot;).timeit()
  print &quot;split:&quot;, timeit.Timer(&quot;splitit('a|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import splitit&quot;).timeit()
  print &quot;split:&quot;, timeit.Timer(&quot;split2('a|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import split2&quot;).timeit()
  print &quot;regex:&quot;, timeit.Timer(&quot;regexit('a|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;aha|b|c|de|f&lt;&gt;ge&lt;&gt;ah')&quot;,&quot;from __main__ import regexit&quot;).timeit()
  </code></pre>
  <p>The results:</p>
  <pre class="lang-none prettyprint-override"><code>mgibs: 14.7349407408
  split: 6.403942732
  split2: 3.68306812233
  regex: 5.28414318792
  mgibs: 107.046683735
  split: 46.0844590775
  split2: 26.5595985591
  regex: 28.6513302646
  </code></pre>
  <p>At the moment, it looks like split2 by duncan beats all other algorithms, regardless of length (with this limited dataset at least), and it also looks like mgibsonbr's solution has some performance issues (sorry about that, but thanks for the solution regardless).</p>